2024-07-26T08:07:24.058 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T08:07:24.087 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T08:07:24.363 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818712 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} email: vshankar@redhat.com first_in_suite: false job_id: '7818712' kernel: client: branch: testing flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug cephsqlite: 20 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mds: mds_bal_replicate_threshold: 1 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhjYSp45Zswj8xqADuwZX3M3UrlTklbPVOea1JTjwBybyouxaGf/10wSmW8eLMzakSrac40Tsu8jfVi41GSNdw= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD0fnxlt1vjBYWSR/XBTClQ4UHqLocfLohn84HZI8cODZW6CaJNZqbB7Vs+wFBOsB6GijXxQkIWOwCAtR/IyTF0= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC51wj8doaXqVlA7rDQlArBdIS1HKoSJlD7o81VpL6mJGVsiFxnntFHgEHy+vao0h0Y8iQ85ufSHgsLf6D6t7Os= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T08:07:24.363 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-26T08:07:24.364 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-26T08:07:24.364 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T08:07:24.366 INFO:teuthology.task.internal:Checking packages... 2024-07-26T08:07:24.388 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-26T08:07:24.388 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T08:07:24.388 INFO:teuthology.packaging:ref: None 2024-07-26T08:07:24.388 INFO:teuthology.packaging:tag: None 2024-07-26T08:07:24.388 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-26T08:07:24.388 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:07:24.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-26T08:07:24.526 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-26T08:07:24.528 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T08:07:24.537 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T08:07:24.537 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T08:07:24.552 INFO:teuthology.task.internal:Saving configuration 2024-07-26T08:07:24.574 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T08:07:24.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T08:07:24.610 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818712', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 08:02:19.527264', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhjYSp45Zswj8xqADuwZX3M3UrlTklbPVOea1JTjwBybyouxaGf/10wSmW8eLMzakSrac40Tsu8jfVi41GSNdw='} 2024-07-26T08:07:24.631 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818712', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 08:02:19.525264', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD0fnxlt1vjBYWSR/XBTClQ4UHqLocfLohn84HZI8cODZW6CaJNZqbB7Vs+wFBOsB6GijXxQkIWOwCAtR/IyTF0='} 2024-07-26T08:07:24.655 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818712', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 08:02:19.529058', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC51wj8doaXqVlA7rDQlArBdIS1HKoSJlD7o81VpL6mJGVsiFxnntFHgEHy+vao0h0Y8iQ85ufSHgsLf6D6t7Os='} 2024-07-26T08:07:24.655 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T08:07:24.666 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-26T08:07:24.666 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-26T08:07:24.666 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-26T08:07:24.666 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T08:07:24.783 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f685650fac0>, signals=[15]) 2024-07-26T08:07:24.783 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T08:07:24.792 INFO:teuthology.task.internal:Opening connections... 2024-07-26T08:07:24.792 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:07:24.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:07:24.874 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-07-26T08:07:24.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:07:24.948 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-07-26T08:07:24.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:07:25.021 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T08:07:25.033 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-07-26T08:07:25.052 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-07-26T08:07:25.053 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-07-26T08:07:25.111 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T08:07:25.112 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-07-26T08:07:25.112 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T08:07:25.112 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T08:07:25.112 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T08:07:25.112 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-07-26T08:07:25.137 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-07-26T08:07:25.155 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-07-26T08:07:25.155 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-07-26T08:07:25.212 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T08:07:25.213 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T08:07:25.214 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-07-26T08:07:25.245 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-07-26T08:07:25.263 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-07-26T08:07:25.263 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-07-26T08:07:25.321 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-07-26T08:07:25.321 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="9" 2024-07-26T08:07:25.321 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-07-26T08:07:25.321 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="9" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el9" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:LOGO="fedora-logo-icon" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T08:07:25.322 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T08:07:25.323 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-07-26T08:07:25.346 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T08:07:25.356 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T08:07:25.366 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T08:07:25.366 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-07-26T08:07:25.368 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-07-26T08:07:25.371 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-07-26T08:07:25.387 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T08:07:25.398 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T08:07:25.398 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-07-26T08:07:25.426 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-07-26T08:07:25.428 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-07-26T08:07:25.455 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T08:07:25.518 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T08:07:25.533 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T08:07:25.533 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-07-26T08:07:25.533 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-07-26T08:07:25.534 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T08:07:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T08:07:25.552 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-07-26T08:07:25.724 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T08:07:25.724 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T08:07:25.724 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-07-26T08:07:25.742 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-437.el9.x86_64 2024-07-26T08:07:25.742 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T08:07:25.742 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T08:07:25.743 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum upgrade -y linux-firmware 2024-07-26T08:07:28.202 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.2 MB 00:00 2024-07-26T08:07:29.978 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-07-26T08:07:34.770 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.5 MB 00:00 2024-07-26T08:07:36.747 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 18 kB 00:00 2024-07-26T08:07:37.683 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-07-26T08:07:43.189 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-26T08:07:45.385 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-07-26T08:07:45.399 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-07-26T08:07:45.399 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-07-26T08:07:45.399 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-07-26T08:07:45.399 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-07-26T08:07:45.399 INFO:teuthology.orchestra.run.smithi044.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T08:07:45.399 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:07:45.400 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-07-26T08:07:45.400 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-07-26T08:07:45.400 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 1 Package 2024-07-26T08:07:45.400 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:07:45.400 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 383 M 2024-07-26T08:07:45.400 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-07-26T08:08:18.080 INFO:teuthology.orchestra.run.smithi044.stdout:linux-firmware-20240716-145.el9.noarch.rpm 12 MB/s | 383 MB 00:32 2024-07-26T08:08:18.081 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-07-26T08:08:18.081 INFO:teuthology.orchestra.run.smithi044.stdout:Total 12 MB/s | 383 MB 00:32 2024-07-26T08:08:21.306 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-07-26T08:08:21.387 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-07-26T08:08:21.387 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-07-26T08:08:24.775 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-07-26T08:08:24.775 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-07-26T08:08:33.899 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-07-26T08:08:34.701 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T08:08:34.792 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T08:08:35.347 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T08:08:35.347 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T08:08:36.089 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T08:08:36.089 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:08:36.089 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-07-26T08:08:36.089 INFO:teuthology.orchestra.run.smithi044.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T08:08:36.089 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:08:36.089 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-07-26T08:08:36.404 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T08:08:36.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T08:08:36.555 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T08:08:36.555 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T08:08:36.558 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi044.front.sepia.ceph.com to complete... 2024-07-26T08:08:42.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26 08:08:42 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T08:08:42.844 INFO:teuthology.orchestra.run.smithi044.stderr:FINISHED --2024-07-26 08:08:42-- 2024-07-26T08:08:42.844 INFO:teuthology.orchestra.run.smithi044.stderr:Total wall clock time: 6.3s 2024-07-26T08:08:42.844 INFO:teuthology.orchestra.run.smithi044.stderr:Downloaded: 1 files, 342M in 6.0s (57.1 MB/s) 2024-07-26T08:08:42.848 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T08:08:42.848 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T08:08:45.005 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying... ######################################## 2024-07-26T08:08:45.057 INFO:teuthology.orchestra.run.smithi044.stdout:Preparing... ######################################## 2024-07-26T08:08:45.111 INFO:teuthology.orchestra.run.smithi044.stdout:Updating / installing... 2024-07-26T08:08:53.442 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T08:09:34.350 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T08:09:34.350 DEBUG:teuthology.orchestra.run.smithi044:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T08:09:34.445 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T08:09:34.446 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T08:09:34.447 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T08:09:34.448 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T08:09:34.449 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T08:09:34.450 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T08:09:34.451 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T08:09:34.452 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T08:09:34.453 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T08:09:34.454 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T08:09:34.455 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T08:09:34.456 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T08:09:34.457 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T08:09:34.458 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T08:09:34.459 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T08:09:34.460 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T08:09:34.461 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T08:09:34.462 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T08:09:34.463 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T08:09:34.464 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T08:09:34.465 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T08:09:34.466 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T08:09:34.467 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T08:09:34.468 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T08:09:34.469 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T08:09:34.470 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T08:09:34.471 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T08:09:34.472 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T08:09:34.473 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T08:09:34.474 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T08:09:34.475 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T08:09:34.476 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T08:09:34.477 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T08:09:34.478 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T08:09:34.479 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T08:09:34.480 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T08:09:34.481 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T08:09:34.482 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T08:09:34.483 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T08:09:34.484 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T08:09:34.485 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T08:09:34.486 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T08:09:34.487 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T08:09:34.488 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T08:09:34.489 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T08:09:34.490 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T08:09:34.491 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T08:09:34.492 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T08:09:34.493 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T08:09:34.494 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T08:09:34.495 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T08:09:34.496 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T08:09:34.497 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T08:09:34.498 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T08:09:34.499 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T08:09:34.500 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T08:09:34.501 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T08:09:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T08:09:34.503 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T08:09:34.504 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T08:09:34.505 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T08:09:34.506 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T08:09:34.507 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T08:09:34.508 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T08:09:34.509 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T08:09:34.510 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T08:09:34.511 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T08:09:34.512 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T08:09:34.513 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T08:09:34.514 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T08:09:34.515 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T08:09:34.516 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T08:09:34.517 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T08:09:34.518 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T08:09:34.519 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T08:09:34.520 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T08:09:34.521 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T08:09:34.522 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T08:09:34.523 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T08:09:34.524 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T08:09:34.525 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T08:09:34.526 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T08:09:34.527 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T08:09:34.528 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T08:09:34.529 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T08:09:34.530 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T08:09:34.531 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T08:09:34.532 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T08:09:34.533 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T08:09:34.534 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T08:09:34.535 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T08:09:34.536 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T08:09:34.537 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T08:09:34.538 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T08:09:34.539 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T08:09:34.540 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T08:09:34.541 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T08:09:34.542 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T08:09:34.543 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T08:09:34.544 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T08:09:34.545 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T08:09:34.546 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T08:09:34.547 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T08:09:34.548 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T08:09:34.549 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T08:09:34.550 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T08:09:34.551 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T08:09:34.552 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T08:09:34.553 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T08:09:34.554 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T08:09:34.555 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T08:09:34.556 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T08:09:34.557 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T08:09:34.558 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T08:09:34.559 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T08:09:34.560 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T08:09:34.561 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T08:09:34.562 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T08:09:34.563 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T08:09:34.564 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T08:09:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T08:09:34.566 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T08:09:34.567 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T08:09:34.568 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T08:09:34.569 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T08:09:34.570 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T08:09:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T08:09:34.572 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T08:09:34.573 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T08:09:34.574 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T08:09:34.575 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T08:09:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T08:09:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T08:09:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T08:09:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T08:09:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T08:09:34.581 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T08:09:34.582 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T08:09:34.583 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T08:09:34.584 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T08:09:34.585 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T08:09:34.586 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T08:09:34.587 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T08:09:34.588 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T08:09:34.589 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T08:09:34.590 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T08:09:34.591 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T08:09:34.592 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T08:09:34.593 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T08:09:34.594 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T08:09:34.595 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T08:09:34.596 INFO:teuthology.orchestra.run.smithi044.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T08:09:34.600 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T08:09:34.601 DEBUG:teuthology.orchestra.run.smithi044:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T08:09:34.625 INFO:teuthology.orchestra.run.smithi044.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T08:09:34.625 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T08:09:34.625 INFO:teuthology.orchestra.run.smithi044.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T08:09:34.625 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-07-26T08:09:34.625 INFO:teuthology.orchestra.run.smithi044.stdout:for file in vmlinuz System.map config; do 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: fi 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout:done 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-07-26T08:09:34.626 INFO:teuthology.orchestra.run.smithi044.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T08:09:34.627 INFO:teuthology.orchestra.run.smithi044.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T08:09:34.627 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T08:09:34.627 INFO:teuthology.orchestra.run.smithi044.stdout:fi 2024-07-26T08:09:34.628 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-07-26T08:09:34.702 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-07-26T08:09:34.702 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-07-26T08:09:34.702 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-07-26T08:09:34.702 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-07-26T08:09:34.702 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-07-26T08:09:34.703 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:09:34.704 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T08:09:34.705 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T08:09:34.705 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 6.10.0-g3e9255e90710 2024-07-26T08:09:34.705 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T08:09:35.365 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-07-26T08:09:36.809 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T08:09:36.844 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-07-26T08:09:36.846 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-07-26T08:09:36.902 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.1NLW9D6lJ4 2024-07-26T08:09:36.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1NLW9D6lJ4 2024-07-26T08:09:36.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.1NLW9D6lJ4 2024-07-26T08:09:37.105 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.1NLW9D6lJ4 is 6KB 2024-07-26T08:09:37.154 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.1NLW9D6lJ4 2024-07-26T08:09:37.168 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T08:09:37.231 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T08:09:37.231 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T08:09:37.232 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T08:09:37.232 INFO:teuthology.orchestra.run.smithi044.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T08:09:37.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T08:09:37.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-07-26T08:10:07.397 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-26T08:10:07.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-07-26T08:10:07.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:11:07.402 DEBUG:teuthology.orchestra.remote:timed out 2024-07-26T08:11:16.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-07-26T08:11:16.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:11:17.748 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-07-26T08:11:18.587 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-07-26T08:11:18.587 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.10.0-g3e9255e90710"... 2024-07-26T08:11:18.588 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-07-26T08:11:18.630 INFO:teuthology.orchestra.run.smithi044.stdout:6.10.0-g3e9255e90710 2024-07-26T08:11:18.630 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T08:11:18.630 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T08:11:18.630 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T08:11:19.634 DEBUG:teuthology.parallel:result is None 2024-07-26T08:11:19.635 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T08:11:19.646 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T08:11:19.646 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T08:11:19.654 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T08:11:19.656 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T08:11:19.697 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T08:11:19.704 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T08:11:19.711 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T08:11:19.711 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T08:11:19.716 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T08:11:19.719 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T08:11:19.827 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T08:11:19.834 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T08:11:19.834 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T08:11:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T08:11:19.851 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T08:11:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T08:11:19.869 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T08:11:19.875 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T08:11:19.893 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T08:11:19.927 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T08:11:19.945 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T08:11:19.962 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T08:11:19.976 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T08:11:19.996 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T08:11:20.041 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T08:11:20.046 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T08:11:20.052 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T08:11:20.052 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T08:11:20.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T08:11:20.059 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T08:11:20.129 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T08:11:20.139 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T08:11:20.139 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T08:11:20.144 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T08:11:20.146 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T08:11:20.187 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T08:11:20.260 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T08:11:20.336 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T08:11:20.401 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T08:11:20.488 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:11:20.488 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T08:11:20.559 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T08:11:20.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T08:11:20.655 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T08:11:20.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T08:11:20.784 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:11:20.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T08:11:20.851 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T08:11:20.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T08:11:20.944 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T08:11:21.025 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T08:11:21.069 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:11:21.070 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T08:11:21.135 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-07-26T08:11:21.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-07-26T08:11:21.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-07-26T08:11:21.175 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T08:11:21.214 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T08:11:21.252 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T08:11:21.627 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T08:11:21.636 INFO:teuthology.task.internal:Starting timer... 2024-07-26T08:11:21.636 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T08:11:21.646 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T08:11:21.655 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-26T08:11:21.655 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-07-26T08:11:21.792 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-07-26T08:11:21.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-07-26T08:11:21.868 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-07-26T08:11:21.871 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-07-26T08:11:21.943 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-07-26T08:11:21.945 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T08:11:21.946 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-07-26T08:11:21.995 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-07-26T08:11:21.996 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-07-26T08:11:22.021 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-07-26T08:11:22.022 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-07-26T08:11:22.047 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-07-26T08:11:22.048 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-07-26T08:11:22.048 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T08:11:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T08:11:22.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T08:11:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T08:11:22.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T08:11:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T08:11:22.181 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T08:11:22.181 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-07-26T08:11:22.345 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-07-26T08:11:22.372 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-07-26T08:11:22.399 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T08:11:22.409 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T08:11:22.412 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T08:11:22.527 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T08:11:22.537 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T08:11:22.538 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi116.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T08:19:06.138 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-07-26T08:19:06.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-07-26T08:19:06.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:19:06.331 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-07-26T08:19:06.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-07-26T08:19:06.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-07-26T08:19:06.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:19:06.571 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-07-26T08:19:06.655 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-07-26T08:19:06.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-07-26T08:19:06.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T08:19:06.745 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-07-26T08:19:06.827 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-07-26T08:19:06.829 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T08:19:06.841 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T08:19:06.841 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T08:19:06.842 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T08:19:06.850 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T08:19:06.850 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T08:19:06.853 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T08:19:06.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T08:19:06.883 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T08:19:06.898 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T08:19:06.898 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T08:19:06.912 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T08:19:06.923 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-07-26T08:19:06.934 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-07-26T08:19:06.936 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-07-26T08:19:06.939 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T08:19:06.947 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-07-26T08:19:06.947 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T08:19:06.960 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T08:19:06.961 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T08:19:06.973 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T08:19:06.981 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T08:19:07.017 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-07-26T08:19:07.020 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T08:19:07.020 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-07-26T08:19:07.020 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.021 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.021 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.021 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.030 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-07-26T08:19:07.034 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T08:19:07.034 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-07-26T08:19:07.034 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.034 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.034 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.034 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.054 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-07-26T08:19:07.088 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-07-26T08:19:07.134 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T08:19:07.179 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T08:19:07.333 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-07-26T08:19:07.343 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T08:19:07.343 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-07-26T08:19:07.343 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.343 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.343 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.343 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T08:19:07.345 INFO:teuthology.run_tasks:Running task install... 2024-07-26T08:19:07.353 DEBUG:teuthology.task.install:project ceph 2024-07-26T08:19:07.353 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-26T08:19:07.354 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-26T08:19:07.354 INFO:teuthology.task.install:Using flavor: default 2024-07-26T08:19:07.386 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T08:19:07.386 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T08:19:07.386 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T08:19:07.387 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:19:07.389 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T08:19:07.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:19:07.390 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T08:19:07.391 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:19:07.546 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T08:19:07.547 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T08:19:07.551 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T08:19:07.551 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T08:19:07.555 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T08:19:07.555 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T08:19:07.665 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T08:19:07.665 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:19:07.665 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T08:19:07.673 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T08:19:07.673 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:19:07.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T08:19:07.679 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T08:19:07.679 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:19:07.679 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T08:19:07.699 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T08:19:07.702 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T08:19:07.705 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T08:19:07.705 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T08:19:07.765 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T08:19:07.765 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T08:19:07.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T08:19:07.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T08:19:07.841 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T08:19:07.845 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T08:19:07.900 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T08:19:07.908 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-07-26T08:19:07.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-07-26T08:19:07.910 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-07-26T08:19:07.911 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-07-26T08:19:08.019 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T08:19:08.103 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-07-26T08:19:08.112 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-07-26T08:19:08.129 INFO:teuthology.orchestra.run.smithi116.stdout:45 files removed 2024-07-26T08:19:08.137 INFO:teuthology.orchestra.run.smithi171.stdout:45 files removed 2024-07-26T08:19:08.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T08:19:08.163 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T08:19:08.459 INFO:teuthology.orchestra.run.smithi044.stdout:45 files removed 2024-07-26T08:19:08.499 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T08:19:08.755 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 254 kB/s | 83 kB 00:00 2024-07-26T08:19:08.756 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 245 kB/s | 83 kB 00:00 2024-07-26T08:19:09.097 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-07-26T08:19:09.107 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 78 kB/s | 24 kB 00:00 2024-07-26T08:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-07-26T08:19:09.431 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-07-26T08:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-07-26T08:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 57 kB/s | 24 kB 00:00 2024-07-26T08:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-07-26T08:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.2 MB 00:00 2024-07-26T08:19:11.474 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.2 MB 00:01 2024-07-26T08:19:11.515 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-07-26T08:19:13.667 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-07-26T08:19:14.443 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-07-26T08:19:18.026 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-07-26T08:19:19.050 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.5 MB 00:00 2024-07-26T08:19:19.928 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 18 kB 00:00 2024-07-26T08:19:21.093 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 18 kB 00:00 2024-07-26T08:19:21.247 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 19 MB/s | 22 MB 00:01 2024-07-26T08:19:21.988 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 29 MB/s | 22 MB 00:00 2024-07-26T08:19:26.612 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-07-26T08:19:27.302 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-07-26T08:19:28.867 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T08:19:28.868 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T08:19:28.879 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T08:19:28.880 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T08:19:28.880 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T08:19:28.882 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T08:19:28.883 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T08:19:28.885 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T08:19:28.886 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T08:19:28.950 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-07-26T08:19:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-07-26T08:19:28.961 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-07-26T08:19:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-07-26T08:19:28.961 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-07-26T08:19:28.961 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T08:19:28.961 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T08:19:28.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T08:19:28.963 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T08:19:28.964 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T08:19:28.965 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T08:19:28.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T08:19:28.967 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T08:19:28.968 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T08:19:28.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T08:19:28.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T08:19:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T08:19:28.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T08:19:28.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T08:19:28.974 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:Install 148 Packages 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 12 Packages 2024-07-26T08:19:28.975 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:19:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 233 M 2024-07-26T08:19:28.976 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-07-26T08:19:29.571 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T08:19:29.572 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T08:19:29.583 INFO:teuthology.orchestra.run.smithi171.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T08:19:29.584 INFO:teuthology.orchestra.run.smithi171.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T08:19:29.585 INFO:teuthology.orchestra.run.smithi171.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T08:19:29.586 INFO:teuthology.orchestra.run.smithi171.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T08:19:29.588 INFO:teuthology.orchestra.run.smithi171.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T08:19:29.590 INFO:teuthology.orchestra.run.smithi171.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T08:19:29.591 INFO:teuthology.orchestra.run.smithi171.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T08:19:29.596 INFO:teuthology.orchestra.run.smithi116.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-07-26T08:19:29.655 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-07-26T08:19:29.682 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2024-07-26T08:19:29.682 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-07-26T08:19:29.682 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2024-07-26T08:19:29.682 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-07-26T08:19:29.682 INFO:teuthology.orchestra.run.smithi171.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T08:19:29.683 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T08:19:29.684 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-07-26T08:19:29.685 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T08:19:29.686 INFO:teuthology.orchestra.run.smithi171.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T08:19:29.687 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T08:19:29.688 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T08:19:29.689 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T08:19:29.690 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T08:19:29.691 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T08:19:29.692 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T08:19:29.693 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T08:19:29.694 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T08:19:29.695 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout:Install 148 Packages 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 12 Packages 2024-07-26T08:19:29.696 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:19:29.697 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 233 M 2024-07-26T08:19:29.697 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-07-26T08:19:29.796 INFO:teuthology.orchestra.run.smithi116.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.3 MB/s | 872 kB 00:00 2024-07-26T08:19:29.847 INFO:teuthology.orchestra.run.smithi116.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.9 MB/s | 148 kB 00:00 2024-07-26T08:19:30.039 INFO:teuthology.orchestra.run.smithi116.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 8.9 MB/s | 5.5 MB 00:00 2024-07-26T08:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 9.0 MB/s | 2.3 MB 00:00 2024-07-26T08:19:30.189 INFO:teuthology.orchestra.run.smithi116.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 10 MB/s | 1.5 MB 00:00 2024-07-26T08:19:30.445 INFO:teuthology.orchestra.run.smithi171.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-07-26T08:19:30.531 INFO:teuthology.orchestra.run.smithi116.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 11 MB/s | 4.9 MB 00:00 2024-07-26T08:19:30.790 INFO:teuthology.orchestra.run.smithi116.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 20 MB 00:01 2024-07-26T08:19:30.812 INFO:teuthology.orchestra.run.smithi171.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.3 MB/s | 872 kB 00:00 2024-07-26T08:19:30.840 INFO:teuthology.orchestra.run.smithi116.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 507 kB/s | 25 kB 00:00 2024-07-26T08:19:30.915 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:18 2024-07-26T08:19:30.962 INFO:teuthology.orchestra.run.smithi171.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 988 kB/s | 148 kB 00:00 2024-07-26T08:19:31.291 INFO:teuthology.orchestra.run.smithi116.stdout:(10/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 16 MB/s | 18 MB 00:01 2024-07-26T08:19:31.329 INFO:teuthology.orchestra.run.smithi171.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 4.9 MB/s | 5.5 MB 00:01 2024-07-26T08:19:31.416 INFO:teuthology.orchestra.run.smithi116.stdout:(11/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 12 MB/s | 10 MB 00:00 2024-07-26T08:19:31.441 INFO:teuthology.orchestra.run.smithi116.stdout:(12/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 1.5 MB/s | 225 kB 00:00 2024-07-26T08:19:31.446 INFO:teuthology.orchestra.run.smithi171.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 1.5 MB 00:00 2024-07-26T08:19:31.458 INFO:teuthology.orchestra.run.smithi116.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 650 kB/s | 27 kB 00:00 2024-07-26T08:19:31.475 INFO:teuthology.orchestra.run.smithi116.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 946 kB/s | 32 kB 00:00 2024-07-26T08:19:31.525 INFO:teuthology.orchestra.run.smithi116.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 765 kB 00:00 2024-07-26T08:19:31.550 INFO:teuthology.orchestra.run.smithi116.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.0 MB/s | 158 kB 00:00 2024-07-26T08:19:31.554 INFO:teuthology.orchestra.run.smithi171.stdout:(6/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 3.8 MB/s | 2.3 MB 00:00 2024-07-26T08:19:31.593 INFO:teuthology.orchestra.run.smithi116.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.9 MB/s | 127 kB 00:00 2024-07-26T08:19:31.626 INFO:teuthology.orchestra.run.smithi116.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 6.6 MB/s | 503 kB 00:00 2024-07-26T08:19:31.726 INFO:teuthology.orchestra.run.smithi116.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 450 kB/s | 45 kB 00:00 2024-07-26T08:19:31.835 INFO:teuthology.orchestra.run.smithi116.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.2 MB/s | 138 kB 00:00 2024-07-26T08:19:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 2.0 MB/s | 188 kB 00:00 2024-07-26T08:19:32.011 INFO:teuthology.orchestra.run.smithi116.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 5.3 MB 00:00 2024-07-26T08:19:32.013 INFO:teuthology.orchestra.run.smithi171.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 8.6 MB/s | 4.9 MB 00:00 2024-07-26T08:19:32.036 INFO:teuthology.orchestra.run.smithi116.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 3.4 MB/s | 372 kB 00:00 2024-07-26T08:19:32.077 INFO:teuthology.orchestra.run.smithi116.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 5.2 MB/s | 355 kB 00:00 2024-07-26T08:19:32.103 INFO:teuthology.orchestra.run.smithi116.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.7 MB/s | 116 kB 00:00 2024-07-26T08:19:32.136 INFO:teuthology.orchestra.run.smithi116.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.4 MB/s | 86 kB 00:00 2024-07-26T08:19:32.229 INFO:teuthology.orchestra.run.smithi116.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 1.8 MB/s | 170 kB 00:00 2024-07-26T08:19:32.287 INFO:teuthology.orchestra.run.smithi116.stdout:(28/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 17 MB/s | 3.1 MB 00:00 2024-07-26T08:19:32.304 INFO:teuthology.orchestra.run.smithi116.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 457 kB/s | 34 kB 00:00 2024-07-26T08:19:32.354 INFO:teuthology.orchestra.run.smithi116.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.2 MB/s | 149 kB 00:00 2024-07-26T08:19:32.896 INFO:teuthology.orchestra.run.smithi116.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-07-26T08:19:32.971 INFO:teuthology.orchestra.run.smithi116.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 3.7 MB/s | 284 kB 00:00 2024-07-26T08:19:33.038 INFO:teuthology.orchestra.run.smithi116.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 739 kB/s | 49 kB 00:00 2024-07-26T08:19:33.422 INFO:teuthology.orchestra.run.smithi116.stdout:(34/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 17 MB/s | 44 MB 00:02 2024-07-26T08:19:33.447 INFO:teuthology.orchestra.run.smithi116.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 40 kB/s | 16 kB 00:00 2024-07-26T08:19:33.473 INFO:teuthology.orchestra.run.smithi116.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 5.4 MB/s | 274 kB 00:00 2024-07-26T08:19:33.497 INFO:teuthology.orchestra.run.smithi171.stdout:(8/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el9 7.0 MB/s | 10 MB 00:01 2024-07-26T08:19:33.515 INFO:teuthology.orchestra.run.smithi116.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 439 kB/s | 18 kB 00:00 2024-07-26T08:19:33.538 INFO:teuthology.orchestra.run.smithi171.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 607 kB/s | 25 kB 00:00 2024-07-26T08:19:33.540 INFO:teuthology.orchestra.run.smithi116.stdout:(38/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 8.2 MB/s | 771 kB 00:00 2024-07-26T08:19:33.674 INFO:teuthology.orchestra.run.smithi116.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 10 MB/s | 14 MB 00:01 2024-07-26T08:19:34.016 INFO:teuthology.orchestra.run.smithi116.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 159 kB/s | 80 kB 00:00 2024-07-26T08:19:34.183 INFO:teuthology.orchestra.run.smithi116.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-07-26T08:19:34.247 INFO:teuthology.orchestra.run.smithi171.stdout:(10/160): ceph-common-19.0.0-5320.ge8f88bb1.el9 4.9 MB/s | 20 MB 00:04 2024-07-26T08:19:34.283 INFO:teuthology.orchestra.run.smithi116.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 949 kB/s | 253 kB 00:00 2024-07-26T08:19:34.322 INFO:teuthology.orchestra.run.smithi171.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 2.9 MB/s | 225 kB 00:00 2024-07-26T08:19:34.364 INFO:teuthology.orchestra.run.smithi171.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 651 kB/s | 27 kB 00:00 2024-07-26T08:19:34.392 INFO:teuthology.orchestra.run.smithi116.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-07-26T08:19:34.398 INFO:teuthology.orchestra.run.smithi171.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 952 kB/s | 32 kB 00:00 2024-07-26T08:19:34.500 INFO:teuthology.orchestra.run.smithi116.stdout:(44/160): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-07-26T08:19:34.515 INFO:teuthology.orchestra.run.smithi171.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 6.4 MB/s | 765 kB 00:00 2024-07-26T08:19:34.565 INFO:teuthology.orchestra.run.smithi171.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 3.1 MB/s | 158 kB 00:00 2024-07-26T08:19:34.609 INFO:teuthology.orchestra.run.smithi116.stdout:(45/160): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-07-26T08:19:34.615 INFO:teuthology.orchestra.run.smithi171.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 2.5 MB/s | 127 kB 00:00 2024-07-26T08:19:34.642 INFO:teuthology.orchestra.run.smithi116.stdout:(46/160): python3-cryptography-36.0.1-4.el9.x86 2.7 MB/s | 1.2 MB 00:00 2024-07-26T08:19:34.716 INFO:teuthology.orchestra.run.smithi171.stdout:(17/160): libradosstriper1-19.0.0-5320.ge8f88bb 4.9 MB/s | 503 kB 00:00 2024-07-26T08:19:34.726 INFO:teuthology.orchestra.run.smithi116.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-07-26T08:19:34.818 INFO:teuthology.orchestra.run.smithi116.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-07-26T08:19:34.851 INFO:teuthology.orchestra.run.smithi116.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 107 kB 00:00 2024-07-26T08:19:34.885 INFO:teuthology.orchestra.run.smithi116.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 652 kB/s | 22 kB 00:00 2024-07-26T08:19:34.944 INFO:teuthology.orchestra.run.smithi116.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 4.2 MB/s | 943 kB 00:00 2024-07-26T08:19:34.969 INFO:teuthology.orchestra.run.smithi116.stdout:(52/160): libgfortran-11.4.1-3.el9.x86_64.rpm 562 kB/s | 803 kB 00:01 2024-07-26T08:19:34.986 INFO:teuthology.orchestra.run.smithi116.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 780 kB/s | 33 kB 00:00 2024-07-26T08:19:35.011 INFO:teuthology.orchestra.run.smithi116.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-07-26T08:19:35.028 INFO:teuthology.orchestra.run.smithi116.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 436 kB/s | 18 kB 00:00 2024-07-26T08:19:35.045 INFO:teuthology.orchestra.run.smithi116.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 332 kB/s | 11 kB 00:00 2024-07-26T08:19:35.070 INFO:teuthology.orchestra.run.smithi116.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-07-26T08:19:35.087 INFO:teuthology.orchestra.run.smithi116.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-07-26T08:19:35.129 INFO:teuthology.orchestra.run.smithi116.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-07-26T08:19:35.150 INFO:teuthology.orchestra.run.smithi171.stdout:(18/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 4.9 MB/s | 18 MB 00:03 2024-07-26T08:19:35.163 INFO:teuthology.orchestra.run.smithi116.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 7.6 MB/s | 581 kB 00:00 2024-07-26T08:19:35.188 INFO:teuthology.orchestra.run.smithi116.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 436 kB/s | 25 kB 00:00 2024-07-26T08:19:35.200 INFO:teuthology.orchestra.run.smithi171.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 909 kB/s | 45 kB 00:00 2024-07-26T08:19:35.213 INFO:teuthology.orchestra.run.smithi116.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-07-26T08:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-07-26T08:19:35.250 INFO:teuthology.orchestra.run.smithi171.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.7 MB/s | 138 kB 00:00 2024-07-26T08:19:35.264 INFO:teuthology.orchestra.run.smithi116.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-07-26T08:19:35.289 INFO:teuthology.orchestra.run.smithi116.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-07-26T08:19:35.300 INFO:teuthology.orchestra.run.smithi171.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.7 MB/s | 188 kB 00:00 2024-07-26T08:19:35.356 INFO:teuthology.orchestra.run.smithi116.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.7 MB/s | 3.0 MB 00:00 2024-07-26T08:19:35.417 INFO:teuthology.orchestra.run.smithi171.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 7.5 MB/s | 5.3 MB 00:00 2024-07-26T08:19:35.440 INFO:teuthology.orchestra.run.smithi116.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-07-26T08:19:35.442 INFO:teuthology.orchestra.run.smithi171.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 2.6 MB/s | 372 kB 00:00 2024-07-26T08:19:35.493 INFO:teuthology.orchestra.run.smithi171.stdout:(24/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T08:19:35.499 INFO:teuthology.orchestra.run.smithi116.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-07-26T08:19:35.518 INFO:teuthology.orchestra.run.smithi171.stdout:(25/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 3.5 MB/s | 355 kB 00:00 2024-07-26T08:19:35.532 INFO:teuthology.orchestra.run.smithi116.stdout:(69/160): postgresql-13.14-1.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-07-26T08:19:35.544 INFO:teuthology.orchestra.run.smithi171.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T08:19:35.558 INFO:teuthology.orchestra.run.smithi116.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-07-26T08:19:35.591 INFO:teuthology.orchestra.run.smithi116.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-07-26T08:19:35.594 INFO:teuthology.orchestra.run.smithi171.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 3.3 MB/s | 170 kB 00:00 2024-07-26T08:19:35.625 INFO:teuthology.orchestra.run.smithi116.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-07-26T08:19:35.642 INFO:teuthology.orchestra.run.smithi116.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-07-26T08:19:35.644 INFO:teuthology.orchestra.run.smithi171.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 689 kB/s | 34 kB 00:00 2024-07-26T08:19:35.684 INFO:teuthology.orchestra.run.smithi116.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-07-26T08:19:35.694 INFO:teuthology.orchestra.run.smithi171.stdout:(29/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.9 MB/s | 149 kB 00:00 2024-07-26T08:19:35.759 INFO:teuthology.orchestra.run.smithi116.stdout:(75/160): postgresql-server-13.14-1.el9.x86_64. 26 MB/s | 5.8 MB 00:00 2024-07-26T08:19:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-07-26T08:19:35.810 INFO:teuthology.orchestra.run.smithi116.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-07-26T08:19:35.835 INFO:teuthology.orchestra.run.smithi116.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-07-26T08:19:35.860 INFO:teuthology.orchestra.run.smithi116.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-07-26T08:19:35.885 INFO:teuthology.orchestra.run.smithi116.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-07-26T08:19:35.902 INFO:teuthology.orchestra.run.smithi116.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 829 kB/s | 35 kB 00:00 2024-07-26T08:19:35.912 INFO:teuthology.orchestra.run.smithi171.stdout:(30/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 7.8 MB/s | 3.1 MB 00:00 2024-07-26T08:19:35.986 INFO:teuthology.orchestra.run.smithi116.stdout:(82/160): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-07-26T08:19:36.011 INFO:teuthology.orchestra.run.smithi116.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-07-26T08:19:36.037 INFO:teuthology.orchestra.run.smithi116.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-07-26T08:19:36.062 INFO:teuthology.orchestra.run.smithi116.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-07-26T08:19:36.087 INFO:teuthology.orchestra.run.smithi116.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-07-26T08:19:36.121 INFO:teuthology.orchestra.run.smithi116.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-07-26T08:19:36.146 INFO:teuthology.orchestra.run.smithi116.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-07-26T08:19:36.171 INFO:teuthology.orchestra.run.smithi116.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-07-26T08:19:36.197 INFO:teuthology.orchestra.run.smithi116.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-07-26T08:19:36.222 INFO:teuthology.orchestra.run.smithi116.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-07-26T08:19:36.297 INFO:teuthology.orchestra.run.smithi116.stdout:(92/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-07-26T08:19:36.322 INFO:teuthology.orchestra.run.smithi116.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-07-26T08:19:36.340 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.5 MB 00:01 2024-07-26T08:19:36.348 INFO:teuthology.orchestra.run.smithi116.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 885 kB/s | 44 kB 00:00 2024-07-26T08:19:36.373 INFO:teuthology.orchestra.run.smithi116.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-07-26T08:19:36.398 INFO:teuthology.orchestra.run.smithi116.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-07-26T08:19:36.424 INFO:teuthology.orchestra.run.smithi116.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-07-26T08:19:36.449 INFO:teuthology.orchestra.run.smithi116.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-07-26T08:19:36.474 INFO:teuthology.orchestra.run.smithi116.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-07-26T08:19:36.499 INFO:teuthology.orchestra.run.smithi116.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-07-26T08:19:36.525 INFO:teuthology.orchestra.run.smithi116.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-07-26T08:19:36.550 INFO:teuthology.orchestra.run.smithi116.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-07-26T08:19:36.620 INFO:teuthology.orchestra.run.smithi171.stdout:(31/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 14 MB/s | 44 MB 00:03 2024-07-26T08:19:36.687 INFO:teuthology.orchestra.run.smithi171.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 4.2 MB/s | 284 kB 00:00 2024-07-26T08:19:36.737 INFO:teuthology.orchestra.run.smithi171.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 991 kB/s | 49 kB 00:00 2024-07-26T08:19:36.776 INFO:teuthology.orchestra.run.smithi116.stdout:(103/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-07-26T08:19:36.779 INFO:teuthology.orchestra.run.smithi171.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 396 kB/s | 16 kB 00:00 2024-07-26T08:19:36.830 INFO:teuthology.orchestra.run.smithi171.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 5.4 MB/s | 274 kB 00:00 2024-07-26T08:19:36.842 INFO:teuthology.orchestra.run.smithi116.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 8.1 MB/s | 548 kB 00:00 2024-07-26T08:19:36.867 INFO:teuthology.orchestra.run.smithi116.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-07-26T08:19:36.893 INFO:teuthology.orchestra.run.smithi116.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-07-26T08:19:36.905 INFO:teuthology.orchestra.run.smithi171.stdout:(36/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 10 MB/s | 771 kB 00:00 2024-07-26T08:19:36.947 INFO:teuthology.orchestra.run.smithi171.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 437 kB/s | 18 kB 00:00 2024-07-26T08:19:37.001 INFO:teuthology.orchestra.run.smithi116.stdout:(107/160): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-07-26T08:19:37.077 INFO:teuthology.orchestra.run.smithi116.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 992 kB/s | 26 kB 00:00 2024-07-26T08:19:37.080 INFO:teuthology.orchestra.run.smithi171.stdout:(38/160): ceph-mgr-diskprediction-local-19.0.0- 6.3 MB/s | 7.4 MB 00:01 2024-07-26T08:19:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-26T08:19:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-26T08:19:37.144 INFO:teuthology.orchestra.run.smithi116.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 9.4 MB/s | 151 kB 00:00 2024-07-26T08:19:37.204 INFO:teuthology.orchestra.run.smithi116.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-07-26T08:19:37.219 INFO:teuthology.orchestra.run.smithi116.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-07-26T08:19:37.245 INFO:teuthology.orchestra.run.smithi116.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T08:19:37.270 INFO:teuthology.orchestra.run.smithi116.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T08:19:37.295 INFO:teuthology.orchestra.run.smithi116.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 568 kB/s | 14 kB 00:00 2024-07-26T08:19:37.321 INFO:teuthology.orchestra.run.smithi116.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 6.7 MB/s | 172 kB 00:00 2024-07-26T08:19:37.354 INFO:teuthology.orchestra.run.smithi116.stdout:(118/160): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-07-26T08:19:37.380 INFO:teuthology.orchestra.run.smithi116.stdout:(119/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 12 MB/s | 303 kB 00:00 2024-07-26T08:19:37.396 INFO:teuthology.orchestra.run.smithi116.stdout:(120/160): lua-devel-5.4.4-4.el9.x86_64.rpm 26 kB/s | 22 kB 00:00 2024-07-26T08:19:37.422 INFO:teuthology.orchestra.run.smithi116.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 5.0 MB/s | 216 kB 00:00 2024-07-26T08:19:37.472 INFO:teuthology.orchestra.run.smithi116.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.5 MB/s | 144 kB 00:00 2024-07-26T08:19:37.489 INFO:teuthology.orchestra.run.smithi116.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 750 kB/s | 11 kB 00:00 2024-07-26T08:19:37.506 INFO:teuthology.orchestra.run.smithi116.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 1.0 MB/s | 18 kB 00:00 2024-07-26T08:19:37.522 INFO:teuthology.orchestra.run.smithi171.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 7.7 MB/s | 14 MB 00:01 2024-07-26T08:19:37.531 INFO:teuthology.orchestra.run.smithi116.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 923 kB/s | 23 kB 00:00 2024-07-26T08:19:37.548 INFO:teuthology.orchestra.run.smithi171.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 133 kB/s | 80 kB 00:00 2024-07-26T08:19:37.573 INFO:teuthology.orchestra.run.smithi116.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 467 kB/s | 19 kB 00:00 2024-07-26T08:19:37.601 INFO:teuthology.orchestra.run.smithi116.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 727 kB/s | 20 kB 00:00 2024-07-26T08:19:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-07-26T08:19:37.685 INFO:teuthology.orchestra.run.smithi116.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T08:19:37.710 INFO:teuthology.orchestra.run.smithi116.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T08:19:37.735 INFO:teuthology.orchestra.run.smithi116.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T08:19:37.769 INFO:teuthology.orchestra.run.smithi116.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-07-26T08:19:37.794 INFO:teuthology.orchestra.run.smithi116.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 664 kB/s | 16 kB 00:00 2024-07-26T08:19:37.828 INFO:teuthology.orchestra.run.smithi116.stdout:(134/160): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-07-26T08:19:37.845 INFO:teuthology.orchestra.run.smithi116.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T08:19:37.870 INFO:teuthology.orchestra.run.smithi116.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-07-26T08:19:37.873 INFO:teuthology.orchestra.run.smithi171.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 558 kB/s | 195 kB 00:00 2024-07-26T08:19:37.895 INFO:teuthology.orchestra.run.smithi116.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-07-26T08:19:37.907 INFO:teuthology.orchestra.run.smithi171.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 973 kB/s | 803 kB 00:00 2024-07-26T08:19:37.937 INFO:teuthology.orchestra.run.smithi116.stdout:(138/160): protobuf-compiler-3.14.0-13.el9.x86_ 622 kB/s | 863 kB 00:01 2024-07-26T08:19:37.963 INFO:teuthology.orchestra.run.smithi116.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 882 kB/s | 59 kB 00:00 2024-07-26T08:19:38.013 INFO:teuthology.orchestra.run.smithi116.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-07-26T08:19:38.015 INFO:teuthology.orchestra.run.smithi171.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-07-26T08:19:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T08:19:38.063 INFO:teuthology.orchestra.run.smithi116.stdout:(142/160): python3-tempora-5.0.0-2.el9.noarch.r 284 kB/s | 36 kB 00:00 2024-07-26T08:19:38.066 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 18 kB 00:00 2024-07-26T08:19:38.089 INFO:teuthology.orchestra.run.smithi116.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-26T08:19:38.114 INFO:teuthology.orchestra.run.smithi116.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 803 kB/s | 20 kB 00:00 2024-07-26T08:19:38.124 INFO:teuthology.orchestra.run.smithi171.stdout:(44/160): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-07-26T08:19:38.139 INFO:teuthology.orchestra.run.smithi116.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-07-26T08:19:38.164 INFO:teuthology.orchestra.run.smithi116.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T08:19:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-07-26T08:19:38.216 INFO:teuthology.orchestra.run.smithi171.stdout:(45/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 379 kB/s | 253 kB 00:00 2024-07-26T08:19:38.232 INFO:teuthology.orchestra.run.smithi116.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-07-26T08:19:38.274 INFO:teuthology.orchestra.run.smithi171.stdout:(46/160): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-07-26T08:19:38.300 INFO:teuthology.orchestra.run.smithi171.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 711 kB/s | 125 kB 00:00 2024-07-26T08:19:38.408 INFO:teuthology.orchestra.run.smithi171.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.1 MB/s | 215 kB 00:00 2024-07-26T08:19:38.440 INFO:teuthology.orchestra.run.smithi116.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 12 MB/s | 3.4 MB 00:00 2024-07-26T08:19:38.483 INFO:teuthology.orchestra.run.smithi171.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-07-26T08:19:38.491 INFO:teuthology.orchestra.run.smithi116.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 10 MB/s | 3.1 MB 00:00 2024-07-26T08:19:38.525 INFO:teuthology.orchestra.run.smithi171.stdout:(50/160): bison-3.7.4-5.el9.x86_64.rpm 4.1 MB/s | 943 kB 00:00 2024-07-26T08:19:38.551 INFO:teuthology.orchestra.run.smithi171.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 325 kB/s | 22 kB 00:00 2024-07-26T08:19:38.576 INFO:teuthology.orchestra.run.smithi171.stdout:(52/160): boost-program-options-1.75.0-8.el9.x8 639 kB/s | 107 kB 00:00 2024-07-26T08:19:38.601 INFO:teuthology.orchestra.run.smithi171.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 650 kB/s | 33 kB 00:00 2024-07-26T08:19:38.629 INFO:teuthology.orchestra.run.smithi171.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-07-26T08:19:38.645 INFO:teuthology.orchestra.run.smithi171.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 415 kB/s | 18 kB 00:00 2024-07-26T08:19:38.663 INFO:teuthology.orchestra.run.smithi171.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 332 kB/s | 11 kB 00:00 2024-07-26T08:19:38.666 INFO:teuthology.orchestra.run.smithi116.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 233 kB/s | 41 kB 00:00 2024-07-26T08:19:38.688 INFO:teuthology.orchestra.run.smithi171.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-07-26T08:19:38.705 INFO:teuthology.orchestra.run.smithi171.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-07-26T08:19:38.747 INFO:teuthology.orchestra.run.smithi171.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-07-26T08:19:38.780 INFO:teuthology.orchestra.run.smithi171.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 7.5 MB/s | 581 kB 00:00 2024-07-26T08:19:38.814 INFO:teuthology.orchestra.run.smithi171.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 381 kB/s | 25 kB 00:00 2024-07-26T08:19:38.839 INFO:teuthology.orchestra.run.smithi171.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-07-26T08:19:38.865 INFO:teuthology.orchestra.run.smithi171.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-07-26T08:19:38.890 INFO:teuthology.orchestra.run.smithi171.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-07-26T08:19:38.915 INFO:teuthology.orchestra.run.smithi171.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-07-26T08:19:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 927 kB/s | 247 kB 00:00 2024-07-26T08:19:38.965 INFO:teuthology.orchestra.run.smithi171.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.7 MB/s | 3.0 MB 00:00 2024-07-26T08:19:39.033 INFO:teuthology.orchestra.run.smithi116.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 367 kB/s | 37 kB 00:00 2024-07-26T08:19:39.058 INFO:teuthology.orchestra.run.smithi171.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-07-26T08:19:39.075 INFO:teuthology.orchestra.run.smithi116.stdout:(154/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.7 MB/s | 2.3 MB 00:00 2024-07-26T08:19:39.091 INFO:teuthology.orchestra.run.smithi171.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-07-26T08:19:39.133 INFO:teuthology.orchestra.run.smithi171.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-07-26T08:19:39.142 INFO:teuthology.orchestra.run.smithi116.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-07-26T08:19:39.158 INFO:teuthology.orchestra.run.smithi171.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-07-26T08:19:39.184 INFO:teuthology.orchestra.run.smithi116.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-07-26T08:19:39.225 INFO:teuthology.orchestra.run.smithi171.stdout:(71/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-07-26T08:19:39.259 INFO:teuthology.orchestra.run.smithi171.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-07-26T08:19:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-07-26T08:19:39.326 INFO:teuthology.orchestra.run.smithi171.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-07-26T08:19:39.410 INFO:teuthology.orchestra.run.smithi171.stdout:(75/160): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-07-26T08:19:39.435 INFO:teuthology.orchestra.run.smithi171.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-07-26T08:19:39.460 INFO:teuthology.orchestra.run.smithi171.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-07-26T08:19:39.485 INFO:teuthology.orchestra.run.smithi171.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-07-26T08:19:39.509 INFO:teuthology.orchestra.run.smithi116.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 3.7 MB/s | 1.4 MB 00:00 2024-07-26T08:19:39.511 INFO:teuthology.orchestra.run.smithi171.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-07-26T08:19:39.536 INFO:teuthology.orchestra.run.smithi171.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-07-26T08:19:39.561 INFO:teuthology.orchestra.run.smithi171.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-07-26T08:19:39.565 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 17 MB/s | 22 MB 00:01 2024-07-26T08:19:39.635 INFO:teuthology.orchestra.run.smithi116.stdout:(158/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 8.7 MB/s | 1.1 MB 00:00 2024-07-26T08:19:39.653 INFO:teuthology.orchestra.run.smithi171.stdout:(82/160): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-07-26T08:19:39.679 INFO:teuthology.orchestra.run.smithi171.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-07-26T08:19:39.704 INFO:teuthology.orchestra.run.smithi171.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-07-26T08:19:39.729 INFO:teuthology.orchestra.run.smithi171.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-07-26T08:19:39.755 INFO:teuthology.orchestra.run.smithi171.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-07-26T08:19:39.780 INFO:teuthology.orchestra.run.smithi171.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-07-26T08:19:39.805 INFO:teuthology.orchestra.run.smithi171.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-07-26T08:19:39.830 INFO:teuthology.orchestra.run.smithi171.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-07-26T08:19:39.856 INFO:teuthology.orchestra.run.smithi171.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-07-26T08:19:39.881 INFO:teuthology.orchestra.run.smithi171.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-07-26T08:19:39.965 INFO:teuthology.orchestra.run.smithi171.stdout:(92/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-07-26T08:19:39.990 INFO:teuthology.orchestra.run.smithi171.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-07-26T08:19:40.016 INFO:teuthology.orchestra.run.smithi171.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-07-26T08:19:40.040 INFO:teuthology.orchestra.run.smithi171.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-07-26T08:19:40.066 INFO:teuthology.orchestra.run.smithi171.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-07-26T08:19:40.091 INFO:teuthology.orchestra.run.smithi171.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-07-26T08:19:40.116 INFO:teuthology.orchestra.run.smithi171.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-07-26T08:19:40.141 INFO:teuthology.orchestra.run.smithi171.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-07-26T08:19:40.167 INFO:teuthology.orchestra.run.smithi171.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-07-26T08:19:40.192 INFO:teuthology.orchestra.run.smithi171.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-07-26T08:19:40.217 INFO:teuthology.orchestra.run.smithi171.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-07-26T08:19:40.393 INFO:teuthology.orchestra.run.smithi116.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-07-26T08:19:40.435 INFO:teuthology.orchestra.run.smithi116.stdout:(160/160): NetworkManager-libnm-1.48.2-2.el9.x8 925 kB/s | 1.8 MB 00:01 2024-07-26T08:19:40.441 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-07-26T08:19:40.441 INFO:teuthology.orchestra.run.smithi116.stdout:Total 20 MB/s | 233 MB 00:11 2024-07-26T08:19:40.476 INFO:teuthology.orchestra.run.smithi171.stdout:(103/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-07-26T08:19:40.493 INFO:teuthology.orchestra.run.smithi171.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-07-26T08:19:40.593 INFO:teuthology.orchestra.run.smithi171.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-07-26T08:19:40.610 INFO:teuthology.orchestra.run.smithi171.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-07-26T08:19:40.635 INFO:teuthology.orchestra.run.smithi171.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.1 MB/s | 309 kB 00:00 2024-07-26T08:19:40.661 INFO:teuthology.orchestra.run.smithi171.stdout:(108/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-07-26T08:19:40.711 INFO:teuthology.orchestra.run.smithi171.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-07-26T08:19:40.736 INFO:teuthology.orchestra.run.smithi171.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-07-26T08:19:40.762 INFO:teuthology.orchestra.run.smithi171.stdout:(111/160): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-07-26T08:19:40.787 INFO:teuthology.orchestra.run.smithi171.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-07-26T08:19:40.829 INFO:teuthology.orchestra.run.smithi171.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-07-26T08:19:40.854 INFO:teuthology.orchestra.run.smithi171.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T08:19:40.879 INFO:teuthology.orchestra.run.smithi171.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T08:19:40.905 INFO:teuthology.orchestra.run.smithi171.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 569 kB/s | 14 kB 00:00 2024-07-26T08:19:40.938 INFO:teuthology.orchestra.run.smithi171.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-07-26T08:19:40.972 INFO:teuthology.orchestra.run.smithi171.stdout:(118/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-07-26T08:19:41.006 INFO:teuthology.orchestra.run.smithi171.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-07-26T08:19:41.040 INFO:teuthology.orchestra.run.smithi171.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-07-26T08:19:41.064 INFO:teuthology.orchestra.run.smithi171.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.6 MB/s | 216 kB 00:00 2024-07-26T08:19:41.140 INFO:teuthology.orchestra.run.smithi171.stdout:(122/160): libarrow-9.0.0-11.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-07-26T08:19:41.165 INFO:teuthology.orchestra.run.smithi171.stdout:(123/160): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-07-26T08:19:41.190 INFO:teuthology.orchestra.run.smithi171.stdout:(124/160): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-07-26T08:19:41.216 INFO:teuthology.orchestra.run.smithi171.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-07-26T08:19:41.241 INFO:teuthology.orchestra.run.smithi171.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-07-26T08:19:41.266 INFO:teuthology.orchestra.run.smithi171.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-07-26T08:19:41.291 INFO:teuthology.orchestra.run.smithi171.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-07-26T08:19:41.317 INFO:teuthology.orchestra.run.smithi171.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T08:19:41.342 INFO:teuthology.orchestra.run.smithi171.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T08:19:41.367 INFO:teuthology.orchestra.run.smithi171.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-07-26T08:19:41.409 INFO:teuthology.orchestra.run.smithi171.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-07-26T08:19:41.434 INFO:teuthology.orchestra.run.smithi171.stdout:(133/160): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-07-26T08:19:41.460 INFO:teuthology.orchestra.run.smithi171.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-07-26T08:19:41.485 INFO:teuthology.orchestra.run.smithi171.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T08:19:41.510 INFO:teuthology.orchestra.run.smithi171.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-07-26T08:19:41.536 INFO:teuthology.orchestra.run.smithi171.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-07-26T08:19:41.561 INFO:teuthology.orchestra.run.smithi171.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-26T08:19:41.586 INFO:teuthology.orchestra.run.smithi171.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-07-26T08:19:41.612 INFO:teuthology.orchestra.run.smithi171.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-07-26T08:19:41.637 INFO:teuthology.orchestra.run.smithi171.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T08:19:41.662 INFO:teuthology.orchestra.run.smithi171.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-07-26T08:19:41.686 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-07-26T08:19:41.713 INFO:teuthology.orchestra.run.smithi171.stdout:(143/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.0 MB/s | 2.0 MB 00:00 2024-07-26T08:19:41.738 INFO:teuthology.orchestra.run.smithi171.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-07-26T08:19:41.763 INFO:teuthology.orchestra.run.smithi171.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-07-26T08:19:41.788 INFO:teuthology.orchestra.run.smithi171.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T08:19:41.814 INFO:teuthology.orchestra.run.smithi171.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-07-26T08:19:41.856 INFO:teuthology.orchestra.run.smithi171.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-07-26T08:19:41.877 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-07-26T08:19:41.877 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-07-26T08:19:42.231 INFO:teuthology.orchestra.run.smithi171.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 7.5 MB/s | 3.1 MB 00:00 2024-07-26T08:19:42.360 INFO:teuthology.orchestra.run.smithi171.stdout:(150/160): librados2-19.0.0-5320.ge8f88bb1.el9. 5.9 MB/s | 3.4 MB 00:00 2024-07-26T08:19:42.623 INFO:teuthology.orchestra.run.smithi171.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 155 kB/s | 41 kB 00:00 2024-07-26T08:19:42.790 INFO:teuthology.orchestra.run.smithi171.stdout:(152/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-07-26T08:19:42.890 INFO:teuthology.orchestra.run.smithi171.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 367 kB/s | 37 kB 00:00 2024-07-26T08:19:42.974 INFO:teuthology.orchestra.run.smithi171.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 705 kB/s | 247 kB 00:00 2024-07-26T08:19:42.999 INFO:teuthology.orchestra.run.smithi171.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-07-26T08:19:43.100 INFO:teuthology.orchestra.run.smithi171.stdout:(156/160): NetworkManager-libnm-1.48.2-2.el9.x8 2.1 MB/s | 1.8 MB 00:00 2024-07-26T08:19:43.141 INFO:teuthology.orchestra.run.smithi171.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 9.6 MB/s | 1.4 MB 00:00 2024-07-26T08:19:43.166 INFO:teuthology.orchestra.run.smithi171.stdout:(158/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.3 MB/s | 260 kB 00:00 2024-07-26T08:19:43.333 INFO:teuthology.orchestra.run.smithi171.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-07-26T08:19:43.731 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-07-26T08:19:43.731 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-07-26T08:19:44.633 INFO:teuthology.orchestra.run.smithi171.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 742 kB/s | 1.1 MB 00:01 2024-07-26T08:19:44.639 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-07-26T08:19:44.640 INFO:teuthology.orchestra.run.smithi171.stdout:Total 16 MB/s | 233 MB 00:14 2024-07-26T08:19:45.860 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-07-26T08:19:46.053 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-07-26T08:19:46.053 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-07-26T08:19:46.151 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-07-26T08:19:47.924 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-07-26T08:19:47.925 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-07-26T08:19:48.910 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T08:19:48.912 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T08:19:48.933 INFO:teuthology.orchestra.run.smithi044.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T08:19:48.934 INFO:teuthology.orchestra.run.smithi044.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T08:19:48.936 INFO:teuthology.orchestra.run.smithi044.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T08:19:48.937 INFO:teuthology.orchestra.run.smithi044.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T08:19:48.940 INFO:teuthology.orchestra.run.smithi044.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T08:19:48.943 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T08:19:48.945 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T08:19:49.024 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-07-26T08:19:49.043 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-07-26T08:19:49.043 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-07-26T08:19:49.043 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T08:19:49.044 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T08:19:49.045 INFO:teuthology.orchestra.run.smithi044.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T08:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T08:19:49.047 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T08:19:49.048 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T08:19:49.049 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T08:19:49.050 INFO:teuthology.orchestra.run.smithi044.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T08:19:49.051 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T08:19:49.052 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T08:19:49.053 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T08:19:49.054 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T08:19:49.055 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T08:19:49.056 INFO:teuthology.orchestra.run.smithi044.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout:Install 148 Packages 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 12 Packages 2024-07-26T08:19:49.057 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:19:49.058 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 233 M 2024-07-26T08:19:49.058 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-07-26T08:19:49.736 INFO:teuthology.orchestra.run.smithi044.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-07-26T08:19:50.161 INFO:teuthology.orchestra.run.smithi044.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.0 MB/s | 872 kB 00:00 2024-07-26T08:19:50.187 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-07-26T08:19:50.253 INFO:teuthology.orchestra.run.smithi044.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.6 MB/s | 148 kB 00:00 2024-07-26T08:19:50.255 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T08:19:50.302 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T08:19:50.356 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T08:19:50.390 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T08:19:50.415 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T08:19:50.457 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T08:19:50.554 INFO:teuthology.orchestra.run.smithi044.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 5.0 MB/s | 5.5 MB 00:01 2024-07-26T08:19:50.791 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T08:19:50.796 INFO:teuthology.orchestra.run.smithi044.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 4.2 MB/s | 2.3 MB 00:00 2024-07-26T08:19:50.879 INFO:teuthology.orchestra.run.smithi044.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-07-26T08:19:50.900 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:19:51.026 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:19:51.051 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:19:51.146 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:19:51.163 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T08:19:51.196 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T08:19:51.388 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T08:19:51.437 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T08:19:51.464 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:19:51.552 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:19:51.571 INFO:teuthology.orchestra.run.smithi044.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 6.3 MB/s | 4.9 MB 00:00 2024-07-26T08:19:51.582 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T08:19:51.636 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T08:19:51.658 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T08:19:51.677 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T08:19:51.859 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T08:19:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T08:19:52.131 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T08:19:52.282 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T08:19:52.335 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T08:19:52.435 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:19:52.519 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:19:52.721 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:19:52.831 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T08:19:52.857 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T08:19:52.894 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T08:19:53.110 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T08:19:53.121 INFO:teuthology.orchestra.run.smithi044.stdout:(8/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el9 6.7 MB/s | 10 MB 00:01 2024-07-26T08:19:53.171 INFO:teuthology.orchestra.run.smithi044.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 516 kB/s | 25 kB 00:00 2024-07-26T08:19:53.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T08:19:53.309 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T08:19:53.353 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T08:19:53.385 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T08:19:53.422 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T08:19:53.447 INFO:teuthology.orchestra.run.smithi044.stdout:(10/160): ceph-common-19.0.0-5320.ge8f88bb1.el9 5.0 MB/s | 20 MB 00:03 2024-07-26T08:19:53.500 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T08:19:53.553 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T08:19:53.583 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T08:19:53.606 INFO:teuthology.orchestra.run.smithi044.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 1.4 MB/s | 225 kB 00:00 2024-07-26T08:19:53.627 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T08:19:53.689 INFO:teuthology.orchestra.run.smithi044.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 327 kB/s | 27 kB 00:00 2024-07-26T08:19:53.704 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T08:19:53.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T08:19:53.781 INFO:teuthology.orchestra.run.smithi044.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 349 kB/s | 32 kB 00:00 2024-07-26T08:19:53.783 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T08:19:53.809 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T08:19:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T08:19:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T08:19:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T08:19:54.006 INFO:teuthology.orchestra.run.smithi044.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 3.3 MB/s | 765 kB 00:00 2024-07-26T08:19:54.065 INFO:teuthology.orchestra.run.smithi044.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.7 MB/s | 158 kB 00:00 2024-07-26T08:19:54.124 INFO:teuthology.orchestra.run.smithi044.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 2.2 MB/s | 127 kB 00:00 2024-07-26T08:19:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T08:19:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T08:19:54.215 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-07-26T08:19:54.287 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T08:19:54.326 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T08:19:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T08:19:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T08:19:54.416 INFO:teuthology.orchestra.run.smithi044.stdout:(17/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 5.0 MB/s | 18 MB 00:03 2024-07-26T08:19:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T08:19:54.449 INFO:teuthology.orchestra.run.smithi044.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 1.5 MB/s | 503 kB 00:00 2024-07-26T08:19:54.481 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T08:19:54.525 INFO:teuthology.orchestra.run.smithi044.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 604 kB/s | 45 kB 00:00 2024-07-26T08:19:54.608 INFO:teuthology.orchestra.run.smithi044.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.6 MB/s | 138 kB 00:00 2024-07-26T08:19:54.708 INFO:teuthology.orchestra.run.smithi044.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 1.8 MB/s | 188 kB 00:00 2024-07-26T08:19:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T08:19:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T08:19:54.822 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T08:19:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T08:19:54.867 INFO:teuthology.orchestra.run.smithi044.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 372 kB 00:00 2024-07-26T08:19:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T08:19:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T08:19:54.948 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:19:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T08:19:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T08:19:55.034 INFO:teuthology.orchestra.run.smithi044.stdout:(23/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 2.1 MB/s | 355 kB 00:00 2024-07-26T08:19:55.075 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:19:55.099 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:19:55.135 INFO:teuthology.orchestra.run.smithi044.stdout:(24/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.1 MB/s | 116 kB 00:00 2024-07-26T08:19:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T08:19:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T08:19:55.227 INFO:teuthology.orchestra.run.smithi044.stdout:(25/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 952 kB/s | 86 kB 00:00 2024-07-26T08:19:55.302 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T08:19:55.336 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:19:55.361 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T08:19:55.367 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T08:19:55.394 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T08:19:55.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T08:19:55.470 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T08:19:55.507 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T08:19:55.552 INFO:teuthology.orchestra.run.smithi044.stdout:(26/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 4.6 MB/s | 5.3 MB 00:01 2024-07-26T08:19:55.553 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T08:19:55.584 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T08:19:55.589 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T08:19:55.626 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T08:19:55.633 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T08:19:55.636 INFO:teuthology.orchestra.run.smithi044.stdout:(27/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 7.6 MB/s | 3.1 MB 00:00 2024-07-26T08:19:55.654 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:19:55.669 INFO:teuthology.orchestra.run.smithi044.stdout:(28/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 1.4 MB/s | 170 kB 00:00 2024-07-26T08:19:55.675 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T08:19:55.703 INFO:teuthology.orchestra.run.smithi044.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 525 kB/s | 34 kB 00:00 2024-07-26T08:19:55.743 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:19:55.753 INFO:teuthology.orchestra.run.smithi044.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.8 MB/s | 149 kB 00:00 2024-07-26T08:19:55.772 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T08:19:55.812 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T08:19:55.818 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T08:19:55.840 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T08:19:55.852 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T08:19:55.860 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T08:19:55.893 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T08:19:55.991 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T08:19:56.047 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T08:19:56.139 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T08:19:56.211 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T08:19:56.252 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T08:19:56.285 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T08:19:56.309 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T08:19:56.310 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T08:19:56.399 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T08:19:56.447 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T08:19:56.507 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T08:19:56.530 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T08:19:56.574 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T08:19:56.602 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T08:19:56.609 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:19:56.637 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T08:19:56.676 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T08:19:56.708 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T08:19:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T08:19:56.744 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:19:56.771 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T08:19:56.805 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T08:19:56.847 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T08:19:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T08:19:56.938 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:19:56.968 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T08:19:57.020 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T08:19:57.049 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T08:19:57.051 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T08:19:57.078 INFO:teuthology.orchestra.run.smithi044.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-07-26T08:19:57.079 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T08:19:57.114 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T08:19:57.154 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T08:19:57.188 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T08:19:57.202 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T08:19:57.232 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T08:19:57.237 INFO:teuthology.orchestra.run.smithi044.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 1.8 MB/s | 284 kB 00:00 2024-07-26T08:19:57.264 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T08:19:57.308 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T08:19:57.329 INFO:teuthology.orchestra.run.smithi044.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 542 kB/s | 49 kB 00:00 2024-07-26T08:19:57.340 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T08:19:57.368 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T08:19:57.421 INFO:teuthology.orchestra.run.smithi044.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 180 kB/s | 16 kB 00:00 2024-07-26T08:19:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T08:19:57.491 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T08:19:57.521 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T08:19:57.576 INFO:teuthology.orchestra.run.smithi044.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 2.5 MB/s | 274 kB 00:00 2024-07-26T08:19:57.576 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T08:19:57.641 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T08:19:57.675 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T08:19:57.720 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T08:19:57.747 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T08:19:57.780 INFO:teuthology.orchestra.run.smithi044.stdout:(36/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 3.0 MB/s | 771 kB 00:00 2024-07-26T08:19:57.817 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T08:19:57.855 INFO:teuthology.orchestra.run.smithi044.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 246 kB/s | 18 kB 00:00 2024-07-26T08:19:57.861 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T08:19:57.922 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T08:19:57.937 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T08:19:57.964 INFO:teuthology.orchestra.run.smithi044.stdout:(38/160): fuse-2.9.9-16.el9.x86_64.rpm 744 kB/s | 80 kB 00:00 2024-07-26T08:19:57.966 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T08:19:57.999 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T08:19:58.072 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T08:19:58.081 INFO:teuthology.orchestra.run.smithi044.stdout:(39/160): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 803 kB 00:00 2024-07-26T08:19:58.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T08:19:58.122 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T08:19:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T08:19:58.200 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T08:19:58.273 INFO:teuthology.orchestra.run.smithi044.stdout:(40/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 5.4 MB/s | 14 MB 00:02 2024-07-26T08:19:58.287 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T08:19:58.306 INFO:teuthology.orchestra.run.smithi044.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 871 kB/s | 195 kB 00:00 2024-07-26T08:19:58.327 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T08:19:58.390 INFO:teuthology.orchestra.run.smithi044.stdout:(42/160): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-07-26T08:19:58.441 INFO:teuthology.orchestra.run.smithi044.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-07-26T08:19:58.474 INFO:teuthology.orchestra.run.smithi044.stdout:(44/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-07-26T08:19:58.504 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T08:19:58.508 INFO:teuthology.orchestra.run.smithi044.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-07-26T08:19:58.541 INFO:teuthology.orchestra.run.smithi044.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-07-26T08:19:58.544 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T08:19:58.575 INFO:teuthology.orchestra.run.smithi044.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-07-26T08:19:58.627 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T08:19:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T08:19:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T08:19:58.733 INFO:teuthology.orchestra.run.smithi044.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-07-26T08:19:58.755 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T08:19:58.791 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T08:19:58.800 INFO:teuthology.orchestra.run.smithi044.stdout:(49/160): bison-3.7.4-5.el9.x86_64.rpm 4.1 MB/s | 943 kB 00:00 2024-07-26T08:19:58.826 INFO:teuthology.orchestra.run.smithi044.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-07-26T08:19:58.843 INFO:teuthology.orchestra.run.smithi044.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 525 kB/s | 22 kB 00:00 2024-07-26T08:19:58.849 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T08:19:58.877 INFO:teuthology.orchestra.run.smithi044.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 6.2 MB/s | 313 kB 00:00 2024-07-26T08:19:58.895 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T08:19:58.927 INFO:teuthology.orchestra.run.smithi044.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 395 kB/s | 33 kB 00:00 2024-07-26T08:19:58.937 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T08:19:58.961 INFO:teuthology.orchestra.run.smithi044.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 562 kB/s | 18 kB 00:00 2024-07-26T08:19:58.995 INFO:teuthology.orchestra.run.smithi044.stdout:(55/160): libaio-devel-0.3.111-13.el9.x86_64.rp 342 kB/s | 11 kB 00:00 2024-07-26T08:19:59.070 INFO:teuthology.orchestra.run.smithi044.stdout:(56/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-07-26T08:19:59.083 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T08:19:59.103 INFO:teuthology.orchestra.run.smithi044.stdout:(57/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.4 MB/s | 45 kB 00:00 2024-07-26T08:19:59.148 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T08:19:59.239 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T08:19:59.270 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T08:19:59.304 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T08:19:59.337 INFO:teuthology.orchestra.run.smithi044.stdout:(58/160): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-07-26T08:19:59.377 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T08:19:59.391 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T08:19:59.408 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T08:19:59.444 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T08:19:59.491 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T08:19:59.519 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T08:19:59.537 INFO:teuthology.orchestra.run.smithi044.stdout:(59/160): libtool-2.4.6-46.el9.x86_64.rpm 2.8 MB/s | 581 kB 00:00 2024-07-26T08:19:59.562 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T08:19:59.587 INFO:teuthology.orchestra.run.smithi044.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 517 kB/s | 25 kB 00:00 2024-07-26T08:19:59.604 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T08:19:59.621 INFO:teuthology.orchestra.run.smithi044.stdout:(61/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-07-26T08:19:59.741 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T08:19:59.789 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T08:19:59.805 INFO:teuthology.orchestra.run.smithi044.stdout:(62/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-07-26T08:19:59.830 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T08:19:59.838 INFO:teuthology.orchestra.run.smithi044.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-07-26T08:19:59.872 INFO:teuthology.orchestra.run.smithi044.stdout:(64/160): openblas-0.3.21-2.el9.x86_64.rpm 1.1 MB/s | 35 kB 00:00 2024-07-26T08:19:59.922 INFO:teuthology.orchestra.run.smithi044.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-07-26T08:19:59.929 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T08:19:59.964 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T08:19:59.995 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T08:20:00.021 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T08:20:00.045 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T08:20:00.120 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T08:20:00.247 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T08:20:00.406 INFO:teuthology.orchestra.run.smithi044.stdout:(66/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 6.1 MB/s | 44 MB 00:07 2024-07-26T08:20:00.487 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T08:20:00.515 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T08:20:00.550 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T08:20:00.590 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T08:20:00.611 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T08:20:00.629 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T08:20:00.658 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T08:20:00.693 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T08:20:00.695 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T08:20:00.715 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T08:20:00.726 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T08:20:00.740 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T08:20:00.765 INFO:teuthology.orchestra.run.smithi044.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-07-26T08:20:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T08:20:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T08:20:00.893 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T08:20:00.949 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T08:20:00.980 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T08:20:01.057 INFO:teuthology.orchestra.run.smithi044.stdout:(68/160): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-07-26T08:20:01.065 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T08:20:01.084 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T08:20:01.103 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T08:20:01.138 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T08:20:01.140 INFO:teuthology.orchestra.run.smithi044.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-07-26T08:20:01.197 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T08:20:01.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T08:20:01.257 INFO:teuthology.orchestra.run.smithi044.stdout:(70/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-07-26T08:20:01.261 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T08:20:01.508 INFO:teuthology.orchestra.run.smithi044.stdout:(71/160): openblas-openmp-0.3.21-2.el9.x86_64.r 2.9 MB/s | 4.7 MB 00:01 2024-07-26T08:20:01.566 INFO:teuthology.orchestra.run.smithi044.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-07-26T08:20:01.658 INFO:teuthology.orchestra.run.smithi044.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-07-26T08:20:01.700 INFO:teuthology.orchestra.run.smithi044.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.5 MB/s | 61 kB 00:00 2024-07-26T08:20:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T08:20:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T08:20:01.792 INFO:teuthology.orchestra.run.smithi044.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-07-26T08:20:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T08:20:01.867 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T08:20:01.884 INFO:teuthology.orchestra.run.smithi044.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.7 MB/s | 248 kB 00:00 2024-07-26T08:20:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T08:20:01.919 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T08:20:01.926 INFO:teuthology.orchestra.run.smithi044.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 628 kB/s | 26 kB 00:00 2024-07-26T08:20:01.944 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T08:20:01.968 INFO:teuthology.orchestra.run.smithi044.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 469 kB/s | 19 kB 00:00 2024-07-26T08:20:02.043 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T08:20:02.044 INFO:teuthology.orchestra.run.smithi044.stdout:(79/160): python3-mako-1.1.4-6.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-07-26T08:20:02.094 INFO:teuthology.orchestra.run.smithi044.stdout:(80/160): python3-markupsafe-1.1.1-12.el9.x86_6 699 kB/s | 35 kB 00:00 2024-07-26T08:20:02.099 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T08:20:02.144 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T08:20:02.232 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T08:20:02.280 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T08:20:02.459 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T08:20:02.495 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T08:20:02.572 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T08:20:03.002 INFO:teuthology.orchestra.run.smithi044.stdout:(81/160): postgresql-server-13.14-1.el9.x86_64. 3.1 MB/s | 5.8 MB 00:01 2024-07-26T08:20:03.102 INFO:teuthology.orchestra.run.smithi044.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-07-26T08:20:03.203 INFO:teuthology.orchestra.run.smithi044.stdout:(83/160): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-07-26T08:20:03.219 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T08:20:03.261 INFO:teuthology.orchestra.run.smithi044.stdout:(84/160): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-07-26T08:20:03.295 INFO:teuthology.orchestra.run.smithi044.stdout:(85/160): python3-prettytable-0.7.2-27.el9.noar 1.3 MB/s | 42 kB 00:00 2024-07-26T08:20:03.334 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T08:20:03.471 INFO:teuthology.orchestra.run.smithi044.stdout:(86/160): python3-babel-2.9.1-2.el9.noarch.rpm 3.1 MB/s | 6.0 MB 00:01 2024-07-26T08:20:03.526 INFO:teuthology.orchestra.run.smithi044.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 1.1 MB/s | 269 kB 00:00 2024-07-26T08:20:03.588 INFO:teuthology.orchestra.run.smithi044.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-07-26T08:20:03.621 INFO:teuthology.orchestra.run.smithi044.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-07-26T08:20:03.663 INFO:teuthology.orchestra.run.smithi044.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-07-26T08:20:03.680 INFO:teuthology.orchestra.run.smithi044.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 929 kB/s | 54 kB 00:00 2024-07-26T08:20:03.781 INFO:teuthology.orchestra.run.smithi044.stdout:(92/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.0 MB/s | 5.1 MB 00:01 2024-07-26T08:20:03.798 INFO:teuthology.orchestra.run.smithi044.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-07-26T08:20:03.823 INFO:teuthology.orchestra.run.smithi044.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.1 MB/s | 44 kB 00:00 2024-07-26T08:20:03.865 INFO:teuthology.orchestra.run.smithi044.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 880 kB/s | 36 kB 00:00 2024-07-26T08:20:03.907 INFO:teuthology.orchestra.run.smithi044.stdout:(96/160): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-07-26T08:20:03.932 INFO:teuthology.orchestra.run.smithi044.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 719 kB/s | 47 kB 00:00 2024-07-26T08:20:03.974 INFO:teuthology.orchestra.run.smithi044.stdout:(98/160): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-07-26T08:20:04.033 INFO:teuthology.orchestra.run.smithi044.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-07-26T08:20:04.058 INFO:teuthology.orchestra.run.smithi044.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.1 MB/s | 96 kB 00:00 2024-07-26T08:20:04.100 INFO:teuthology.orchestra.run.smithi044.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 965 kB/s | 64 kB 00:00 2024-07-26T08:20:04.125 INFO:teuthology.orchestra.run.smithi044.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-07-26T08:20:04.169 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:04.184 INFO:teuthology.orchestra.run.smithi044.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-07-26T08:20:04.212 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:04.318 INFO:teuthology.orchestra.run.smithi044.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.0 MB/s | 548 kB 00:00 2024-07-26T08:20:04.360 INFO:teuthology.orchestra.run.smithi044.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 7.4 MB/s | 309 kB 00:00 2024-07-26T08:20:04.385 INFO:teuthology.orchestra.run.smithi044.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 797 kB/s | 19 kB 00:00 2024-07-26T08:20:04.552 INFO:teuthology.orchestra.run.smithi044.stdout:(107/160): libarrow-9.0.0-11.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2024-07-26T08:20:04.567 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T08:20:04.594 INFO:teuthology.orchestra.run.smithi044.stdout:(108/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-07-26T08:20:04.639 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T08:20:04.667 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T08:20:04.694 INFO:teuthology.orchestra.run.smithi044.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-07-26T08:20:04.700 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T08:20:04.728 INFO:teuthology.orchestra.run.smithi044.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 366 kB/s | 49 kB 00:00 2024-07-26T08:20:04.753 INFO:teuthology.orchestra.run.smithi044.stdout:(111/160): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-07-26T08:20:04.787 INFO:teuthology.orchestra.run.smithi044.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-07-26T08:20:04.790 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:04.823 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T08:20:04.845 INFO:teuthology.orchestra.run.smithi044.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 9.4 MB/s | 548 kB 00:00 2024-07-26T08:20:04.871 INFO:teuthology.orchestra.run.smithi044.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T08:20:04.896 INFO:teuthology.orchestra.run.smithi044.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T08:20:04.912 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T08:20:04.912 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T08:20:04.912 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T08:20:04.913 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:04.930 INFO:teuthology.orchestra.run.smithi044.stdout:(116/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 839 kB 00:00 2024-07-26T08:20:04.941 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T08:20:04.955 INFO:teuthology.orchestra.run.smithi044.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 244 kB/s | 14 kB 00:00 2024-07-26T08:20:04.989 INFO:teuthology.orchestra.run.smithi044.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-07-26T08:20:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T08:20:05.031 INFO:teuthology.orchestra.run.smithi044.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-07-26T08:20:05.064 INFO:teuthology.orchestra.run.smithi044.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-07-26T08:20:05.080 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T08:20:05.098 INFO:teuthology.orchestra.run.smithi044.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.2 MB/s | 216 kB 00:00 2024-07-26T08:20:05.132 INFO:teuthology.orchestra.run.smithi044.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T08:20:05.149 INFO:teuthology.orchestra.run.smithi044.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 671 kB/s | 11 kB 00:00 2024-07-26T08:20:05.177 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T08:20:05.210 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T08:20:05.246 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T08:20:05.291 INFO:teuthology.orchestra.run.smithi044.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 127 kB/s | 18 kB 00:00 2024-07-26T08:20:05.316 INFO:teuthology.orchestra.run.smithi044.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 998 kB/s | 23 kB 00:00 2024-07-26T08:20:05.350 INFO:teuthology.orchestra.run.smithi044.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 596 kB/s | 19 kB 00:00 2024-07-26T08:20:05.375 INFO:teuthology.orchestra.run.smithi044.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 817 kB/s | 20 kB 00:00 2024-07-26T08:20:05.476 INFO:teuthology.orchestra.run.smithi044.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-07-26T08:20:05.501 INFO:teuthology.orchestra.run.smithi044.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T08:20:05.542 INFO:teuthology.orchestra.run.smithi044.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-07-26T08:20:05.586 INFO:teuthology.orchestra.run.smithi044.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-07-26T08:20:05.627 INFO:teuthology.orchestra.run.smithi044.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 6.5 MB/s | 272 kB 00:00 2024-07-26T08:20:05.694 INFO:teuthology.orchestra.run.smithi044.stdout:(133/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.2 MB/s | 2.0 MB 00:00 2024-07-26T08:20:05.711 INFO:teuthology.orchestra.run.smithi044.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-07-26T08:20:05.737 INFO:teuthology.orchestra.run.smithi044.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T08:20:05.751 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T08:20:05.762 INFO:teuthology.orchestra.run.smithi044.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-07-26T08:20:05.787 INFO:teuthology.orchestra.run.smithi044.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 622 kB/s | 31 kB 00:00 2024-07-26T08:20:05.800 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T08:20:05.872 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T08:20:05.918 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T08:20:05.962 INFO:teuthology.orchestra.run.smithi044.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 339 kB/s | 59 kB 00:00 2024-07-26T08:20:05.988 INFO:teuthology.orchestra.run.smithi044.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-07-26T08:20:06.021 INFO:teuthology.orchestra.run.smithi044.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 730 kB/s | 188 kB 00:00 2024-07-26T08:20:06.055 INFO:teuthology.orchestra.run.smithi044.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 771 kB/s | 51 kB 00:00 2024-07-26T08:20:06.080 INFO:teuthology.orchestra.run.smithi044.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 3.7 MB/s | 90 kB 00:00 2024-07-26T08:20:06.114 INFO:teuthology.orchestra.run.smithi044.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-07-26T08:20:06.148 INFO:teuthology.orchestra.run.smithi044.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 614 kB/s | 20 kB 00:00 2024-07-26T08:20:06.190 INFO:teuthology.orchestra.run.smithi044.stdout:(145/160): re2-20211101-3.el9.x86_64.rpm 4.6 MB/s | 192 kB 00:00 2024-07-26T08:20:06.223 INFO:teuthology.orchestra.run.smithi044.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-07-26T08:20:06.315 INFO:teuthology.orchestra.run.smithi044.stdout:(147/160): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-07-26T08:20:06.666 INFO:teuthology.orchestra.run.smithi044.stdout:(148/160): librados2-19.0.0-5320.ge8f88bb1.el9. 7.6 MB/s | 3.4 MB 00:00 2024-07-26T08:20:06.899 INFO:teuthology.orchestra.run.smithi044.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 5.4 MB/s | 3.1 MB 00:00 2024-07-26T08:20:06.999 INFO:teuthology.orchestra.run.smithi044.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 7.0 MB/s | 2.3 MB 00:00 2024-07-26T08:20:07.042 INFO:teuthology.orchestra.run.smithi044.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 981 kB/s | 41 kB 00:00 2024-07-26T08:20:07.092 INFO:teuthology.orchestra.run.smithi044.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 4.9 MB/s | 247 kB 00:00 2024-07-26T08:20:07.151 INFO:teuthology.orchestra.run.smithi044.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 7.2 MB/s | 1.8 MB 00:00 2024-07-26T08:20:07.176 INFO:teuthology.orchestra.run.smithi044.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 442 kB/s | 37 kB 00:00 2024-07-26T08:20:07.210 INFO:teuthology.orchestra.run.smithi044.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 3.5 MB/s | 207 kB 00:00 2024-07-26T08:20:07.243 INFO:teuthology.orchestra.run.smithi044.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 3.8 MB/s | 260 kB 00:00 2024-07-26T08:20:07.310 INFO:teuthology.orchestra.run.smithi044.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-07-26T08:20:07.410 INFO:teuthology.orchestra.run.smithi044.stdout:(158/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-07-26T08:20:07.461 INFO:teuthology.orchestra.run.smithi044.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 7.2 MB/s | 1.1 MB 00:00 2024-07-26T08:20:08.194 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:08.808 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:08.845 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T08:20:08.928 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T08:20:08.928 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T08:20:08.929 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T08:20:08.929 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:08.955 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T08:20:09.561 INFO:teuthology.orchestra.run.smithi044.stdout:(160/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.8 MB/s | 16 MB 00:05 2024-07-26T08:20:09.572 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-07-26T08:20:09.573 INFO:teuthology.orchestra.run.smithi044.stdout:Total 11 MB/s | 233 MB 00:20 2024-07-26T08:20:13.026 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-07-26T08:20:13.336 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-07-26T08:20:13.336 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-07-26T08:20:16.186 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-07-26T08:20:16.187 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-07-26T08:20:16.464 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:16.631 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T08:20:16.646 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T08:20:16.646 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:16.646 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T08:20:16.647 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:18.242 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T08:20:18.267 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T08:20:18.268 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:18.268 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T08:20:18.268 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T08:20:18.268 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T08:20:18.268 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:18.492 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T08:20:18.515 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T08:20:18.516 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:18.516 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T08:20:18.516 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T08:20:18.516 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T08:20:18.516 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:18.976 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T08:20:19.000 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T08:20:19.000 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:19.000 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T08:20:19.000 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T08:20:19.001 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T08:20:19.001 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:19.106 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T08:20:19.163 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T08:20:19.227 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T08:20:19.414 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T08:20:19.460 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T08:20:19.538 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T08:20:19.651 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T08:20:19.800 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T08:20:19.835 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T08:20:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T08:20:19.895 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T08:20:20.338 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T08:20:20.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T08:20:20.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T08:20:20.466 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T08:20:20.491 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T08:20:20.900 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /root 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-07-26T08:20:20.901 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:21.065 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T08:20:21.080 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T08:20:21.080 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:21.080 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T08:20:21.080 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:22.651 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T08:20:22.676 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T08:20:22.677 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:22.677 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T08:20:22.677 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T08:20:22.677 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T08:20:22.677 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:22.912 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T08:20:22.934 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T08:20:22.935 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:22.935 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T08:20:22.935 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T08:20:22.935 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T08:20:22.935 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:23.115 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-07-26T08:20:23.219 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T08:20:23.291 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T08:20:23.392 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T08:20:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T08:20:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T08:20:23.428 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:23.428 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T08:20:23.428 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T08:20:23.428 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T08:20:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:23.547 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T08:20:23.552 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T08:20:23.578 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T08:20:23.609 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T08:20:23.654 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T08:20:23.666 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T08:20:23.683 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T08:20:23.705 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T08:20:23.852 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T08:20:23.899 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T08:20:23.978 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T08:20:24.067 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T08:20:24.135 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T08:20:24.223 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T08:20:24.265 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:20:24.267 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T08:20:24.307 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T08:20:24.360 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T08:20:24.460 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:20:24.488 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:20:24.626 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:20:24.671 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T08:20:24.706 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T08:20:24.787 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T08:20:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T08:20:24.819 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T08:20:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T08:20:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T08:20:24.862 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T08:20:24.923 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T08:20:24.959 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T08:20:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T08:20:25.024 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T08:20:25.047 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T08:20:25.047 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:25.047 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T08:20:25.048 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T08:20:25.048 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T08:20:25.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:25.092 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T08:20:25.103 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T08:20:25.187 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T08:20:25.201 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T08:20:25.234 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T08:20:25.248 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T08:20:25.259 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:20:25.425 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:20:25.452 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T08:20:25.530 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T08:20:25.561 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T08:20:25.624 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T08:20:25.907 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T08:20:26.184 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T08:20:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T08:20:26.252 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T08:20:26.275 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T08:20:26.276 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:26.276 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T08:20:26.276 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T08:20:26.276 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T08:20:26.276 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:26.320 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T08:20:26.341 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T08:20:26.342 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:26.342 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T08:20:26.342 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:26.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T08:20:26.409 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T08:20:26.409 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:26.410 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T08:20:26.410 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T08:20:26.410 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T08:20:26.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T08:20:26.729 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T08:20:26.745 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T08:20:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:26.834 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T08:20:27.044 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:20:27.123 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:20:27.434 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T08:20:27.598 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T08:20:27.637 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T08:20:27.690 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T08:20:27.821 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T08:20:27.883 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T08:20:27.928 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T08:20:27.991 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T08:20:28.039 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T08:20:28.078 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T08:20:28.147 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T08:20:28.200 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T08:20:28.225 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T08:20:28.273 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T08:20:28.314 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T08:20:28.373 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T08:20:28.485 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T08:20:28.517 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T08:20:28.571 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T08:20:28.599 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T08:20:28.719 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T08:20:28.779 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T08:20:28.893 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T08:20:29.083 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T08:20:29.149 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T08:20:29.183 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T08:20:29.217 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T08:20:29.330 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T08:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T08:20:29.528 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T08:20:29.552 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T08:20:29.552 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:29.552 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T08:20:29.552 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T08:20:29.552 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T08:20:29.552 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:29.596 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T08:20:29.607 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T08:20:29.706 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T08:20:29.761 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T08:20:29.883 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T08:20:29.993 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T08:20:30.049 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T08:20:30.087 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T08:20:30.153 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T08:20:30.209 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T08:20:30.274 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T08:20:30.494 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T08:20:30.577 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T08:20:30.715 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T08:20:30.718 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T08:20:30.749 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T08:20:30.772 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T08:20:30.772 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:30.772 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T08:20:30.772 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T08:20:30.772 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T08:20:30.772 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:30.791 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T08:20:30.817 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T08:20:30.839 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T08:20:30.839 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:30.839 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T08:20:30.839 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:30.880 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T08:20:30.884 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T08:20:30.907 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T08:20:30.907 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:30.907 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T08:20:30.907 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T08:20:30.907 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T08:20:30.907 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:30.919 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T08:20:30.957 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T08:20:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T08:20:31.066 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T08:20:31.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T08:20:31.130 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T08:20:31.166 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T08:20:31.207 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T08:20:31.214 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T08:20:31.227 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T08:20:31.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T08:20:31.249 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T08:20:31.249 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:31.249 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T08:20:31.249 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T08:20:31.249 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T08:20:31.250 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:31.266 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T08:20:31.431 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T08:20:31.440 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T08:20:31.485 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T08:20:31.486 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T08:20:31.514 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T08:20:31.541 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T08:20:31.547 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T08:20:31.577 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T08:20:31.623 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T08:20:31.682 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T08:20:31.725 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T08:20:31.794 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T08:20:31.835 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T08:20:31.867 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T08:20:32.074 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T08:20:32.217 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:20:32.261 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T08:20:32.264 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:20:32.283 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:20:32.329 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T08:20:32.370 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T08:20:32.417 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T08:20:32.476 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T08:20:32.527 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T08:20:32.577 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T08:20:32.622 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T08:20:32.674 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T08:20:32.751 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T08:20:32.797 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T08:20:32.943 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T08:20:33.004 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T08:20:33.045 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T08:20:33.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T08:20:33.196 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T08:20:33.196 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T08:20:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T08:20:33.258 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T08:20:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T08:20:33.366 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T08:20:33.395 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T08:20:33.416 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T08:20:33.443 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T08:20:33.443 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T08:20:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T08:20:33.560 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T08:20:33.560 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T08:20:33.647 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T08:20:33.672 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T08:20:33.677 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:20:33.677 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:20:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:20:33.857 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T08:20:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T08:20:33.966 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T08:20:33.992 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T08:20:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:20:34.270 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T08:20:34.332 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T08:20:34.366 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T08:20:34.494 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T08:20:34.566 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T08:20:34.622 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T08:20:34.746 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T08:20:34.793 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T08:20:35.039 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T08:20:35.073 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T08:20:35.189 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T08:20:35.632 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T08:20:35.682 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T08:20:35.723 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T08:20:35.755 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T08:20:35.787 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T08:20:35.971 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T08:20:36.025 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T08:20:36.054 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T08:20:36.057 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T08:20:36.098 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T08:20:36.128 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T08:20:36.166 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T08:20:36.201 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T08:20:36.764 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:20:36.810 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:20:36.830 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:20:37.678 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T08:20:37.771 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T08:20:37.820 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T08:20:37.880 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T08:20:37.903 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T08:20:37.935 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T08:20:37.959 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T08:20:37.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T08:20:37.985 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T08:20:38.009 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T08:20:38.020 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T08:20:38.020 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T08:20:38.043 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T08:20:38.113 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T08:20:38.169 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T08:20:38.169 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T08:20:38.256 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T08:20:38.281 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T08:20:38.286 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:20:38.286 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:20:38.467 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:20:38.467 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T08:20:38.475 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T08:20:38.517 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T08:20:38.551 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T08:20:38.587 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T08:20:38.613 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T08:20:38.646 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T08:20:38.685 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:20:38.702 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T08:20:38.769 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T08:20:39.366 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T08:20:39.396 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T08:20:39.498 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T08:20:39.531 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T08:20:40.152 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:20:40.152 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T08:20:40.152 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T08:20:40.153 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T08:20:40.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T08:20:40.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T08:20:40.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T08:20:40.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T08:20:40.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T08:20:40.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T08:20:40.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T08:20:40.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T08:20:40.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T08:20:40.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T08:20:40.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T08:20:40.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T08:20:40.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T08:20:40.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T08:20:40.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T08:20:40.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T08:20:40.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T08:20:40.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T08:20:40.168 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T08:20:40.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T08:20:40.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T08:20:40.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T08:20:40.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T08:20:40.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T08:20:40.169 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T08:20:40.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T08:20:42.294 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:42.336 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T08:20:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:43.015 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.016 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T08:20:43.017 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T08:20:43.018 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T08:20:43.019 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T08:20:43.020 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T08:20:43.021 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T08:20:43.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T08:20:43.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.024 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-07-26T08:20:43.025 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-07-26T08:20:43.031 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T08:20:43.073 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T08:20:43.234 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T08:20:43.234 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T08:20:43.234 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T08:20:43.234 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:20:43.274 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T08:20:43.463 DEBUG:teuthology.parallel:result is None 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:20:45.343 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T08:20:45.344 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T08:20:45.345 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T08:20:45.345 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T08:20:45.345 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T08:20:45.345 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T08:20:45.345 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T08:20:45.345 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T08:20:45.357 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T08:20:45.357 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T08:20:45.357 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T08:20:45.357 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T08:20:45.357 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T08:20:45.357 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T08:20:45.358 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T08:20:45.359 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T08:20:45.360 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T08:20:45.361 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T08:20:45.362 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T08:20:45.363 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T08:20:45.364 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T08:20:45.365 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T08:20:45.366 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T08:20:45.367 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T08:20:45.368 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:47.963 INFO:teuthology.orchestra.run.smithi171.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T08:20:47.964 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.965 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.966 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T08:20:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.968 INFO:teuthology.orchestra.run.smithi171.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T08:20:47.969 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T08:20:47.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T08:20:47.972 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T08:20:47.973 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.974 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T08:20:47.975 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T08:20:47.976 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T08:20:47.977 INFO:teuthology.orchestra.run.smithi171.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T08:20:47.977 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:20:47.977 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-07-26T08:20:48.420 DEBUG:teuthology.parallel:result is None 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-07-26T08:20:56.594 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:20:56.839 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T08:20:56.892 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T08:20:56.892 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:56.892 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T08:20:56.892 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:20:58.708 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T08:20:58.795 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T08:20:58.795 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:58.795 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T08:20:58.796 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T08:20:58.796 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T08:20:58.796 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:20:59.058 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T08:20:59.144 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T08:20:59.144 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:59.144 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T08:20:59.144 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T08:20:59.145 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T08:20:59.145 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:20:59.675 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T08:20:59.748 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T08:20:59.749 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:20:59.749 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T08:20:59.749 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T08:20:59.749 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T08:20:59.749 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:20:59.885 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T08:20:59.958 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T08:21:00.043 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T08:21:00.289 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T08:21:00.347 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T08:21:00.436 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T08:21:00.556 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T08:21:00.735 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T08:21:00.779 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T08:21:00.819 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T08:21:00.861 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T08:21:01.489 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T08:21:01.521 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T08:21:01.573 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T08:21:01.682 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T08:21:01.716 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T08:21:05.623 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T08:21:05.690 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T08:21:06.854 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T08:21:06.897 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T08:21:07.068 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T08:21:07.259 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T08:21:07.291 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T08:21:07.367 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T08:21:07.367 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:21:07.367 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T08:21:07.367 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T08:21:07.367 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T08:21:07.367 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:07.425 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T08:21:07.465 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T08:21:07.615 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T08:21:07.690 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T08:21:08.838 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T08:21:08.878 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T08:21:08.950 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T08:21:08.950 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:21:08.950 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T08:21:08.950 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T08:21:08.951 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T08:21:08.951 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:09.003 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T08:21:09.077 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T08:21:09.077 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:21:09.077 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T08:21:09.077 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:09.146 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T08:21:09.228 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T08:21:09.228 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:21:09.228 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T08:21:09.228 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T08:21:09.228 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T08:21:09.228 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T08:21:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T08:21:09.729 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T08:21:09.729 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T08:21:09.729 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T08:21:09.729 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T08:21:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:14.795 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T08:21:14.858 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T08:21:14.914 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T08:21:14.952 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T08:21:14.991 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T08:21:15.312 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T08:21:15.366 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T08:21:15.401 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T08:21:15.447 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T08:21:15.497 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T08:21:15.557 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T08:21:16.496 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:21:16.546 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:21:16.602 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T08:21:18.055 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T08:21:18.181 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T08:21:18.238 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T08:21:18.381 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T08:21:18.417 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T08:21:18.451 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T08:21:18.485 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T08:21:18.527 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T08:21:18.528 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T08:21:18.659 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T08:21:18.705 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T08:21:18.706 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T08:21:18.843 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T08:21:18.881 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T08:21:18.916 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:21:18.917 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:21:19.255 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T08:21:19.255 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T08:21:19.400 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T08:21:19.445 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T08:21:19.483 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T08:21:19.618 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:21:19.884 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:21:19.884 INFO:teuthology.orchestra.run.smithi044.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T08:21:19.884 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:26.576 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T08:21:26.576 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T08:21:26.577 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T08:21:26.578 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T08:21:26.579 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T08:21:26.579 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T08:21:26.591 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T08:21:26.591 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T08:21:26.591 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T08:21:26.592 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T08:21:26.593 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T08:21:26.594 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T08:21:26.595 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T08:21:26.596 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T08:21:26.597 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T08:21:26.598 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T08:21:26.599 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T08:21:26.600 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T08:21:26.601 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T08:21:26.602 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T08:21:26.603 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T08:21:26.606 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T08:21:26.606 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T08:21:26.606 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T08:21:26.606 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T08:21:26.606 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T08:21:29.057 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T08:21:29.058 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:29.058 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T08:21:29.059 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.060 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.061 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T08:21:29.062 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.063 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T08:21:29.064 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T08:21:29.065 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T08:21:29.066 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T08:21:29.067 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T08:21:29.068 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.069 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T08:21:29.070 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T08:21:29.071 INFO:teuthology.orchestra.run.smithi044.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T08:21:29.072 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T08:21:29.072 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T08:21:29.072 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T08:21:29.072 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:21:29.072 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-07-26T08:21:29.671 DEBUG:teuthology.parallel:result is None 2024-07-26T08:21:29.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:21:29.822 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T08:21:29.878 INFO:teuthology.orchestra.run.smithi044.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T08:21:29.878 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T08:21:29.879 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T08:21:29.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:21:30.027 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T08:21:30.053 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T08:21:30.054 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T08:21:30.054 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T08:21:30.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:21:30.195 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T08:21:30.221 INFO:teuthology.orchestra.run.smithi171.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T08:21:30.221 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T08:21:30.221 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T08:21:30.223 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T08:21:30.223 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:21:30.223 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T08:21:30.293 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:21:30.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T08:21:30.335 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:21:30.336 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T08:21:30.367 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T08:21:30.367 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:21:30.367 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T08:21:30.433 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T08:21:30.539 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:21:30.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T08:21:30.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T08:21:30.640 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:21:30.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T08:21:30.669 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T08:21:30.735 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T08:21:30.736 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:21:30.736 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T08:21:30.803 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T08:21:30.925 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:21:30.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T08:21:30.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T08:21:31.012 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:21:31.012 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T08:21:31.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T08:21:31.120 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T08:21:31.120 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:21:31.120 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T08:21:31.205 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T08:21:31.284 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:21:31.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T08:21:31.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T08:21:31.370 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:21:31.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T08:21:31.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T08:21:31.457 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-26T08:21:31.563 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug cephsqlite': 20, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug'}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mds': {'mds_bal_replicate_threshold': 1}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-07-26T08:21:31.564 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:21:31.564 INFO:tasks.cephadm:Cluster fsid is 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:21:31.564 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-26T08:21:31.565 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.44', 'mon.b': '172.21.15.116', 'mon.c': '172.21.15.171'} 2024-07-26T08:21:31.565 INFO:tasks.cephadm:First mon is mon.a on smithi044 2024-07-26T08:21:31.565 INFO:tasks.cephadm:First mgr is x 2024-07-26T08:21:31.565 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-26T08:21:31.565 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2024-07-26T08:21:31.639 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-07-26T08:21:31.663 DEBUG:teuthology.orchestra.run.smithi171:> sudo hostname $(hostname -s) 2024-07-26T08:21:31.686 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-26T08:21:31.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:21:31.827 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-26T08:21:31.979 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:21:31.980 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T08:21:31.980 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T08:21:31.981 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:32.412 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 08:21 /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:32.412 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:32.764 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 08:21 /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:32.764 DEBUG:teuthology.orchestra.run.smithi171:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:33.138 INFO:teuthology.orchestra.run.smithi171.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 08:21 /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:33.138 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:33.193 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:33.212 DEBUG:teuthology.orchestra.run.smithi171:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T08:21:33.274 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-26T08:21:33.275 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T08:21:33.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T08:21:33.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T08:21:34.122 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T08:21:34.144 INFO:teuthology.orchestra.run.smithi171.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T08:21:35.018 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T08:22:22.772 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-07-26T08:22:22.772 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T08:22:22.772 INFO:teuthology.orchestra.run.smithi171.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T08:22:22.772 INFO:teuthology.orchestra.run.smithi171.stdout: "repo_digests": [ 2024-07-26T08:22:22.772 INFO:teuthology.orchestra.run.smithi171.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T08:22:22.773 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-07-26T08:22:22.773 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-07-26T08:22:26.979 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-07-26T08:22:29.542 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:22:29.542 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T08:22:29.543 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T08:22:29.543 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2024-07-26T08:22:29.543 INFO:teuthology.orchestra.run.smithi044.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T08:22:29.543 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-07-26T08:22:29.543 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:22:29.587 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2024-07-26T08:22:29.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-07-26T08:22:29.695 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph 2024-07-26T08:22:29.728 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2024-07-26T08:22:29.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-07-26T08:22:29.850 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /etc/ceph 2024-07-26T08:22:29.879 INFO:tasks.cephadm:Writing seed config... 2024-07-26T08:22:29.945 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-26T08:22:29.946 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-26T08:22:29.947 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-26T08:22:29.948 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-26T08:22:29.948 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-07-26T08:22:29.948 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:22:29.948 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-26T08:22:30.011 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 10471e4e-4b28-11ef-bcb7-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [mds] mds_bal_replicate_threshold = 1 2024-07-26T08:22:30.011 DEBUG:teuthology.orchestra.run.smithi044:mon.a> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a.service 2024-07-26T08:22:30.017 DEBUG:teuthology.orchestra.run.smithi044:mgr.x> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.x.service 2024-07-26T08:22:30.064 INFO:tasks.cephadm:Bootstrapping... 2024-07-26T08:22:30.065 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.44 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-26T08:22:30.356 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-07-26T08:22:30.356 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.44', '--skip-admin-label'] 2024-07-26T08:22:30.357 INFO:teuthology.orchestra.run.smithi044.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-26T08:22:30.357 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying podman|docker is present... 2024-07-26T08:22:30.418 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 5.1.2 2024-07-26T08:22:30.419 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying lvm2 is present... 2024-07-26T08:22:30.419 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying time synchronization is in place... 2024-07-26T08:22:30.441 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T08:22:30.441 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T08:22:30.460 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T08:22:30.460 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-07-26T08:22:30.485 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-07-26T08:22:30.507 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-07-26T08:22:30.507 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-07-26T08:22:30.508 INFO:teuthology.orchestra.run.smithi044.stdout:Repeating the final host check... 2024-07-26T08:22:30.570 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 5.1.2 2024-07-26T08:22:30.570 INFO:teuthology.orchestra.run.smithi044.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-26T08:22:30.570 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl is present 2024-07-26T08:22:30.570 INFO:teuthology.orchestra.run.smithi044.stdout:lvcreate is present 2024-07-26T08:22:30.586 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T08:22:30.586 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T08:22:30.601 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T08:22:30.601 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-07-26T08:22:30.625 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-07-26T08:22:30.648 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-07-26T08:22:30.648 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-07-26T08:22:30.648 INFO:teuthology.orchestra.run.smithi044.stdout:Host looks OK 2024-07-26T08:22:30.648 INFO:teuthology.orchestra.run.smithi044.stdout:Cluster fsid: 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:30.648 INFO:teuthology.orchestra.run.smithi044.stdout:Acquiring lock 139652184028304 on /run/cephadm/10471e4e-4b28-11ef-bcb7-c7b262605968.lock 2024-07-26T08:22:30.649 INFO:teuthology.orchestra.run.smithi044.stdout:Lock 139652184028304 acquired on /run/cephadm/10471e4e-4b28-11ef-bcb7-c7b262605968.lock 2024-07-26T08:22:30.649 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 3300 ... 2024-07-26T08:22:30.650 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 6789 ... 2024-07-26T08:22:30.650 INFO:teuthology.orchestra.run.smithi044.stdout:Base mon IP(s) is [172.21.15.44:3300, 172.21.15.44:6789], mon addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-07-26T08:22:30.661 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.44 metric 100 2024-07-26T08:22:30.661 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2024-07-26T08:22:30.670 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-26T08:22:30.671 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-07-26T08:22:30.671 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-07-26T08:22:30.681 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-26T08:22:30.682 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-26T08:22:30.682 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T08:22:30.682 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-07-26T08:22:30.682 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-07-26T08:22:30.682 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T08:22:30.684 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-07-26T08:22:30.684 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-07-26T08:22:30.684 INFO:teuthology.orchestra.run.smithi044.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-26T08:22:30.686 INFO:teuthology.orchestra.run.smithi044.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-26T08:22:30.688 INFO:teuthology.orchestra.run.smithi044.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Getting image source signatures 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T08:22:31.357 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-26T08:22:32.674 INFO:teuthology.orchestra.run.smithi044.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:22:32.675 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:22:32.675 INFO:teuthology.orchestra.run.smithi044.stdout:Extracting ceph user uid/gid from container image... 2024-07-26T08:22:33.823 INFO:teuthology.orchestra.run.smithi044.stdout:stat: stdout 167 167 2024-07-26T08:22:33.824 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial keys... 2024-07-26T08:22:35.007 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQDKXKNmszdcGBAAIvjC3BxmQQgf/vs/KYpK3A== 2024-07-26T08:22:36.176 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQDLXKNmE9UQIhAAF8aoqjFXhX7JxwaN49q+lg== 2024-07-26T08:22:37.268 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQDMXKNm5cbeKBAAk7WEoL1Ps86HI85/c9tqAw== 2024-07-26T08:22:37.270 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial monmap... 2024-07-26T08:22:38.450 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool for a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = quincy 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: set fsid to 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:22:38.451 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mon... 2024-07-26T08:22:39.699 INFO:teuthology.orchestra.run.smithi044.stdout:create mon.a on 2024-07-26T08:22:40.082 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-26T08:22:40.331 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-26T08:22:40.621 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968.target. 2024-07-26T08:22:40.621 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968.target. 2024-07-26T08:22:41.039 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a 2024-07-26T08:22:41.039 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a.service: Unit ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a.service not loaded. 2024-07-26T08:22:41.347 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968.target.wants/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a.service → /etc/systemd/system/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@.service. 2024-07-26T08:22:42.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:41 smithi044 podman[32017]: 2024-07-26 08:22:41.748463687 +0000 UTC m=+0.130100588 container create db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T08:22:42.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:42 smithi044 podman[32017]: 2024-07-26 08:22:42.002086221 +0000 UTC m=+0.383723156 container init db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T08:22:42.140 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T08:22:42.140 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-07-26T08:22:42.161 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T08:22:42.161 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-07-26T08:22:42.161 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-07-26T08:22:42.161 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T08:22:42.162 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon to start... 2024-07-26T08:22:42.162 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon... 2024-07-26T08:22:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:42 smithi044 podman[32017]: 2024-07-26 08:22:42.013451748 +0000 UTC m=+0.395088636 container start db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:22:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:42 smithi044 bash[32017]: db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd 2024-07-26T08:22:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:42 smithi044 systemd[1]: Started Ceph mon.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:22:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:42 smithi044 ceph-mon[32036]: mkfs 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:42 smithi044 ceph-mon[32036]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T08:22:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T08:22:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: monmap epoch 1 2024-07-26T08:22:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: last_changed 2024-07-26T08:22:37.823942+0000 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: min_mon_release 19 (squid) 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: election_strategy: 1 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: fsmap 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: mgrmap e1: no daemons active 2024-07-26T08:22:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:43 smithi044 ceph-mon[32036]: from='client.? 172.21.15.44:0/3540782994' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-26T08:22:43.601 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout cluster: 2024-07-26T08:22:43.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout id: 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:43.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-26T08:22:43.602 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:22:43.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout services: 2024-07-26T08:22:43.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.795509s) 2024-07-26T08:22:43.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-26T08:22:43.603 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout data: 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pgs: 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:mon is available 2024-07-26T08:22:43.604 INFO:teuthology.orchestra.run.smithi044.stdout:Assimilating anything we can from ceph.conf... 2024-07-26T08:22:44.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:44 smithi044 ceph-mon[32036]: from='client.? 172.21.15.44:0/1098613237' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T08:22:44.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:44 smithi044 ceph-mon[32036]: from='client.? 172.21.15.44:0/1098613237' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-26T08:22:44.970 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T08:22:44.971 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T08:22:44.972 INFO:teuthology.orchestra.run.smithi044.stdout:Generating new minimal ceph.conf... 2024-07-26T08:22:46.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:45 smithi044 ceph-mon[32036]: from='client.? 172.21.15.44:0/2179464376' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:46.333 INFO:teuthology.orchestra.run.smithi044.stdout:Restarting the monitor... 2024-07-26T08:22:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:46 smithi044 systemd[1]: Stopping Ceph mon.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:22:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:50 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a[32032]: 2024-07-26T08:22:50.858+0000 7fda77000640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-26T08:22:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:50 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a[32032]: 2024-07-26T08:22:50.858+0000 7fda77000640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-26T08:22:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:51 smithi044 podman[32278]: 2024-07-26 08:22:51.032860699 +0000 UTC m=+4.340765740 container stop db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:22:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:51 smithi044 podman[32278]: 2024-07-26 08:22:51.067064656 +0000 UTC m=+4.374969673 container died db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux ) 2024-07-26T08:22:58.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 podman[32278]: 2024-07-26 08:22:58.01796894 +0000 UTC m=+11.325873984 container cleanup db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T08:22:58.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 bash[32278]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a 2024-07-26T08:22:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 podman[32291]: 2024-07-26 08:22:58.260301676 +0000 UTC m=+7.205691103 container remove db09d1071c43c87efa6dbe1df2cdc2f45147dfdfa7a7d9682548f965a2c045fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:22:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-26T08:22:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 systemd[1]: Stopped Ceph mon.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:22:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.a.service: Consumed 2.125s CPU time. 2024-07-26T08:22:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:58 smithi044 systemd[1]: Starting Ceph mon.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:22:59.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 podman[32384]: 2024-07-26 08:22:59.144965359 +0000 UTC m=+0.125968438 container create dde30f330908b2584cfa19376f28d927df73fad66f38d7ee12e6618688d46375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:22:59.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 podman[32384]: 2024-07-26 08:22:59.062155477 +0000 UTC m=+0.043158536 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:22:59.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 podman[32384]: 2024-07-26 08:22:59.391090989 +0000 UTC m=+0.372094108 container init dde30f330908b2584cfa19376f28d927df73fad66f38d7ee12e6618688d46375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-26T08:22:59.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 podman[32384]: 2024-07-26 08:22:59.405158309 +0000 UTC m=+0.386161401 container start dde30f330908b2584cfa19376f28d927df73fad66f38d7ee12e6618688d46375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T08:22:59.494 INFO:teuthology.orchestra.run.smithi044.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-26T08:22:59.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 bash[32384]: dde30f330908b2584cfa19376f28d927df73fad66f38d7ee12e6618688d46375 2024-07-26T08:22:59.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 systemd[1]: Started Ceph mon.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:22:59.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T08:22:59.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T08:22:59.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: pidfile_write: ignore empty --pid-file 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: load: jerasure load: lrc 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: RocksDB version: 7.9.2 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Git sha 0 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: DB SUMMARY 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: DB Session ID: OZ03RNH8OS3RNKU7N3AG 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: CURRENT file: CURRENT 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95001 ; 2024-07-26T08:22:59.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.error_if_exists: 0 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.create_if_missing: 0 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.paranoid_checks: 1 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.env: 0x55806e1f0280 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.info_log: 0x55809cdf7060 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.statistics: (nil) 2024-07-26T08:22:59.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.use_fsync: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_log_file_size: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.allow_fallocate: 1 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.use_direct_reads: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T08:22:59.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.db_log_dir: 2024-07-26T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.wal_dir: 2024-07-26T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T08:22:59.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.write_buffer_manager: 0x55809cdfb5e0 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T08:22:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.unordered_write: 0 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.row_cache: None 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.wal_filter: None 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T08:22:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.two_write_queues: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.wal_compression: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.atomic_flush: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.log_readahead_size: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T08:22:59.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_background_jobs: 2 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_background_compactions: -1 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_subcompactions: 1 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T08:22:59.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_open_files: -1 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_background_flushes: -1 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Compression algorithms supported: 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kZSTD supported: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kXpressCompression supported: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kBZip2Compression supported: 0 2024-07-26T08:22:59.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kLZ4Compression supported: 1 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kZlibCompression supported: 1 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: kSnappyCompression supported: 1 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.merge_operator: 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_filter: None 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T08:22:59.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55809cdf6c00) 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_type: 0 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_index_type: 0 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_shortening: 1 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T08:22:59.691 INFO:journalctl@ceph.mon.a.smithi044.stdout: checksum: 4 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: no_block_cache: 0 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache: 0x558098dbb610 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_options: 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: capacity : 536870912 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_shard_bits : 4 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: strict_capacity_limit : 0 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_compressed: (nil) 2024-07-26T08:22:59.692 INFO:journalctl@ceph.mon.a.smithi044.stdout: persistent_cache: (nil) 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size: 4096 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size_deviation: 10 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_restart_interval: 16 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_block_restart_interval: 1 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: metadata_block_size: 4096 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: partition_filters: 0 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: use_delta_encoding: 1 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: filter_policy: bloomfilter 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: whole_key_filtering: 1 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: verify_compression: 0 2024-07-26T08:22:59.693 INFO:journalctl@ceph.mon.a.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: format_version: 5 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_index_compression: 1 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_align: 0 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: max_auto_readahead_size: 262144 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: prepopulate_block_cache: 0 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: initial_auto_readahead_size: 8192 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression: NoCompression 2024-07-26T08:22:59.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.num_levels: 7 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T08:22:59.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T08:22:59.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T08:22:59.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T08:22:59.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T08:22:59.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T08:22:59.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T08:22:59.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T08:22:59.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T08:22:59.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T08:22:59.699 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.inplace_update_support: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.bloom_locality: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.max_successive_merges: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T08:22:59.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.ttl: 2592000 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.enable_blob_files: false 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.min_blob_size: 0 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T08:22:59.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d3aa56af-8893-4c9a-9ced-4499654fd745 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982179530577, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982179532314, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91152, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 89200, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11619, "raw_average_key_size": 48, "raw_value_size": 82674, "raw_average_value_size": 343, "num_data_blocks": 12, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721982179, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d3aa56af-8893-4c9a-9ced-4499654fd745", "db_session_id": "OZ03RNH8OS3RNKU7N3AG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982179532566, "job": 1, "event": "recovery_finished"} 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-26T08:22:59.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55809ce1aa80 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: DB pointer 0x55809dbba000 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** DB Stats ** 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T08:22:59.703 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 2/0 90.88 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 90.88 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T08:22:59.704 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 11.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 11.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache BinnedLRUCache@0x558098dbb610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-07-26T08:22:59.705 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???) e1 preinit fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).mds e1 new map 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).mds e1 print_map 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout: e1 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout: btime 2024-07-26T08:22:42:160289+0000 2024-07-26T08:22:59.709 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout: legacy client fscid: -1 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout: No filesystems configured 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: monmap epoch 1 2024-07-26T08:22:59.710 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: last_changed 2024-07-26T08:22:37.823942+0000 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: min_mon_release 19 (squid) 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: election_strategy: 1 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: fsmap 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T08:22:59.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:22:59 smithi044 ceph-mon[32404]: mgrmap e1: no daemons active 2024-07-26T08:23:00.879 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-26T08:23:00.904 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-26T08:23:00.904 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mgr... 2024-07-26T08:23:00.905 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-26T08:23:00.906 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-26T08:23:01.370 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.x 2024-07-26T08:23:01.371 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.x.service: Unit ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.x.service not loaded. 2024-07-26T08:23:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:01 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3252940134' entity='client.admin' 2024-07-26T08:23:01.637 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968.target.wants/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@.service. 2024-07-26T08:23:02.016 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:01 smithi044 systemd[1]: Starting Ceph mgr.x for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:23:02.468 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:02 smithi044 podman[32627]: 2024-07-26 08:23:01.936438789 +0000 UTC m=+0.037669525 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:23:02.468 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:02 smithi044 podman[32627]: 2024-07-26 08:23:02.031598598 +0000 UTC m=+0.132829349 container create d9075cef9982b1a6ac34a647774ab86b83f1a10c356acfb8a4839133611dca98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-07-26T08:23:02.468 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:02 smithi044 podman[32627]: 2024-07-26 08:23:02.260511906 +0000 UTC m=+0.361742704 container init d9075cef9982b1a6ac34a647774ab86b83f1a10c356acfb8a4839133611dca98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-26T08:23:02.468 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:02 smithi044 podman[32627]: 2024-07-26 08:23:02.270935168 +0000 UTC m=+0.372165923 container start d9075cef9982b1a6ac34a647774ab86b83f1a10c356acfb8a4839133611dca98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-07-26T08:23:02.511 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T08:23:02.512 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-07-26T08:23:02.534 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T08:23:02.534 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-07-26T08:23:02.534 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-07-26T08:23:02.535 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T08:23:02.561 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T08:23:02.561 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-07-26T08:23:02.583 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T08:23:02.583 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-07-26T08:23:02.583 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-07-26T08:23:02.584 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-26T08:23:02.584 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr to start... 2024-07-26T08:23:02.584 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr... 2024-07-26T08:23:02.740 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:02 smithi044 bash[32627]: d9075cef9982b1a6ac34a647774ab86b83f1a10c356acfb8a4839133611dca98 2024-07-26T08:23:02.740 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:02 smithi044 systemd[1]: Started Ceph mgr.x for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:23:03.639 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:03 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4288197139' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T08:23:03.952 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:23:03.953 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:03.953 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:23:03.953 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T08:23:03.953 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T08:23:03.953 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T08:23:03.954 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T08:23:03.954 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.954 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T08:23:03.955 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T08:23:03.955 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-07-26T08:23:03.955 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:03.955 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T08:23:03.956 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T08:23:03.958 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:03.958 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-07-26T08:23:03.958 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T08:23:03.960 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T08:23:03.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T08:23:03.961 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T08:23:03.964 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T08:22:42:160289+0000", 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T08:23:03.965 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T08:22:42.161876+0000", 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T08:23:03.966 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:03.967 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T08:23:03.967 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:03.967 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (1/15)... 2024-07-26T08:23:07.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:06 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3201789154' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T08:23:07.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T08:23:07.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T08:23:07.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T08:23:07.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T08:23:07.355 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T08:23:07.358 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T08:22:42:160289+0000", 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T08:23:07.359 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T08:22:42.161876+0000", 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:07.360 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T08:23:07.361 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:07.361 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (2/15)... 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: Activating manager daemon x 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: mgrmap e2: x(active, starting, since 0.0134217s) 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T08:23:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: Manager daemon x is now available 2024-07-26T08:23:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:23:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' 2024-07-26T08:23:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:23:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:08 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' 2024-07-26T08:23:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:09 smithi044 ceph-mon[32404]: from='mgr.14100 172.21.15.44:0/513345183' entity='mgr.x' 2024-07-26T08:23:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:09 smithi044 ceph-mon[32404]: mgrmap e3: x(active, since 1.0274s) 2024-07-26T08:23:10.980 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:10 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2095517876' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T08:23:10.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:23:10.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:10.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:23:10.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T08:23:10.996 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T08:23:10.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:10.998 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T08:22:42:160289+0000", 2024-07-26T08:23:10.999 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T08:23:11.000 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T08:23:11.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T08:22:42.161876+0000", 2024-07-26T08:23:11.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T08:23:11.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-07-26T08:23:11.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T08:23:11.001 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:11.001 INFO:teuthology.orchestra.run.smithi044.stdout:mgr is available 2024-07-26T08:23:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:11 smithi044 ceph-mon[32404]: mgrmap e4: x(active, since 2s) 2024-07-26T08:23:12.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:23:12.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T08:23:12.458 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling cephadm module... 2024-07-26T08:23:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:12 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/379842868' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T08:23:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:13 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4201432578' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-26T08:23:14.016 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:13 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ignoring --setuser ceph since I am not root 2024-07-26T08:23:14.016 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:13 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ignoring --setgroup ceph since I am not root 2024-07-26T08:23:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:14 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4201432578' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-26T08:23:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:14 smithi044 ceph-mon[32404]: mgrmap e5: x(active, since 5s) 2024-07-26T08:23:15.823 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:15 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2336834985' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T08:23:15.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-26T08:23:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T08:23:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T08:23:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T08:23:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-07-26T08:23:15.843 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 5... 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: Active manager daemon x restarted 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: Activating manager daemon x 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: osdmap e2: 0 total, 0 up, 0 in 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: mgrmap e6: x(active, starting, since 0.0129851s) 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T08:23:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T08:23:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T08:23:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T08:23:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: Manager daemon x is now available 2024-07-26T08:23:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:20 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: Found migration_current of "None". Setting to last migration. 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:21 smithi044 ceph-mon[32404]: mgrmap e7: x(active, since 1.01924s) 2024-07-26T08:23:21.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:21.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-26T08:23:21.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T08:23:21.845 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:21.846 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 5 is available 2024-07-26T08:23:21.846 INFO:teuthology.orchestra.run.smithi044.stdout:Setting orchestrator backend to cephadm... 2024-07-26T08:23:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-mon[32404]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T08:23:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-mon[32404]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:23:22.366+0000 7f278dc00640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:08:23:22] ENGINE Error in HTTPServer.serve 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: Traceback (most recent call last): 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._connections.run(self.expiration_interval) 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._run(expiration_interval) 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:23:22.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: s = self.context.wrap_socket( 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: return self.sslsocket_class._create( 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self.do_handshake() 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._sslobj.do_handshake() 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:23:22.517 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:22 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:23:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: mgrmap e8: x(active, since 2s) 2024-07-26T08:23:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:22] ENGINE Bus STARTING 2024-07-26T08:23:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:22] ENGINE Serving on https://172.21.15.44:7150 2024-07-26T08:23:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:22] ENGINE Error in HTTPServer.serve 2024-07-26T08:23:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout: Traceback (most recent call last): 2024-07-26T08:23:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._run(expiration_interval) 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: s = self.context.wrap_socket( 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: return self.sslsocket_class._create( 2024-07-26T08:23:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout: self.do_handshake() 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._sslobj.do_handshake() 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:22] ENGINE Serving on http://172.21.15.44:8765 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:22] ENGINE Bus STARTED 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:23.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:23 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:25.100 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:24 smithi044 ceph-mon[32404]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:25.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-26T08:23:25.110 INFO:teuthology.orchestra.run.smithi044.stdout:Generating ssh key... 2024-07-26T08:23:26.472 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: Generating public/private rsa key pair. 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: Your identification has been saved in /tmp/tmpjpg2znyj/key 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: Your public key has been saved in /tmp/tmpjpg2znyj/key.pub 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: The key fingerprint is: 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: SHA256:BFc5K8hKPPq/ZBN4EYLQcMQc7TTSEI4JQZTATGJa1mg ceph-10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: The key's randomart image is: 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: +---[RSA 3072]----+ 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: |#&XX. o .... | 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: |=@E *. + o | 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: |+..= o... o | 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: | =.oo. . | 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: | o.oo S. | 2024-07-26T08:23:26.473 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: | . .. . | 2024-07-26T08:23:26.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: | . + | 2024-07-26T08:23:26.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: | .o . | 2024-07-26T08:23:26.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: | .o. | 2024-07-26T08:23:26.474 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: +----[SHA256]-----+ 2024-07-26T08:23:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:27 smithi044 ceph-mon[32404]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:27 smithi044 ceph-mon[32404]: Generating ssh key... 2024-07-26T08:23:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:27 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:27 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:28 smithi044 ceph-mon[32404]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:28.308 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:23:28.309 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-26T08:23:28.310 INFO:teuthology.orchestra.run.smithi044.stdout:Adding key to root@localhost authorized_keys... 2024-07-26T08:23:28.314 INFO:teuthology.orchestra.run.smithi044.stdout:Adding host smithi044... 2024-07-26T08:23:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:29 smithi044 ceph-mon[32404]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:30 smithi044 ceph-mon[32404]: Deploying cephadm binary to smithi044 2024-07-26T08:23:35.515 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Added host 'smithi044' with addr '172.21.15.44' 2024-07-26T08:23:35.515 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mon service... 2024-07-26T08:23:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:35 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:35 smithi044 ceph-mon[32404]: Added host smithi044 2024-07-26T08:23:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:35 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:37.438 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-26T08:23:37.439 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mgr service... 2024-07-26T08:23:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:37 smithi044 ceph-mon[32404]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:37 smithi044 ceph-mon[32404]: Saving service mon spec with placement count:5 2024-07-26T08:23:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:37 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:39.606 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:39 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:39.606 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:39 smithi044 ceph-mon[32404]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:39.606 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:39 smithi044 ceph-mon[32404]: Saving service mgr spec with placement count:2 2024-07-26T08:23:39.607 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:39 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:39.607 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:39 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:39.619 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-26T08:23:41.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:41 smithi044 ceph-mon[32404]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:23:41.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:41 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/69480583' entity='client.admin' 2024-07-26T08:23:41.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:41 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:43.178 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling the dashboard module... 2024-07-26T08:23:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:43 smithi044 ceph-mon[32404]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:23:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:43 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/210622361' entity='client.admin' 2024-07-26T08:23:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:43 smithi044 ceph-mon[32404]: from='mgr.14118 172.21.15.44:0/1214800853' entity='mgr.x' 2024-07-26T08:23:44.509 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:44 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1422942441' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-26T08:23:44.765 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:44 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ignoring --setuser ceph since I am not root 2024-07-26T08:23:44.765 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:44 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ignoring --setgroup ceph since I am not root 2024-07-26T08:23:45.673 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:45 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1422942441' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-26T08:23:45.673 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:45 smithi044 ceph-mon[32404]: mgrmap e9: x(active, since 24s) 2024-07-26T08:23:46.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:46 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/726024078' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:47.200 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-07-26T08:23:47.201 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 9... 2024-07-26T08:23:51.237 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: Active manager daemon x restarted 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: Activating manager daemon x 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: osdmap e3: 0 total, 0 up, 0 in 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: mgrmap e10: x(active, starting, since 0.0101482s) 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T08:23:51.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: Manager daemon x is now available 2024-07-26T08:23:51.239 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:50 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:52.231 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:51 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:23:52.231 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:51 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:23:52.231 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:51 smithi044 ceph-mon[32404]: mgrmap e11: x(active, since 1.01236s) 2024-07-26T08:23:52.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-07-26T08:23:52.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-26T08:23:52.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T08:23:52.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-07-26T08:23:52.542 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 9 is available 2024-07-26T08:23:52.542 INFO:teuthology.orchestra.run.smithi044.stdout:Generating a dashboard self-signed certificate... 2024-07-26T08:23:52.837 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:23:52.749+0000 7f2443600640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:08:23:52] ENGINE Error in HTTPServer.serve 2024-07-26T08:23:52.837 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: Traceback (most recent call last): 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._connections.run(self.expiration_interval) 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._run(expiration_interval) 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:23:52.838 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: s = self.context.wrap_socket( 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: return self.sslsocket_class._create( 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self.do_handshake() 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._sslobj.do_handshake() 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:23:52.839 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:23:53.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:52] ENGINE Bus STARTING 2024-07-26T08:23:53.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:52 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:52] ENGINE Serving on http://172.21.15.44:8765 2024-07-26T08:23:54.117 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:52] ENGINE Serving on https://172.21.15.44:7150 2024-07-26T08:23:54.117 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:52] ENGINE Bus STARTED 2024-07-26T08:23:54.117 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: [26/Jul/2024:08:23:52] ENGINE Error in HTTPServer.serve 2024-07-26T08:23:54.117 INFO:journalctl@ceph.mon.a.smithi044.stdout: Traceback (most recent call last): 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._run(expiration_interval) 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: s = self.context.wrap_socket( 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:23:54.118 INFO:journalctl@ceph.mon.a.smithi044.stdout: return self.sslsocket_class._create( 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout: self.do_handshake() 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._sslobj.do_handshake() 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: mgrmap e12: x(active, since 2s) 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:23:54.119 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:53 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:23:54.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-26T08:23:54.409 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial admin user... 2024-07-26T08:23:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:55 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:23:56.327 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Kfw9CPfPs6y/IF/NZ4Z/e.cVSk3aZV9/SOBosznC/IHIgyJHBNsGy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1721982235, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-26T08:23:56.327 INFO:teuthology.orchestra.run.smithi044.stdout:Fetching dashboard port number... 2024-07-26T08:23:56.923 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:56 smithi044 ceph-mon[32404]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:23:56.923 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:56 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:23:56.924 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:56 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:23:57.857 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:57 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/945429020' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-26T08:23:57.997 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 8443 2024-07-26T08:23:58.032 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T08:23:58.032 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-07-26T08:23:58.059 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T08:23:58.059 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-07-26T08:23:58.060 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-07-26T08:23:58.060 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-26T08:23:58.067 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph Dashboard is now available at: 2024-07-26T08:23:58.068 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:23:58.068 INFO:teuthology.orchestra.run.smithi044.stdout: URL: https://smithi044.front.sepia.ceph.com:8443/ 2024-07-26T08:23:58.068 INFO:teuthology.orchestra.run.smithi044.stdout: User: admin 2024-07-26T08:23:58.068 INFO:teuthology.orchestra.run.smithi044.stdout: Password: eq48ysricd 2024-07-26T08:23:58.068 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:23:58.069 INFO:teuthology.orchestra.run.smithi044.stdout:Saving cluster configuration to /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config directory 2024-07-26T08:23:58.878 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:23:58 smithi044 ceph-mon[32404]: mgrmap e13: x(active, since 6s) 2024-07-26T08:24:00.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-26T08:24:00.413 INFO:teuthology.orchestra.run.smithi044.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-26T08:24:00.413 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout:Or, if you are only running a single cluster on this host: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph telemetry on 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout:For more information see: 2024-07-26T08:24:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-26T08:24:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:00.415 INFO:teuthology.orchestra.run.smithi044.stdout:Bootstrap complete. 2024-07-26T08:24:00.506 INFO:tasks.cephadm:Fetching config... 2024-07-26T08:24:00.506 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:24:00.506 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-26T08:24:00.553 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-26T08:24:00.553 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:24:00.553 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-26T08:24:00.631 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-26T08:24:00.631 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:24:00.632 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-26T08:24:00.746 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-26T08:24:00.747 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:24:00.747 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-26T08:24:00.838 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-26T08:24:00.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T08:24:00.933 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:00 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/356696171' entity='client.admin' 2024-07-26T08:24:00.968 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:01.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T08:24:01.052 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:01.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T08:24:01.104 INFO:teuthology.orchestra.run.smithi171.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdIZl5d3matGfD8YwlA4tyfOjjUQ7e6K+5gAvcbrxZefmnNaRRA1gOcys02D0DCo1BOAcTb13Q33X8MhoahNQNfYj7xTVUCHR0iwexV4dkcsAJFzs8Jiw6A/9fFVPIwggJ1WWC/uQCv+wOgK2xq3Zd9D9bywc4wsxSysri/XUxHUmfEtsG8O0eCnMRwt5oVAKaLbKNMUTt4bxa2B9VSFUbS5vAP2+newbL2w6tW4G/Gp6YeQMdqq7unUTG3gI+zsWKnXDM1mzv838yCfOKJ0nCMIdsKIPi33fxoWhIujqLvWq+91qKcy0kuT47jwjUwayUftHLcvuiZuIkHZh7nTCYhQIxes3AURZ0ggInxmaESGSGJ0PfsUIMm2SB25Oe01YBjB8+y7GTQwGp6XcBVsXXZU4GyXFCtnzIwM69QxDCXAbDpz1OxCxCpD/nJ7VUh30lDP9dbD74kSZAIkxjmP4d2RRPBFjSgEnszMvyRVH7cd5raqmkNua/bPAHftqZ1o8= ceph-10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:01.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-26T08:24:01.551 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:05.517 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-26T08:24:05.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-26T08:24:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:05 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4006238619' entity='client.admin' 2024-07-26T08:24:06.553 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:07.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:07 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:07.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:07 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:07.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:07 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:07.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:07 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:07.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:07 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:07.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:07 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:09.726 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi116 2024-07-26T08:24:09.727 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:24:09.727 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.conf 2024-07-26T08:24:09.746 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:24:09.747 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:09.804 INFO:tasks.cephadm:Adding host smithi116 to orchestrator... 2024-07-26T08:24:09.805 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch host add smithi116 2024-07-26T08:24:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: Updating smithi044:/etc/ceph/ceph.conf 2024-07-26T08:24:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:09 smithi044 ceph-mon[32404]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:10.234 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:10 smithi044 ceph-mon[32404]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:12 smithi044 ceph-mon[32404]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:12 smithi044 ceph-mon[32404]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:12 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:12 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:12 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:13 smithi044 ceph-mon[32404]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:14 smithi044 ceph-mon[32404]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:15 smithi044 ceph-mon[32404]: Deploying cephadm binary to smithi116 2024-07-26T08:24:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:16 smithi044 ceph-mon[32404]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:16.976 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi116' with addr '172.21.15.116' 2024-07-26T08:24:17.674 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T08:24:18.139 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:17 smithi044 ceph-mon[32404]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:17 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:17 smithi044 ceph-mon[32404]: Added host smithi116 2024-07-26T08:24:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:17 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:17 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:17 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:19 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:20.369 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:20.369 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}] 2024-07-26T08:24:20.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:20 smithi044 ceph-mon[32404]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:21.044 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi171 2024-07-26T08:24:21.044 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:24:21.044 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.conf 2024-07-26T08:24:21.062 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:24:21.062 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:21.119 INFO:tasks.cephadm:Adding host smithi171 to orchestrator... 2024-07-26T08:24:21.119 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch host add smithi171 2024-07-26T08:24:21.602 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:21.762 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:21.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:21 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:22.599 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:22.600 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: Updating smithi116:/etc/ceph/ceph.conf 2024-07-26T08:24:22.600 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:22.600 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:22.600 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:22.600 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:22.600 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:22 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:23.487 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:23 smithi044 ceph-mon[32404]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:24 smithi044 ceph-mon[32404]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:25 smithi044 ceph-mon[32404]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:26 smithi044 ceph-mon[32404]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:26 smithi044 ceph-mon[32404]: Deploying cephadm binary to smithi171 2024-07-26T08:24:28.279 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi171' with addr '172.21.15.171' 2024-07-26T08:24:28.568 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:28 smithi044 ceph-mon[32404]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:28.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:28 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:28.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:28 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:28.982 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T08:24:29.447 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:29 smithi044 ceph-mon[32404]: Added host smithi171 2024-07-26T08:24:29.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:29 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:29.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:29 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:30.944 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:30 smithi044 ceph-mon[32404]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:30.944 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:30 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:31.714 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:24:31.714 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.171", "hostname": "smithi171", "labels": [], "status": ""}] 2024-07-26T08:24:32.475 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-26T08:24:32.475 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd crush tunables default 2024-07-26T08:24:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:32 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:32.918 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:33.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:33 smithi044 ceph-mon[32404]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:24:33.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:33 smithi044 ceph-mon[32404]: Updating smithi171:/etc/ceph/ceph.conf 2024-07-26T08:24:33.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:33 smithi044 ceph-mon[32404]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:33.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:33 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:33.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:33 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:33.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:33 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:34 smithi044 ceph-mon[32404]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:34 smithi044 ceph-mon[32404]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:34 smithi044 ceph-mon[32404]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:35.676 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-07-26T08:24:35.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:35 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2231056346' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T08:24:36.311 INFO:tasks.cephadm:Adding mon.a on smithi044 2024-07-26T08:24:36.311 INFO:tasks.cephadm:Adding mon.b on smithi116 2024-07-26T08:24:36.311 INFO:tasks.cephadm:Adding mon.c on smithi171 2024-07-26T08:24:36.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch apply mon '3;smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c' 2024-07-26T08:24:36.563 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:36 smithi044 ceph-mon[32404]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:36 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2231056346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T08:24:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:36 smithi044 ceph-mon[32404]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:38.196 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mon update... 2024-07-26T08:24:38.747 DEBUG:teuthology.orchestra.run.smithi116:mon.b> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.b.service 2024-07-26T08:24:38.750 DEBUG:teuthology.orchestra.run.smithi171:mon.c> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mon.c.service 2024-07-26T08:24:38.752 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T08:24:38.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T08:24:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:24:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:38 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:39.435 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:39 smithi044 ceph-mon[32404]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:39 smithi044 ceph-mon[32404]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c;count:3 2024-07-26T08:24:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:39 smithi044 ceph-mon[32404]: Deploying daemon mon.c on smithi171 2024-07-26T08:24:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:40 smithi044 ceph-mon[32404]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:42.242 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:24:42.242 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","modified":"2024-07-26T08:22:37.823942Z","created":"2024-07-26T08:22:37.823942Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-26T08:24:42.243 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2024-07-26T08:24:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:42 smithi171 systemd[1]: Starting Ceph mon.c for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:24:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:42 smithi044 ceph-mon[32404]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:42 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/1187642105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:43.285 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 podman[37771]: 2024-07-26 08:24:43.177752994 +0000 UTC m=+0.096847290 container create 8130b0964b0308619606e6a9faef5316eceb317ad08792fc76a5a8a6bfe528ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:24:43.285 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 podman[37771]: 2024-07-26 08:24:43.098214616 +0000 UTC m=+0.017308923 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 podman[37771]: 2024-07-26 08:24:43.380121158 +0000 UTC m=+0.299215456 container init 8130b0964b0308619606e6a9faef5316eceb317ad08792fc76a5a8a6bfe528ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 podman[37771]: 2024-07-26 08:24:43.383051396 +0000 UTC m=+0.302145696 container start 8130b0964b0308619606e6a9faef5316eceb317ad08792fc76a5a8a6bfe528ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pidfile_write: ignore empty --pid-file 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: load: jerasure load: lrc 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: RocksDB version: 7.9.2 2024-07-26T08:24:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Git sha 0 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: DB SUMMARY 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: DB Session ID: OKE18J0E8IO6DOZOL15P 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: CURRENT file: CURRENT 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.error_if_exists: 0 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.create_if_missing: 0 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.paranoid_checks: 1 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T08:24:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T08:24:43.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T08:24:43.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.env: 0x55db99e24280 2024-07-26T08:24:43.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T08:24:43.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.info_log: 0x55db9eb8f080 2024-07-26T08:24:43.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T08:24:43.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.statistics: (nil) 2024-07-26T08:24:43.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.use_fsync: 0 2024-07-26T08:24:43.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_log_file_size: 0 2024-07-26T08:24:43.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.allow_fallocate: 1 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.use_direct_reads: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.db_log_dir: 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.wal_dir: 2024-07-26T08:24:43.591 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.write_buffer_manager: 0x55db9eb935e0 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T08:24:43.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.unordered_write: 0 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.row_cache: None 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.wal_filter: None 2024-07-26T08:24:43.593 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.two_write_queues: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.wal_compression: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.atomic_flush: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.log_readahead_size: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T08:24:43.594 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_background_jobs: 2 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_background_compactions: -1 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_subcompactions: 1 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T08:24:43.595 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_open_files: -1 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_background_flushes: -1 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Compression algorithms supported: 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kZSTD supported: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kXpressCompression supported: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kBZip2Compression supported: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kLZ4Compression supported: 1 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kZlibCompression supported: 1 2024-07-26T08:24:43.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: kSnappyCompression supported: 1 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.merge_operator: 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_filter: None 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db9eb8ec80) 2024-07-26T08:24:43.597 INFO:journalctl@ceph.mon.c.smithi171.stdout: cache_index_and_filter_blocks: 1 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: pin_top_level_index_and_filter: 1 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: index_type: 0 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: data_block_index_type: 0 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: index_shortening: 1 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: checksum: 4 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: no_block_cache: 0 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_cache: 0x55db9ab53610 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_cache_name: BinnedLRUCache 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_cache_options: 2024-07-26T08:24:43.598 INFO:journalctl@ceph.mon.c.smithi171.stdout: capacity : 536870912 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: num_shard_bits : 4 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: strict_capacity_limit : 0 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: high_pri_pool_ratio: 0.000 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_cache_compressed: (nil) 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: persistent_cache: (nil) 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_size: 4096 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_size_deviation: 10 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_restart_interval: 16 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: index_block_restart_interval: 1 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: metadata_block_size: 4096 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: partition_filters: 0 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: use_delta_encoding: 1 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: filter_policy: bloomfilter 2024-07-26T08:24:43.599 INFO:journalctl@ceph.mon.c.smithi171.stdout: whole_key_filtering: 1 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: verify_compression: 0 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: read_amp_bytes_per_bit: 0 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: format_version: 5 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: enable_index_compression: 1 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: block_align: 0 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: max_auto_readahead_size: 262144 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: prepopulate_block_cache: 0 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: initial_auto_readahead_size: 8192 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression: NoCompression 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T08:24:43.600 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.num_levels: 7 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T08:24:43.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T08:24:43.602 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T08:24:43.603 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T08:24:43.604 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.inplace_update_support: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.bloom_locality: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.max_successive_merges: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.ttl: 2592000 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T08:24:43.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.enable_blob_files: false 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.min_blob_size: 0 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T08:24:43.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0045f505-f6aa-4ad2-a8f4-b1ebd72763f3 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982283419276, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982283419797, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721982283, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0045f505-f6aa-4ad2-a8f4-b1ebd72763f3", "db_session_id": "OKE18J0E8IO6DOZOL15P", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982283419927, "job": 1, "event": "recovery_finished"} 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55db9ebb0a80 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: DB pointer 0x55db9f950000 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** DB Stats ** 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T08:24:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** Compaction Stats [default] ** 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:24:43.608 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** Compaction Stats [default] ** 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T08:24:43.609 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T08:24:43.610 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T08:24:43.610 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T08:24:43.610 INFO:journalctl@ceph.mon.c.smithi171.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T08:24:43.610 INFO:journalctl@ceph.mon.c.smithi171.stdout: Block cache BinnedLRUCache@0x55db9ab53610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-07-26T08:24:43.610 INFO:journalctl@ceph.mon.c.smithi171.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: using public_addr v2:172.21.15.171:0/0 -> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: starting mon.c rank -1 at public addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] at bind addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(???) e0 preinit fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 bash[37771]: 8130b0964b0308619606e6a9faef5316eceb317ad08792fc76a5a8a6bfe528ce 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 systemd[1]: Started Ceph mon.c for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).mds e1 new map 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).mds e1 print_map 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout: e1 2024-07-26T08:24:43.611 INFO:journalctl@ceph.mon.c.smithi171.stdout: btime 2024-07-26T08:22:42:160289+0000 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout: legacy client fscid: -1 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout: No filesystems configured 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:24:43.612 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Added host smithi116 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi116:/etc/ceph/ceph.conf 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.614 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Deploying cephadm binary to smithi171 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Added host smithi171 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.615 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi171:/etc/ceph/ceph.conf 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:43.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2231056346' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2231056346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:43.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c;count:3 2024-07-26T08:24:43.618 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: Deploying daemon mon.c on smithi171 2024-07-26T08:24:43.618 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.618 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:43.618 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/1187642105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:43.618 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:43 smithi171 ceph-mon[37789]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T08:24:44.005 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T08:24:44.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T08:24:44.254 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: mon.a calling monitor election 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: mon.c calling monitor election 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: monmap epoch 2 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: last_changed 2024-07-26T08:24:43.459373+0000 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: min_mon_release 19 (squid) 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: election_strategy: 1 2024-07-26T08:24:48.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: fsmap 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: mgrmap e13: x(active, since 57s) 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: overall HEALTH_OK 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:24:48.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:48 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: mon.a calling monitor election 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: mon.c calling monitor election 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:49.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: monmap epoch 2 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: last_changed 2024-07-26T08:24:43.459373+0000 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: min_mon_release 19 (squid) 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: election_strategy: 1 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: fsmap 2024-07-26T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: mgrmap e13: x(active, since 57s) 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: overall HEALTH_OK 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:24:49.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:48 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:49.044 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:24:49.045 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":2,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","modified":"2024-07-26T08:24:43.459373Z","created":"2024-07-26T08:22:37.823942Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T08:24:49.045 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 2 2024-07-26T08:24:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:49 smithi171 ceph-mon[37789]: Deploying daemon mon.b on smithi116 2024-07-26T08:24:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:49 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/1679059779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:49 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:49 smithi044 ceph-mon[32404]: Deploying daemon mon.b on smithi116 2024-07-26T08:24:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:49 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/1679059779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:49 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:50.497 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T08:24:50.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T08:24:50.757 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:24:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:50 smithi044 ceph-mon[32404]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:50.766 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:24:50 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:24:50.461+0000 7f248aa00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-07-26T08:24:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:50 smithi171 ceph-mon[37789]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:51.840 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:51 smithi116 systemd[1]: Starting Ceph mon.b for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:24:52.104 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 podman[37078]: 2024-07-26 08:24:52.011244528 +0000 UTC m=+0.098643158 container create 0b9bd2d4518711991fbebf5d692937e9cf07bdf42c59b4071c66bf1af7f8e72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T08:24:52.104 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 podman[37078]: 2024-07-26 08:24:51.930393844 +0000 UTC m=+0.017792467 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:24:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 podman[37078]: 2024-07-26 08:24:52.205490615 +0000 UTC m=+0.292889247 container init 0b9bd2d4518711991fbebf5d692937e9cf07bdf42c59b4071c66bf1af7f8e72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 podman[37078]: 2024-07-26 08:24:52.208599082 +0000 UTC m=+0.295997708 container start 0b9bd2d4518711991fbebf5d692937e9cf07bdf42c59b4071c66bf1af7f8e72f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pidfile_write: ignore empty --pid-file 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: load: jerasure load: lrc 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: RocksDB version: 7.9.2 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Git sha 0 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T08:24:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: DB SUMMARY 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: DB Session ID: C4XGZEUS3CTOTV0IRUKA 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: CURRENT file: CURRENT 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.error_if_exists: 0 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.create_if_missing: 0 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.paranoid_checks: 1 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T08:24:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T08:24:52.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.env: 0x55ecda5e5280 2024-07-26T08:24:52.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T08:24:52.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.info_log: 0x55ecdfa1b080 2024-07-26T08:24:52.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.statistics: (nil) 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.use_fsync: 0 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_log_file_size: 0 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T08:24:52.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.allow_fallocate: 1 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.use_direct_reads: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.db_log_dir: 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.wal_dir: 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T08:24:52.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.write_buffer_manager: 0x55ecdfa1f5e0 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T08:24:52.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.unordered_write: 0 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.row_cache: None 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.wal_filter: None 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T08:24:52.530 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.two_write_queues: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.wal_compression: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.atomic_flush: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.log_readahead_size: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T08:24:52.531 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_background_jobs: 2 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_background_compactions: -1 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_subcompactions: 1 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T08:24:52.532 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_open_files: -1 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_background_flushes: -1 2024-07-26T08:24:52.533 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Compression algorithms supported: 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kZSTD supported: 0 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kXpressCompression supported: 0 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kBZip2Compression supported: 0 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kLZ4Compression supported: 1 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kZlibCompression supported: 1 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: kSnappyCompression supported: 1 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T08:24:52.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.merge_operator: 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_filter: None 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ecdfa1ac80) 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T08:24:52.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: index_type: 0 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: data_block_index_type: 0 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: index_shortening: 1 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: checksum: 4 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: no_block_cache: 0 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache: 0x55ecdb9df610 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache_options: 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: capacity : 536870912 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: num_shard_bits : 4 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: strict_capacity_limit : 0 2024-07-26T08:24:52.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache_compressed: (nil) 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: persistent_cache: (nil) 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_size: 4096 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_size_deviation: 10 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_restart_interval: 16 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: index_block_restart_interval: 1 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: metadata_block_size: 4096 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: partition_filters: 0 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: use_delta_encoding: 1 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: filter_policy: bloomfilter 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: whole_key_filtering: 1 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: verify_compression: 0 2024-07-26T08:24:52.537 INFO:journalctl@ceph.mon.b.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: format_version: 5 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: enable_index_compression: 1 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_align: 0 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: max_auto_readahead_size: 262144 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: prepopulate_block_cache: 0 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: initial_auto_readahead_size: 8192 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression: NoCompression 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T08:24:52.538 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T08:24:52.539 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T08:24:52.539 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.num_levels: 7 2024-07-26T08:24:52.539 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T08:24:52.539 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T08:24:52.539 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T08:24:52.542 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T08:24:52.542 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T08:24:52.542 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T08:24:52.542 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T08:24:52.542 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T08:24:52.542 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T08:24:52.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T08:24:52.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T08:24:52.545 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T08:24:52.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.inplace_update_support: 0 2024-07-26T08:24:52.547 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.bloom_locality: 0 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.max_successive_merges: 0 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T08:24:52.548 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T08:24:52.549 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T08:24:52.549 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.ttl: 2592000 2024-07-26T08:24:52.549 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T08:24:52.549 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T08:24:52.549 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.enable_blob_files: false 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.min_blob_size: 0 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T08:24:52.550 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 03bb1d9d-ef51-431f-bc8b-a2f48b08d506 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982292245168, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982292245595, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721982292, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "03bb1d9d-ef51-431f-bc8b-a2f48b08d506", "db_session_id": "C4XGZEUS3CTOTV0IRUKA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721982292245698, "job": 1, "event": "recovery_finished"} 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T08:24:52.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ecdfa3ca80 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: DB pointer 0x55ece07dc000 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** DB Stats ** 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-07-26T08:24:52.552 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** Compaction Stats [default] ** 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** Compaction Stats [default] ** 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T08:24:52.553 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Block cache BinnedLRUCache@0x55ecdb9df610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-07-26T08:24:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: using public_addr v2:172.21.15.116:0/0 -> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: starting mon.b rank -1 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(???) e0 preinit fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).mds e1 new map 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).mds e1 print_map 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: e1 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: btime 2024-07-26T08:22:42:160289+0000 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: legacy client fscid: -1 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-07-26T08:24:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout: No filesystems configured 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.556 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Added host smithi116 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:52.557 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi116:/etc/ceph/ceph.conf 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Deploying cephadm binary to smithi171 2024-07-26T08:24:52.558 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Added host smithi171 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.559 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi171:/etc/ceph/ceph.conf 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2231056346' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T08:24:52.560 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2231056346' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c;count:3 2024-07-26T08:24:52.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Deploying daemon mon.c on smithi171 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/1187642105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.a calling monitor election 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.c calling monitor election 2024-07-26T08:24:52.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: monmap epoch 2 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: last_changed 2024-07-26T08:24:43.459373+0000 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: min_mon_release 19 (squid) 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: election_strategy: 1 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:24:52.563 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: fsmap 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mgrmap e13: x(active, since 57s) 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: overall HEALTH_OK 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: Deploying daemon mon.b on smithi116 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/1679059779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 ceph-mon[37097]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T08:24:52.564 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 bash[37078]: 0b9bd2d4518711991fbebf5d692937e9cf07bdf42c59b4071c66bf1af7f8e72f 2024-07-26T08:24:52.565 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:52 smithi116 systemd[1]: Started Ceph mon.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:24:57.302 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-07-26T08:24:57.302 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":3,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","modified":"2024-07-26T08:24:52.273792Z","created":"2024-07-26T08:22:37.823942Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T08:24:57.302 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 3 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: mon.c calling monitor election 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: mon.a calling monitor election 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/4010026758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: monmap epoch 3 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: last_changed 2024-07-26T08:24:52.273792+0000 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: min_mon_release 19 (squid) 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: election_strategy: 1 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:24:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: fsmap 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: mgrmap e13: x(active, since 66s) 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: overall HEALTH_OK 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:57 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: mon.c calling monitor election 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: mon.a calling monitor election 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/4010026758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: monmap epoch 3 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: last_changed 2024-07-26T08:24:52.273792+0000 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: min_mon_release 19 (squid) 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: election_strategy: 1 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: fsmap 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: mgrmap e13: x(active, since 66s) 2024-07-26T08:24:57.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: overall HEALTH_OK 2024-07-26T08:24:57.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:57 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:57.950 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-26T08:24:57.950 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config generate-minimal-conf 2024-07-26T08:24:58.414 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:24:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:58 smithi171 ceph-mon[37789]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:58 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:24:58 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:58 smithi044 ceph-mon[32404]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:58.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:58 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:58.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:24:58 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: mon.c calling monitor election 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: mon.a calling monitor election 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/4010026758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: monmap epoch 3 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: last_changed 2024-07-26T08:24:52.273792+0000 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: min_mon_release 19 (squid) 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: election_strategy: 1 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:24:59.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: fsmap 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: mgrmap e13: x(active, since 66s) 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: overall HEALTH_OK 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:59.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:24:59 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:00.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: mon.b calling monitor election 2024-07-26T08:25:00.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: mon.b calling monitor election 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: mon.c calling monitor election 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: mon.a calling monitor election 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: monmap epoch 3 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: last_changed 2024-07-26T08:24:52.273792+0000 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:25:00.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: min_mon_release 19 (squid) 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: election_strategy: 1 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: fsmap 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: mgrmap e13: x(active, since 68s) 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: overall HEALTH_OK 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:00.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:00.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:00 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: mon.b calling monitor election 2024-07-26T08:25:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: mon.b calling monitor election 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: mon.c calling monitor election 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: mon.a calling monitor election 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: monmap epoch 3 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: last_changed 2024-07-26T08:24:52.273792+0000 2024-07-26T08:25:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: min_mon_release 19 (squid) 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: election_strategy: 1 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: fsmap 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: mgrmap e13: x(active, since 68s) 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: overall HEALTH_OK 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:00.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:00.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:00 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: mon.b calling monitor election 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: mon.b calling monitor election 2024-07-26T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: mon.c calling monitor election 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: mon.a calling monitor election 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: monmap epoch 3 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: last_changed 2024-07-26T08:24:52.273792+0000 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: created 2024-07-26T08:22:37.823942+0000 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: min_mon_release 19 (squid) 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: election_strategy: 1 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-07-26T08:25:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: fsmap 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: mgrmap e13: x(active, since 68s) 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: overall HEALTH_OK 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:00.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:00 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:00.800 INFO:teuthology.orchestra.run.smithi044.stdout:# minimal ceph.conf for 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:25:00.801 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-07-26T08:25:00.801 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:25:00.801 INFO:teuthology.orchestra.run.smithi044.stdout: mon_host = [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2024-07-26T08:25:01.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:01 smithi171 ceph-mon[37789]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T08:25:01.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:01 smithi171 ceph-mon[37789]: Reconfiguring daemon mon.a on smithi044 2024-07-26T08:25:01.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:01 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:01.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:01 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3196126231' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:01.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:01 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:01 smithi044 ceph-mon[32404]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T08:25:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:01 smithi044 ceph-mon[32404]: Reconfiguring daemon mon.a on smithi044 2024-07-26T08:25:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:01 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:01 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3196126231' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:01 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:01 smithi116 ceph-mon[37097]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T08:25:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:01 smithi116 ceph-mon[37097]: Reconfiguring daemon mon.a on smithi044 2024-07-26T08:25:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:01 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:01 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3196126231' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:01 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:02.033 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-26T08:25:02.033 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:25:02.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T08:25:02.116 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:25:02.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:02.246 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:25:02.246 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T08:25:02.279 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:25:02.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:02.345 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:25:02.346 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T08:25:02.373 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:25:02.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:02.437 INFO:tasks.cephadm:enabling module snap_schedule 2024-07-26T08:25:02.437 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-07-26T08:25:02.510 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:02 smithi044 ceph-mon[32404]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:02 smithi116 ceph-mon[37097]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:02 smithi171 ceph-mon[37789]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:02.907 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:03.604 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:03.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:03.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: Reconfiguring mon.b (monmap changed)... 2024-07-26T08:25:03.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:03.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:03.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:03.605 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:03 smithi044 ceph-mon[32404]: Reconfiguring daemon mon.b on smithi116 2024-07-26T08:25:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: Reconfiguring mon.b (monmap changed)... 2024-07-26T08:25:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:03 smithi171 ceph-mon[37789]: Reconfiguring daemon mon.b on smithi116 2024-07-26T08:25:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: Reconfiguring mon.b (monmap changed)... 2024-07-26T08:25:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:03 smithi116 ceph-mon[37097]: Reconfiguring daemon mon.b on smithi116 2024-07-26T08:25:04.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:04 smithi044 ceph-mon[32404]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:04.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:04 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:04.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:04 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:04.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:04 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:04.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:04 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:04.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:04 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:04 smithi171 ceph-mon[37789]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:04 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:04 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:04 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:04 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:04 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:04 smithi116 ceph-mon[37097]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:04 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:04 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:04 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T08:25:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:04 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T08:25:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:04 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: Reconfiguring mon.c (monmap changed)... 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: Reconfiguring daemon mon.c on smithi171 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3540707171' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:25:05.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:05 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: Reconfiguring mon.c (monmap changed)... 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: Reconfiguring daemon mon.c on smithi171 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3540707171' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:25:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:05 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: Reconfiguring mon.c (monmap changed)... 2024-07-26T08:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: Reconfiguring daemon mon.c on smithi171 2024-07-26T08:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='mgr.14150 172.21.15.44:0/411358698' entity='mgr.x' 2024-07-26T08:25:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3540707171' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:25:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:05 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:25:06.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:06 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ignoring --setuser ceph since I am not root 2024-07-26T08:25:06.516 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:06 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ignoring --setgroup ceph since I am not root 2024-07-26T08:25:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:06 smithi171 ceph-mon[37789]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:06 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T08:25:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:06 smithi171 ceph-mon[37789]: mgrmap e14: x(active, since 75s) 2024-07-26T08:25:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:06 smithi116 ceph-mon[37097]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:06 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T08:25:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:06 smithi116 ceph-mon[37097]: mgrmap e14: x(active, since 75s) 2024-07-26T08:25:06.923 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:06 smithi044 ceph-mon[32404]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:06.923 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:06 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T08:25:06.923 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:06 smithi044 ceph-mon[32404]: mgrmap e14: x(active, since 75s) 2024-07-26T08:25:06.994 INFO:tasks.cephadm:Adding mgr.x on smithi044 2024-07-26T08:25:06.994 INFO:tasks.cephadm:Adding mgr.y on smithi116 2024-07-26T08:25:06.994 INFO:tasks.cephadm:Adding mgr.z on smithi171 2024-07-26T08:25:06.995 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch apply mgr '3;smithi044=x;smithi116=y;smithi171=z' 2024-07-26T08:25:07.255 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:25:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: Active manager daemon x restarted 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: Activating manager daemon x 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: mgrmap e15: x(active, starting, since 0.0133878s) 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T08:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:12 smithi044 ceph-mon[32404]: Manager daemon x is now available 2024-07-26T08:25:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: Active manager daemon x restarted 2024-07-26T08:25:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: Activating manager daemon x 2024-07-26T08:25:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:25:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: mgrmap e15: x(active, starting, since 0.0133878s) 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T08:25:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:12 smithi116 ceph-mon[37097]: Manager daemon x is now available 2024-07-26T08:25:13.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: Active manager daemon x restarted 2024-07-26T08:25:13.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: Activating manager daemon x 2024-07-26T08:25:13.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:25:13.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:25:13.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: mgrmap e15: x(active, starting, since 0.0133878s) 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T08:25:13.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:12 smithi171 ceph-mon[37789]: Manager daemon x is now available 2024-07-26T08:25:13.601 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mgr update... 2024-07-26T08:25:13.889 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:13.890 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:25:13.890 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:25:13.890 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:25:13.890 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:25:13.890 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: mgrmap e16: x(active, since 1.01399s) 2024-07-26T08:25:13.890 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:25:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:25:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:25:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:25:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: mgrmap e16: x(active, since 1.01399s) 2024-07-26T08:25:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:14.042 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:14.042 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:25:14.042 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T08:25:14.043 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:25:14.043 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T08:25:14.043 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: mgrmap e16: x(active, since 1.01399s) 2024-07-26T08:25:14.043 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:14.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:25:14.496+0000 7f4553800640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:08:25:14] ENGINE Error in HTTPServer.serve 2024-07-26T08:25:14.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: Traceback (most recent call last): 2024-07-26T08:25:14.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:25:14.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._connections.run(self.expiration_interval) 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._run(expiration_interval) 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: s = self.context.wrap_socket( 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: return self.sslsocket_class._create( 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:25:14.768 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self.do_handshake() 2024-07-26T08:25:14.769 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:25:14.769 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: self._sslobj.do_handshake() 2024-07-26T08:25:14.769 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:25:14.769 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:25:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:25:14.798 DEBUG:teuthology.orchestra.run.smithi116:mgr.y> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.y.service 2024-07-26T08:25:14.800 DEBUG:teuthology.orchestra.run.smithi171:mgr.z> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mgr.z.service 2024-07-26T08:25:14.802 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-26T08:25:14.802 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-26T08:25:14.807 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-26T08:25:14.807 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set client debug_client 20 2024-07-26T08:25:14.851 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-26T08:25:14.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set client debug_ms 1 2024-07-26T08:25:14.898 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-26T08:25:14.899 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-26T08:25:14.945 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-26T08:25:14.945 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-26T08:25:14.992 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-26T08:25:14.992 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-26T08:25:15.039 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-26T08:25:15.039 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-26T08:25:15.087 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-26T08:25:15.087 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-26T08:25:15.142 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-26T08:25:15.142 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-26T08:25:15.195 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-26T08:25:15.195 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-26T08:25:15.252 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-26T08:25:15.252 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-26T08:25:15.309 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-26T08:25:15.309 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-26T08:25:15.366 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-26T08:25:15.366 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-26T08:25:15.421 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-26T08:25:15.421 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: Saving service mgr spec with placement smithi044=x;smithi116=y;smithi171=z;count:3 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: [26/Jul/2024:08:25:14] ENGINE Bus STARTING 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: [26/Jul/2024:08:25:14] ENGINE Serving on http://172.21.15.44:8765 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.450 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-26T08:25:15.450 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-26T08:25:15.504 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-26T08:25:15.504 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-26T08:25:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: Saving service mgr spec with placement smithi044=x;smithi116=y;smithi171=z;count:3 2024-07-26T08:25:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: [26/Jul/2024:08:25:14] ENGINE Bus STARTING 2024-07-26T08:25:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: [26/Jul/2024:08:25:14] ENGINE Serving on http://172.21.15.44:8765 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.564 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-26T08:25:15.564 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-26T08:25:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: Saving service mgr spec with placement smithi044=x;smithi116=y;smithi171=z;count:3 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: [26/Jul/2024:08:25:14] ENGINE Bus STARTING 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: [26/Jul/2024:08:25:14] ENGINE Serving on http://172.21.15.44:8765 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:15.597 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:15.605 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T08:25:15.605 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-26T08:25:15.660 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T08:25:15.660 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-26T08:25:15.712 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-07-26T08:25:15.713 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mgr debug_cephsqlite 20 2024-07-26T08:25:15.727 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:15.739 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-07-26T08:25:15.740 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-07-26T08:25:15.794 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-07-26T08:25:15.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-07-26T08:25:15.850 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-07-26T08:25:15.850 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-07-26T08:25:15.901 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-26T08:25:15.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-26T08:25:15.963 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T08:25:15.964 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T08:25:16.022 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-26T08:25:16.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-26T08:25:16.089 DEBUG:tasks.cephadm:set 26 configs 2024-07-26T08:25:16.089 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi044') 2024-07-26T08:25:16.530 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:16 smithi044 ceph-mon[32404]: [26/Jul/2024:08:25:14] ENGINE Serving on https://172.21.15.44:7150 2024-07-26T08:25:16.530 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:16 smithi044 ceph-mon[32404]: [26/Jul/2024:08:25:14] ENGINE Bus STARTED 2024-07-26T08:25:16.530 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:16 smithi044 ceph-mon[32404]: [26/Jul/2024:08:25:14] ENGINE Error in HTTPServer.serve 2024-07-26T08:25:16.530 INFO:journalctl@ceph.mon.a.smithi044.stdout: Traceback (most recent call last): 2024-07-26T08:25:16.530 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:25:16.530 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._run(expiration_interval) 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: s = self.context.wrap_socket( 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: return self.sslsocket_class._create( 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: self.do_handshake() 2024-07-26T08:25:16.531 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:25:16.532 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._sslobj.do_handshake() 2024-07-26T08:25:16.532 INFO:journalctl@ceph.mon.a.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:25:16.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:16 smithi044 ceph-mon[32404]: mgrmap e17: x(active, since 2s) 2024-07-26T08:25:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:16 smithi116 ceph-mon[37097]: [26/Jul/2024:08:25:14] ENGINE Serving on https://172.21.15.44:7150 2024-07-26T08:25:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:16 smithi116 ceph-mon[37097]: [26/Jul/2024:08:25:14] ENGINE Bus STARTED 2024-07-26T08:25:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:16 smithi116 ceph-mon[37097]: [26/Jul/2024:08:25:14] ENGINE Error in HTTPServer.serve 2024-07-26T08:25:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-07-26T08:25:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:25:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout: self._connections.run(self.expiration_interval) 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: self._run(expiration_interval) 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: s = self.context.wrap_socket( 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: return self.sslsocket_class._create( 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: self.do_handshake() 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout: self._sslobj.do_handshake() 2024-07-26T08:25:16.774 INFO:journalctl@ceph.mon.b.smithi116.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:25:16.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:16 smithi116 ceph-mon[37097]: mgrmap e17: x(active, since 2s) 2024-07-26T08:25:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:16 smithi171 ceph-mon[37789]: [26/Jul/2024:08:25:14] ENGINE Serving on https://172.21.15.44:7150 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:16 smithi171 ceph-mon[37789]: [26/Jul/2024:08:25:14] ENGINE Bus STARTED 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:16 smithi171 ceph-mon[37789]: [26/Jul/2024:08:25:14] ENGINE Error in HTTPServer.serve 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: Traceback (most recent call last): 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: self._connections.run(self.expiration_interval) 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: self._run(expiration_interval) 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T08:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: s = self.context.wrap_socket( 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: return self.sslsocket_class._create( 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: self.do_handshake() 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: self._sslobj.do_handshake() 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T08:25:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:16 smithi171 ceph-mon[37789]: mgrmap e17: x(active, since 2s) 2024-07-26T08:25:18.289 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.290 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.325 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi044:/etc/ceph/ceph.conf 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi116:/etc/ceph/ceph.conf 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi171:/etc/ceph/ceph.conf 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:18 smithi044 ceph-mon[32404]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi044:/etc/ceph/ceph.conf 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi116:/etc/ceph/ceph.conf 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi171:/etc/ceph/ceph.conf 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:18 smithi116 ceph-mon[37097]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi044:/etc/ceph/ceph.conf 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi116:/etc/ceph/ceph.conf 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi171:/etc/ceph/ceph.conf 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T08:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi116:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:18.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:18 smithi171 ceph-mon[37789]: Updating smithi171:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:18.889 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.909 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.917 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.918 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:18.977 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:19.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.811 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.811 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.811 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:19 smithi044 ceph-mon[32404]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:19 smithi171 ceph-mon[37789]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:19 smithi116 ceph-mon[37097]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-07-26T08:25:20.849 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:20.865 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:21.294 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:21.294 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:21.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:21.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:21.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:21 smithi044 ceph-mon[32404]: Deploying daemon mgr.z on smithi171 2024-07-26T08:25:21.302 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:21 smithi171 ceph-mon[37789]: Deploying daemon mgr.z on smithi171 2024-07-26T08:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: Updating smithi044:/var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T08:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T08:25:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:21 smithi116 ceph-mon[37097]: Deploying daemon mgr.z on smithi171 2024-07-26T08:25:22.084 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:21 smithi171 systemd[1]: Starting Ceph mgr.z for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:25:22.337 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 podman[40618]: 2024-07-26 08:25:21.985917999 +0000 UTC m=+0.016672350 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:25:22.337 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 podman[40618]: 2024-07-26 08:25:22.092017264 +0000 UTC m=+0.122771617 container create b5c1d34d81604bd62320349b74e353097dacf2000f4f4076a6215c8d4d64391b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:25:22.337 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 podman[40618]: 2024-07-26 08:25:22.328082133 +0000 UTC m=+0.358836486 container init b5c1d34d81604bd62320349b74e353097dacf2000f4f4076a6215c8d4d64391b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-z, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-26T08:25:22.337 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 podman[40618]: 2024-07-26 08:25:22.330973298 +0000 UTC m=+0.361727655 container start b5c1d34d81604bd62320349b74e353097dacf2000f4f4076a6215c8d4d64391b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-z, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T08:25:22.837 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 ceph-mgr[40637]: -- 172.21.15.171:0/1911055219 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d7f0db0e0 con 0x563d83e7a000 2024-07-26T08:25:22.837 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 bash[40618]: b5c1d34d81604bd62320349b74e353097dacf2000f4f4076a6215c8d4d64391b 2024-07-26T08:25:22.837 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Jul 26 08:25:22 smithi171 systemd[1]: Started Ceph mgr.z for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:25:23.633 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.633 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.633 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.633 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:23.633 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:23.633 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T08:25:23.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:23.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:23.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:23 smithi044 ceph-mon[32404]: Deploying daemon mgr.y on smithi116 2024-07-26T08:25:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T08:25:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:23 smithi116 ceph-mon[37097]: Deploying daemon mgr.y on smithi116 2024-07-26T08:25:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:23 smithi171 ceph-mon[37789]: Deploying daemon mgr.y on smithi116 2024-07-26T08:25:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:24 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:24 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:24 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:25.972 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:25.972 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:26.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:27.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:27.392 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T08:25:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: Reconfiguring daemon mgr.x on smithi044 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: Standby manager daemon z started 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T08:25:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T08:25:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: Reconfiguring daemon mgr.x on smithi044 2024-07-26T08:25:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T08:25:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T08:25:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: Standby manager daemon z started 2024-07-26T08:25:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T08:25:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T08:25:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: Reconfiguring daemon mgr.x on smithi044 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: Standby manager daemon z started 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.171:0/3713835601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T08:25:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:29 smithi044 ceph-mon[32404]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T08:25:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T08:25:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:29 smithi116 ceph-mon[37097]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T08:25:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T08:25:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:29 smithi171 ceph-mon[37789]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T08:25:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T08:25:30.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:30 smithi044 ceph-mon[32404]: Standby manager daemon y started 2024-07-26T08:25:30.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:30 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T08:25:30.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:30 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T08:25:30.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:30 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T08:25:30.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:30 smithi044 ceph-mon[32404]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T08:25:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:30 smithi116 ceph-mon[37097]: Standby manager daemon y started 2024-07-26T08:25:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:30 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T08:25:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:30 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T08:25:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:30 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T08:25:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:30 smithi116 ceph-mon[37097]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T08:25:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:30 smithi171 ceph-mon[37789]: Standby manager daemon y started 2024-07-26T08:25:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:30 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T08:25:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:30 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T08:25:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:30 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T08:25:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:30 smithi171 ceph-mon[37789]: from='mgr.? 172.21.15.116:0/2469336037' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T08:25:30.872 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:31.356 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T08:25:31.356 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:31 smithi044 ceph-mon[32404]: mgrmap e19: x(active, since 17s), standbys: z, y 2024-07-26T08:25:31.356 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:31 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2608165188' entity='client.admin' 2024-07-26T08:25:31.487 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T08:25:31.487 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:31 smithi171 ceph-mon[37789]: mgrmap e19: x(active, since 17s), standbys: z, y 2024-07-26T08:25:31.487 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:31 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2608165188' entity='client.admin' 2024-07-26T08:25:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T08:25:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:31 smithi116 ceph-mon[37097]: mgrmap e19: x(active, since 17s), standbys: z, y 2024-07-26T08:25:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:31 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2608165188' entity='client.admin' 2024-07-26T08:25:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:33 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:33 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:33 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:34.420 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:34 smithi044 ceph-mon[32404]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:34 smithi171 ceph-mon[37789]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:34 smithi116 ceph-mon[37097]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:35.655 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:35.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:35 smithi044 ceph-mon[32404]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:35.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:35 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:35.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.727+0000 7fe64a000640 1 -- 172.21.15.44:0/1110407640 <== mon.2 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v14)=0 v14) ==== 109+0+0 (secure 0 0 0) 0x7fe6400afa30 con 0x7fe64c1055d0 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 -- 172.21.15.44:0/1110407640 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe6340b3b90 msgr2=0x7fe6340b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 --2- 172.21.15.44:0/1110407640 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe6340b3b90 0x7fe6340b60a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe64c104890 tx=0x7fe638005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 -- 172.21.15.44:0/1110407640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe64c1055d0 msgr2=0x7fe64c19be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 --2- 172.21.15.44:0/1110407640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe64c1055d0 0x7fe64c19be10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe64000eb00 tx=0x7fe64000c2d0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 -- 172.21.15.44:0/1110407640 shutdown_connections 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 --2- 172.21.15.44:0/1110407640 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe6340b3b90 0x7fe6340b60a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 --2- 172.21.15.44:0/1110407640 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe64c10b860 0x7fe64c1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 --2- 172.21.15.44:0/1110407640 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe64c10af10 0x7fe64c19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 --2- 172.21.15.44:0/1110407640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe64c1055d0 0x7fe64c19be10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.730+0000 7fe652c00640 1 -- 172.21.15.44:0/1110407640 >> 172.21.15.44:0/1110407640 conn(0x7fe64c101040 msgr2=0x7fe64c10a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:35.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.731+0000 7fe652c00640 1 -- 172.21.15.44:0/1110407640 shutdown_connections 2024-07-26T08:25:35.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.731+0000 7fe652c00640 1 -- 172.21.15.44:0/1110407640 wait complete. 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.761+0000 7f6965800640 1 -- 172.21.15.44:0/978126000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6960104af0 msgr2=0x7f696010d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.761+0000 7f6965800640 1 --2- 172.21.15.44:0/978126000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6960104af0 0x7f696010d130 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f694c009900 tx=0x7f694c02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 -- 172.21.15.44:0/978126000 shutdown_connections 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 --2- 172.21.15.44:0/978126000 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 0x7f696010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 --2- 172.21.15.44:0/978126000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6960104af0 0x7f696010d130 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 --2- 172.21.15.44:0/978126000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69601040c0 0x7f69601044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 -- 172.21.15.44:0/978126000 >> 172.21.15.44:0/978126000 conn(0x7f69600fdcf0 msgr2=0x7f6960100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 -- 172.21.15.44:0/978126000 shutdown_connections 2024-07-26T08:25:35.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.762+0000 7f6965800640 1 -- 172.21.15.44:0/978126000 wait complete. 2024-07-26T08:25:35.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.763+0000 7f6965800640 1 Processor -- start 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 -- start start 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69601040c0 0x7f696019bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6960104af0 0x7f696019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 0x7f69601a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f69601132f0 con 0x7f69601040c0 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6960113170 con 0x7f6960104af0 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f6965800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6960113470 con 0x7f696010d670 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f695fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69601040c0 0x7f696019bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:35.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f695fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69601040c0 0x7f696019bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58654/0 (socket says 172.21.15.44:58654) 2024-07-26T08:25:35.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.764+0000 7f695fe00640 1 -- 172.21.15.44:0/1145345508 learned_addr learned my addr 172.21.15.44:0/1145345508 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:35.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.765+0000 7f6964e00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 0x7f69601a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:35.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.765+0000 7f6964e00640 1 -- 172.21.15.44:0/1145345508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6960104af0 msgr2=0x7f696019c460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T08:25:35.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.765+0000 7f6964e00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6960104af0 0x7f696019c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.765+0000 7f6964e00640 1 -- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69601040c0 msgr2=0x7f696019bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:35.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.765+0000 7f6964e00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69601040c0 0x7f696019bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:35.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.765+0000 7f6964e00640 1 -- 172.21.15.44:0/1145345508 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69601a1e30 con 0x7f696010d670 2024-07-26T08:25:35.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.772+0000 7f6964e00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 0x7f69601a1700 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f695800e940 tx=0x7f695800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.772+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958004d20 con 0x7f696010d670 2024-07-26T08:25:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.772+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6958005020 con 0x7f696010d670 2024-07-26T08:25:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.772+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69580116d0 con 0x7f696010d670 2024-07-26T08:25:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.772+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69601a2120 con 0x7f696010d670 2024-07-26T08:25:35.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.772+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69601a8880 con 0x7f696010d670 2024-07-26T08:25:35.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.774+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69601044d0 con 0x7f696010d670 2024-07-26T08:25:35.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.776+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f6958005460 con 0x7f696010d670 2024-07-26T08:25:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.778+0000 7f695cc00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69440b3c10 0x7f69440b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.778+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f69580e25c0 con 0x7f696010d670 2024-07-26T08:25:35.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.778+0000 7f695fe00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69440b3c10 0x7f69440b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:35.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.779+0000 7f695fe00640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69440b3c10 0x7f69440b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f69500095f0 tx=0x7f6950005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:35.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.779+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6958016030 con 0x7f696010d670 2024-07-26T08:25:35.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.991+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f696019cff0 con 0x7f696010d670 2024-07-26T08:25:36.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:35.999+0000 7f695cc00640 1 -- 172.21.15.44:0/1145345508 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v15)=0 v15) ==== 116+0+0 (secure 0 0 0) 0x7f69580ab7f0 con 0x7f696010d670 2024-07-26T08:25:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.002+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69440b3c10 msgr2=0x7f69440b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.002+0000 7f6965800640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69440b3c10 0x7f69440b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f69500095f0 tx=0x7f6950005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:36.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.002+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 msgr2=0x7f69601a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.002+0000 7f6965800640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 0x7f69601a1700 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f695800e940 tx=0x7f695800ee00 comp rx=0 tx=0).stop 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.002+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 shutdown_connections 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69440b3c10 0x7f69440b6120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f696010d670 0x7f69601a1700 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6960104af0 0x7f696019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 --2- 172.21.15.44:0/1145345508 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f69601040c0 0x7f696019bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 >> 172.21.15.44:0/1145345508 conn(0x7f69600fdcf0 msgr2=0x7f6960101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 shutdown_connections 2024-07-26T08:25:36.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:36.003+0000 7f6965800640 1 -- 172.21.15.44:0/1145345508 wait complete. 2024-07-26T08:25:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:35 smithi116 ceph-mon[37097]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:35 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:35 smithi171 ceph-mon[37789]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:35 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:36.983 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:36 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:36.983 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:36 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:36 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:36 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:36 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:36 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:38 smithi044 ceph-mon[32404]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:38 smithi116 ceph-mon[37097]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:38 smithi171 ceph-mon[37789]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:40.139 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:40 smithi044 ceph-mon[32404]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:40.139 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:40 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:40 smithi116 ceph-mon[37097]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:40 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:40 smithi171 ceph-mon[37789]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:40 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 -- 172.21.15.44:0/1473645892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 msgr2=0x7f1bd8075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 --2- 172.21.15.44:0/1473645892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd8075c10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009900 tx=0x7f1bc802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 -- 172.21.15.44:0/1473645892 shutdown_connections 2024-07-26T08:25:40.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 --2- 172.21.15.44:0/1473645892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 0x7f1bd8116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 --2- 172.21.15.44:0/1473645892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd8075c10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 --2- 172.21.15.44:0/1473645892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8076db0 0x7f1bd80751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 -- 172.21.15.44:0/1473645892 >> 172.21.15.44:0/1473645892 conn(0x7f1bd80fee70 msgr2=0x7f1bd81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 -- 172.21.15.44:0/1473645892 shutdown_connections 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.618+0000 7f1bdf200640 1 -- 172.21.15.44:0/1473645892 wait complete. 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.620+0000 7f1bdf200640 1 Processor -- start 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.620+0000 7f1bdf200640 1 -- start start 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdf200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd81a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdf200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8076db0 0x7f1bd81a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdf200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 0x7f1bd81a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdf200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8119940 con 0x7f1bd8076db0 2024-07-26T08:25:40.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdf200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd81197c0 con 0x7f1bd8075800 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdf200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8119ac0 con 0x7f1bd8113cb0 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8076db0 0x7f1bd81a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd81a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd81a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39668/0 (socket says 172.21.15.44:39668) 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bdde00640 1 -- 172.21.15.44:0/3633472134 learned_addr learned my addr 172.21.15.44:0/3633472134 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.621+0000 7f1bde800640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 0x7f1bd81a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.622+0000 7f1bde800640 1 -- 172.21.15.44:0/3633472134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 msgr2=0x7f1bd81a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.622+0000 7f1bde800640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd81a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.622+0000 7f1bde800640 1 -- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8076db0 msgr2=0x7f1bd81a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.622+0000 7f1bde800640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8076db0 0x7f1bd81a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.622+0000 7f1bde800640 1 -- 172.21.15.44:0/3633472134 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd81a6280 con 0x7f1bd8113cb0 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.622+0000 7f1bde800640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 0x7f1bd81a5b50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009fd0 tx=0x7f1bd400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:40.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.623+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd400ce20 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.623+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bd4005020 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.623+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4010690 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.623+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd81a6570 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.623+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd81accd0 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.624+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd80766f0 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.627+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f1bd40054d0 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.628+0000 7f1bd2a00640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1bc00b3c10 0x7f1bc00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.628+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1bd40e2600 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.629+0000 7f1bdde00640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1bc00b3c10 0x7f1bc00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.630+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1bd40ab7b0 con 0x7f1bd8113cb0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.630+0000 7f1bdde00640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1bc00b3c10 0x7f1bc00b6120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc009c90 tx=0x7f1bcc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:40.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.836+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f1bd8075c10 con 0x7f1bd8113cb0 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.843+0000 7f1bd2a00640 1 -- 172.21.15.44:0/3633472134 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v16)=0 v16) ==== 106+0+0 (secure 0 0 0) 0x7f1bd8075c10 con 0x7f1bd8113cb0 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.847+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1bc00b3c10 msgr2=0x7f1bc00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.847+0000 7f1bdf200640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1bc00b3c10 0x7f1bc00b6120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc009c90 tx=0x7f1bcc005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.847+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 msgr2=0x7f1bd81a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.847+0000 7f1bdf200640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 0x7f1bd81a5b50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009fd0 tx=0x7f1bd400ef10 comp rx=0 tx=0).stop 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 shutdown_connections 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1bc00b3c10 0x7f1bc00b6120 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bd8113cb0 0x7f1bd81a5b50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1bd8076db0 0x7f1bd81a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 --2- 172.21.15.44:0/3633472134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd8075800 0x7f1bd81a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 >> 172.21.15.44:0/3633472134 conn(0x7f1bd80fee70 msgr2=0x7f1bd810fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 shutdown_connections 2024-07-26T08:25:40.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:39.848+0000 7f1bdf200640 1 -- 172.21.15.44:0/3633472134 wait complete. 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 -- 172.21.15.44:0/1085996737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f988810af10 msgr2=0x7f988810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 --2- 172.21.15.44:0/1085996737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f988810af10 0x7f988810b320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9870009900 tx=0x7f987002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 -- 172.21.15.44:0/1085996737 shutdown_connections 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 --2- 172.21.15.44:0/1085996737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f988810af10 0x7f988810b320 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 --2- 172.21.15.44:0/1085996737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f9888112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 --2- 172.21.15.44:0/1085996737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98881055d0 0x7f98881059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 -- 172.21.15.44:0/1085996737 >> 172.21.15.44:0/1085996737 conn(0x7f9888101040 msgr2=0x7f9888103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 -- 172.21.15.44:0/1085996737 shutdown_connections 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.797+0000 7f988ca00640 1 -- 172.21.15.44:0/1085996737 wait complete. 2024-07-26T08:25:41.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.798+0000 7f988ca00640 1 Processor -- start 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 -- start start 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98881055d0 0x7f988819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f988810af10 0x7f988819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f98881a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9888115520 con 0x7f988810b860 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98881153a0 con 0x7f988810af10 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f988ca00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f98881156a0 con 0x7f98881055d0 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f9887200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f98881a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f987de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f988810af10 0x7f988819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f9887200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f98881a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52182/0 (socket says 172.21.15.44:52182) 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.799+0000 7f9887200640 1 -- 172.21.15.44:0/898961432 learned_addr learned my addr 172.21.15.44:0/898961432 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:41.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f9886800640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98881055d0 0x7f988819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f987de00640 1 -- 172.21.15.44:0/898961432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98881055d0 msgr2=0x7f988819bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f987de00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98881055d0 0x7f988819bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f987de00640 1 -- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 msgr2=0x7f98881a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f987de00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f98881a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f987de00640 1 -- 172.21.15.44:0/898961432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98881a1ea0 con 0x7f988810af10 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f9886800640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98881055d0 0x7f988819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f9887200640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f98881a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.800+0000 7f987de00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f988810af10 0x7f988819c4d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9870009230 tx=0x7f9870003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.801+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987003f070 con 0x7f988810af10 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.801+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98881a2130 con 0x7f988810af10 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.801+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98881a88f0 con 0x7f988810af10 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.801+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f987003b910 con 0x7f988810af10 2024-07-26T08:25:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.801+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870049060 con 0x7f988810af10 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.802+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98640052c0 con 0x7f988810af10 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.805+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f9870042070 con 0x7f988810af10 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.806+0000 7f987fe00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f986c0b3b90 0x7f986c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.806+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f9870102400 con 0x7f988810af10 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.807+0000 7f9886800640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f986c0b3b90 0x7f986c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.807+0000 7f9886800640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f986c0b3b90 0x7f986c0b60a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9878009fc0 tx=0x7f9878005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.808+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f98700caf60 con 0x7f988810af10 2024-07-26T08:25:42.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.015+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f98640055b0 con 0x7f988810af10 2024-07-26T08:25:42.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.025+0000 7f987fe00640 1 -- 172.21.15.44:0/898961432 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v18)=0 v18) ==== 116+0+0 (secure 0 0 0) 0x7f98700cf4b0 con 0x7f988810af10 2024-07-26T08:25:42.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f986c0b3b90 msgr2=0x7f986c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:42.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f986c0b3b90 0x7f986c0b60a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9878009fc0 tx=0x7f9878005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:42.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f988810af10 msgr2=0x7f988819c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f988810af10 0x7f988819c4d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9870009230 tx=0x7f9870003b90 comp rx=0 tx=0).stop 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 shutdown_connections 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f986c0b3b90 0x7f986c0b60a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f988810b860 0x7f98881a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f988810af10 0x7f988819c4d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 --2- 172.21.15.44:0/898961432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98881055d0 0x7f988819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:42.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 >> 172.21.15.44:0/898961432 conn(0x7f9888101040 msgr2=0x7f9888103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:42.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 shutdown_connections 2024-07-26T08:25:42.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:42.028+0000 7f988ca00640 1 -- 172.21.15.44:0/898961432 wait complete. 2024-07-26T08:25:42.204 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi044') 2024-07-26T08:25:42.204 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi044') 2024-07-26T08:25:42.204 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi044') 2024-07-26T08:25:42.205 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi044') 2024-07-26T08:25:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:42 smithi044 ceph-mon[32404]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:42 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:42 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:42 smithi116 ceph-mon[37097]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:42 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:42 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:42 smithi171 ceph-mon[37789]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:42 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:42 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:43.248 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi044') 2024-07-26T08:25:44.450 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:44 smithi044 ceph-mon[32404]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:44 smithi116 ceph-mon[37097]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:44 smithi171 ceph-mon[37789]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:45.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 -- 172.21.15.44:0/3973337444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010b0f0 msgr2=0x7f128010b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 --2- 172.21.15.44:0/3973337444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010b0f0 0x7f128010b500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1268009900 tx=0x7f126802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 -- 172.21.15.44:0/3973337444 shutdown_connections 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 --2- 172.21.15.44:0/3973337444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f128010ba40 0x7f1280112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 --2- 172.21.15.44:0/3973337444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010b0f0 0x7f128010b500 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 --2- 172.21.15.44:0/3973337444 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f1280105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 -- 172.21.15.44:0/3973337444 >> 172.21.15.44:0/3973337444 conn(0x7f1280101240 msgr2=0x7f1280103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 -- 172.21.15.44:0/3973337444 shutdown_connections 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.871+0000 7f1286600640 1 -- 172.21.15.44:0/3973337444 wait complete. 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.872+0000 7f1286600640 1 Processor -- start 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 -- start start 2024-07-26T08:25:45.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f128019c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f128010b0f0 0x7f128019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010ba40 0x7f12801a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1280115700 con 0x7f128010ba40 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1280115580 con 0x7f128010b0f0 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.873+0000 7f1286600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1280115880 con 0x7f12801057b0 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f128019c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010ba40 0x7f12801a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f1277e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f128010b0f0 0x7f128019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f128019c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48912/0 (socket says 172.21.15.44:48912) 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 -- 172.21.15.44:0/2735835204 learned_addr learned my addr 172.21.15.44:0/2735835204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 -- 172.21.15.44:0/2735835204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f128010b0f0 msgr2=0x7f128019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f128010b0f0 0x7f128019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:45.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 -- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010ba40 msgr2=0x7f12801a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010ba40 0x7f12801a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.874+0000 7f127f400640 1 -- 172.21.15.44:0/2735835204 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12801a2060 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.875+0000 7f127f400640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f128019c0f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f127000ea60 tx=0x7f127000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.875+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127000cde0 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.875+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12801a2350 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.875+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12700035f0 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.875+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270010690 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.876+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12801a8ab0 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.877+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1280105bc0 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.879+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f1270004000 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.881+0000 7f127cc00640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f12640b3b90 0x7f12640b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.881+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f12700e2410 con 0x7f12801057b0 2024-07-26T08:25:45.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.881+0000 7f1277e00640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f12640b3b90 0x7f12640b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:45.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.882+0000 7f1277e00640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f12640b3b90 0x7f12640b60a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f126802a9e0 tx=0x7f1268031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:45.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:45.885+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f12700042f0 con 0x7f12801057b0 2024-07-26T08:25:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.092+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f128019d220 con 0x7f12801057b0 2024-07-26T08:25:46.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.101+0000 7f127cc00640 1 -- 172.21.15.44:0/2735835204 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v19)=0 v19) ==== 127+0+0 (secure 0 0 0) 0x7f12700ab5c0 con 0x7f12801057b0 2024-07-26T08:25:46.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f12640b3b90 msgr2=0x7f12640b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f12640b3b90 0x7f12640b60a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f126802a9e0 tx=0x7f1268031400 comp rx=0 tx=0).stop 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 msgr2=0x7f128019c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f128019c0f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f127000ea60 tx=0x7f127000ef20 comp rx=0 tx=0).stop 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 shutdown_connections 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f12640b3b90 0x7f12640b60a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f128010ba40 0x7f12801a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f128010b0f0 0x7f128019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 --2- 172.21.15.44:0/2735835204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12801057b0 0x7f128019c0f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 >> 172.21.15.44:0/2735835204 conn(0x7f1280101240 msgr2=0x7f1280103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 shutdown_connections 2024-07-26T08:25:46.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.104+0000 7f1286600640 1 -- 172.21.15.44:0/2735835204 wait complete. 2024-07-26T08:25:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 -- 172.21.15.44:0/972731141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc1022a0 msgr2=0x7f01dc1026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 --2- 172.21.15.44:0/972731141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc1022a0 0x7f01dc1026b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009900 tx=0x7f01cc02a410 comp rx=0 tx=0).stop 2024-07-26T08:25:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 -- 172.21.15.44:0/972731141 shutdown_connections 2024-07-26T08:25:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 --2- 172.21.15.44:0/972731141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01dc107be0 0x7f01dc107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 --2- 172.21.15.44:0/972731141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc108530 0x7f01dc10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 --2- 172.21.15.44:0/972731141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc1022a0 0x7f01dc1026b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 -- 172.21.15.44:0/972731141 >> 172.21.15.44:0/972731141 conn(0x7f01dc0fdcf0 msgr2=0x7f01dc100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 -- 172.21.15.44:0/972731141 shutdown_connections 2024-07-26T08:25:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.229+0000 7f01e0c00640 1 -- 172.21.15.44:0/972731141 wait complete. 2024-07-26T08:25:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.230+0000 7f01e0c00640 1 Processor -- start 2024-07-26T08:25:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.230+0000 7f01e0c00640 1 -- start start 2024-07-26T08:25:46.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01e0c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01dc1022a0 0x7f01dc19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01e0c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc107be0 0x7f01dc19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01e0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 0x7f01dc1a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01e0c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc113340 con 0x7f01dc108530 2024-07-26T08:25:46.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01e0c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1131c0 con 0x7f01dc107be0 2024-07-26T08:25:46.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01e0c00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1134c0 con 0x7f01dc1022a0 2024-07-26T08:25:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01dbe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 0x7f01dc1a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc107be0 0x7f01dc19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01dbe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 0x7f01dc1a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52202/0 (socket says 172.21.15.44:52202) 2024-07-26T08:25:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01dbe00640 1 -- 172.21.15.44:0/2741511834 learned_addr learned my addr 172.21.15.44:0/2741511834 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:46.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.231+0000 7f01db400640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01dc1022a0 0x7f01dc19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01dbe00640 1 -- 172.21.15.44:0/2741511834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01dc1022a0 msgr2=0x7f01dc19bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01dbe00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01dc1022a0 0x7f01dc19bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01dbe00640 1 -- 172.21.15.44:0/2741511834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc107be0 msgr2=0x7f01dc19c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01dbe00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc107be0 0x7f01dc19c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01dbe00640 1 -- 172.21.15.44:0/2741511834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc1a1ea0 con 0x7f01dc108530 2024-07-26T08:25:46.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01daa00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc107be0 0x7f01dc19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.232+0000 7f01dbe00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 0x7f01dc1a1770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01d400eb10 tx=0x7f01d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.233+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4004b20 con 0x7f01dc108530 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.233+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d4004cc0 con 0x7f01dc108530 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.233+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc1a2190 con 0x7f01dc108530 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.233+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4010670 con 0x7f01dc108530 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.233+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc1a88f0 con 0x7f01dc108530 2024-07-26T08:25:46.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.234+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01dc1026b0 con 0x7f01dc108530 2024-07-26T08:25:46.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.237+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f01d4005000 con 0x7f01dc108530 2024-07-26T08:25:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.239+0000 7f01d3e00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f01c00b3b90 0x7f01c00b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.239+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f01d40e2810 con 0x7f01dc108530 2024-07-26T08:25:46.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.239+0000 7f01db400640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f01c00b3b90 0x7f01c00b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.240+0000 7f01db400640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f01c00b3b90 0x7f01c00b60a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005ab0 tx=0x7f01cc005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:46.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.242+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f01d4055600 con 0x7f01dc108530 2024-07-26T08:25:46.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.457+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f01dc19d060 con 0x7f01dc108530 2024-07-26T08:25:46.459 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:46.459 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.459 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:46.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:46.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:46.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:46 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:46.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.464+0000 7f01d3e00640 1 -- 172.21.15.44:0/2741511834 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7f01d40ab9c0 con 0x7f01dc108530 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f01c00b3b90 msgr2=0x7f01c00b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f01c00b3b90 0x7f01c00b60a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005ab0 tx=0x7f01cc005a20 comp rx=0 tx=0).stop 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 msgr2=0x7f01dc1a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 0x7f01dc1a1770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01d400eb10 tx=0x7f01d400c2d0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 shutdown_connections 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f01c00b3b90 0x7f01c00b60a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f01dc108530 0x7f01dc1a1770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc107be0 0x7f01dc19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 --2- 172.21.15.44:0/2741511834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01dc1022a0 0x7f01dc19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.467+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 >> 172.21.15.44:0/2741511834 conn(0x7f01dc0fdcf0 msgr2=0x7f01dc1072c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:46.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.468+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 shutdown_connections 2024-07-26T08:25:46.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:46.468+0000 7f01e0c00640 1 -- 172.21.15.44:0/2741511834 wait complete. 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:46 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:46 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 -- 172.21.15.44:0/3313968949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 msgr2=0x7fb7c0112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/3313968949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 0x7fb7c0112090 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009be0 tx=0x7fb7b802a6c0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 -- 172.21.15.44:0/3313968949 shutdown_connections 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/3313968949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 0x7fb7c0112090 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/3313968949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c010af10 0x7fb7c010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/3313968949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c01055d0 0x7fb7c01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 -- 172.21.15.44:0/3313968949 >> 172.21.15.44:0/3313968949 conn(0x7fb7c0101040 msgr2=0x7fb7c0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 -- 172.21.15.44:0/3313968949 shutdown_connections 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.958+0000 7fb7c4e00640 1 -- 172.21.15.44:0/3313968949 wait complete. 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.960+0000 7fb7c4e00640 1 Processor -- start 2024-07-26T08:25:46.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.960+0000 7fb7c4e00640 1 -- start start 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7c4e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c01055d0 0x7fb7c019bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7c4e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c010af10 0x7fb7c019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7c4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 0x7fb7c01a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7c4e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0115530 con 0x7fb7c01055d0 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7c4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01153b0 con 0x7fb7c010b860 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7c4e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01156b0 con 0x7fb7c010af10 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7bf400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c01055d0 0x7fb7c019bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7bf400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c01055d0 0x7fb7c019bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58682/0 (socket says 172.21.15.44:58682) 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7bf400640 1 -- 172.21.15.44:0/1975360055 learned_addr learned my addr 172.21.15.44:0/1975360055 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7b7e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c010af10 0x7fb7c019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7bfe00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 0x7fb7c01a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7b7e00640 1 -- 172.21.15.44:0/1975360055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 msgr2=0x7fb7c01a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7b7e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 0x7fb7c01a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7b7e00640 1 -- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c01055d0 msgr2=0x7fb7c019bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.961+0000 7fb7b7e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c01055d0 0x7fb7c019bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7b7e00640 1 -- 172.21.15.44:0/1975360055 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c01a1e50 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7b7e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c010af10 0x7fb7c019c480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb7c006c270 tx=0x7fb7a800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a800cc90 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a800ee70 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c01a2140 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8018660 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.962+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c01a88a0 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.964+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c01059e0 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.966+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fb7a8010430 con 0x7fb7c010af10 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.968+0000 7fb7bcc00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7a40b3b90 0x7fb7a40b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:46.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.968+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb7a80e2380 con 0x7fb7c010af10 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.968+0000 7fb7bf400640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7a40b3b90 0x7fb7a40b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.969+0000 7fb7bf400640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7a40b3b90 0x7fb7a40b60a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009cd0 tx=0x7fb7b0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:40.969+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7a80ab530 con 0x7fb7c010af10 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.178+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fb7c019d010 con 0x7fb7c010af10 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.190+0000 7fb7bcc00640 1 -- 172.21.15.44:0/1975360055 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v17)=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7fb7a80aae30 con 0x7fb7c010af10 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7a40b3b90 msgr2=0x7fb7a40b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7a40b3b90 0x7fb7a40b60a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009cd0 tx=0x7fb7b0005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c010af10 msgr2=0x7fb7c019c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c010af10 0x7fb7c019c480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb7c006c270 tx=0x7fb7a800ec20 comp rx=0 tx=0).stop 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 shutdown_connections 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7a40b3b90 0x7fb7a40b60a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7c010af10 0x7fb7c019c480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c010b860 0x7fb7c01a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 --2- 172.21.15.44:0/1975360055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7c01055d0 0x7fb7c019bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:46.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 >> 172.21.15.44:0/1975360055 conn(0x7fb7c0101040 msgr2=0x7fb7c0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:46.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 shutdown_connections 2024-07-26T08:25:46.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:41.193+0000 7fb7c4e00640 1 -- 172.21.15.44:0/1975360055 wait complete. 2024-07-26T08:25:47.611 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:47.611 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:47.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:47 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2741511834' entity='client.admin' 2024-07-26T08:25:47.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:47 smithi044 ceph-mon[32404]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:47 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2741511834' entity='client.admin' 2024-07-26T08:25:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:47 smithi116 ceph-mon[37097]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:47 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2741511834' entity='client.admin' 2024-07-26T08:25:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:47 smithi171 ceph-mon[37789]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 -- 172.21.15.44:0/3651690753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d410af10 msgr2=0x7f15d410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 --2- 172.21.15.44:0/3651690753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d410af10 0x7f15d410b320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009900 tx=0x7f15c002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 -- 172.21.15.44:0/3651690753 shutdown_connections 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 --2- 172.21.15.44:0/3651690753 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410b860 0x7f15d4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 --2- 172.21.15.44:0/3651690753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d410af10 0x7f15d410b320 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 --2- 172.21.15.44:0/3651690753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15d41055d0 0x7f15d41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 -- 172.21.15.44:0/3651690753 >> 172.21.15.44:0/3651690753 conn(0x7f15d4101040 msgr2=0x7f15d4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.906+0000 7f15d9400640 1 -- 172.21.15.44:0/3651690753 shutdown_connections 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.907+0000 7f15d9400640 1 -- 172.21.15.44:0/3651690753 wait complete. 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.908+0000 7f15d9400640 1 Processor -- start 2024-07-26T08:25:48.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.908+0000 7f15d9400640 1 -- start start 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d9400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d41055d0 0x7f15d419bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d9400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410af10 0x7f15d419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15d410b860 0x7f15d41a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d9400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f15d4115540 con 0x7f15d41055d0 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d9400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41153c0 con 0x7f15d410b860 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d9400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f15d41156c0 con 0x7f15d410af10 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d41055d0 0x7f15d419bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d3400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410af10 0x7f15d419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d41055d0 0x7f15d419bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52226/0 (socket says 172.21.15.44:52226) 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d3e00640 1 -- 172.21.15.44:0/1477364047 learned_addr learned my addr 172.21.15.44:0/1477364047 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.909+0000 7f15d8a00640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15d410b860 0x7f15d41a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d3400640 1 -- 172.21.15.44:0/1477364047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15d410b860 msgr2=0x7f15d41a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:48.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d3400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15d410b860 0x7f15d41a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d3400640 1 -- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d41055d0 msgr2=0x7f15d419bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d3400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d41055d0 0x7f15d419bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d3400640 1 -- 172.21.15.44:0/1477364047 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d41a1ea0 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d3400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410af10 0x7f15d419c4d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009230 tx=0x7f15c0003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c003f070 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d41a2130 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c0031b50 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0035030 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.910+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d41a88f0 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.912+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d41059e0 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.914+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f15c0042070 con 0x7f15d410af10 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.916+0000 7f15d0c00640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f15b80b3c10 0x7f15b80b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:48.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.916+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f15c0102790 con 0x7f15d410af10 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.916+0000 7f15d3e00640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f15b80b3c10 0x7f15b80b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.917+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f15c00cb2f0 con 0x7f15d410af10 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:47.917+0000 7f15d3e00640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f15b80b3c10 0x7f15b80b6120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15c40095f0 tx=0x7f15c4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.130+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f15d419d060 con 0x7f15d410af10 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.137+0000 7f15d0c00640 1 -- 172.21.15.44:0/1477364047 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v21)=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f15c00cf840 con 0x7f15d410af10 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f15b80b3c10 msgr2=0x7f15b80b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f15b80b3c10 0x7f15b80b6120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15c40095f0 tx=0x7f15c4005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410af10 msgr2=0x7f15d419c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410af10 0x7f15d419c4d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009230 tx=0x7f15c0003b90 comp rx=0 tx=0).stop 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 shutdown_connections 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f15b80b3c10 0x7f15b80b6120 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15d410af10 0x7f15d419c4d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15d410b860 0x7f15d41a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 --2- 172.21.15.44:0/1477364047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15d41055d0 0x7f15d419bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:48.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 >> 172.21.15.44:0/1477364047 conn(0x7f15d4101040 msgr2=0x7f15d4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:48.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 shutdown_connections 2024-07-26T08:25:48.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:48.141+0000 7f15d9400640 1 -- 172.21.15.44:0/1477364047 wait complete. 2024-07-26T08:25:49.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:49 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:49 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:49 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:49.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 -- 172.21.15.44:0/2256017536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20108d20 msgr2=0x7fcd20109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 --2- 172.21.15.44:0/2256017536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20108d20 0x7fcd20109130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009900 tx=0x7fcd0802a410 comp rx=0 tx=0).stop 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 -- 172.21.15.44:0/2256017536 shutdown_connections 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 --2- 172.21.15.44:0/2256017536 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd20109670 0x7fcd2010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 --2- 172.21.15.44:0/2256017536 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20108d20 0x7fcd20109130 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 --2- 172.21.15.44:0/2256017536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd201033e0 0x7fcd201037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 -- 172.21.15.44:0/2256017536 >> 172.21.15.44:0/2256017536 conn(0x7fcd200fee70 msgr2=0x7fcd201012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 -- 172.21.15.44:0/2256017536 shutdown_connections 2024-07-26T08:25:49.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.838+0000 7fcd1f200640 1 -- 172.21.15.44:0/2256017536 wait complete. 2024-07-26T08:25:49.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.839+0000 7fcd1f200640 1 Processor -- start 2024-07-26T08:25:49.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.839+0000 7fcd1f200640 1 -- start start 2024-07-26T08:25:49.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1f200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 0x7fcd2019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:49.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20108d20 0x7fcd2019c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:49.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20109670 0x7fcd201a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:49.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20113350 con 0x7fcd20109670 2024-07-26T08:25:49.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201131d0 con 0x7fcd20108d20 2024-07-26T08:25:49.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1f200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201134d0 con 0x7fcd201033e0 2024-07-26T08:25:49.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1de00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 0x7fcd2019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:49.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1de00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 0x7fcd2019c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48992/0 (socket says 172.21.15.44:48992) 2024-07-26T08:25:49.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1de00640 1 -- 172.21.15.44:0/1788521324 learned_addr learned my addr 172.21.15.44:0/1788521324 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:49.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1d400640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20108d20 0x7fcd2019c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:49.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.840+0000 7fcd1e800640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20109670 0x7fcd201a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd1de00640 1 -- 172.21.15.44:0/1788521324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20108d20 msgr2=0x7fcd2019c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd1de00640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20108d20 0x7fcd2019c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd1de00640 1 -- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20109670 msgr2=0x7fcd201a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd1de00640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20109670 0x7fcd201a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd1de00640 1 -- 172.21.15.44:0/1788521324 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd201a1f70 con 0x7fcd201033e0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd1de00640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 0x7fcd2019c000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000ea60 tx=0x7fcd1000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1000cde0 con 0x7fcd201033e0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd100035f0 con 0x7fcd201033e0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.841+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10010690 con 0x7fcd201033e0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.842+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd201a2260 con 0x7fcd201033e0 2024-07-26T08:25:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.842+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd201a89c0 con 0x7fcd201033e0 2024-07-26T08:25:49.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.843+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd201037f0 con 0x7fcd201033e0 2024-07-26T08:25:49.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.845+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fcd100037a0 con 0x7fcd201033e0 2024-07-26T08:25:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.847+0000 7fcd16a00640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcd040b3c10 0x7fcd040b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.847+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fcd100e22f0 con 0x7fcd201033e0 2024-07-26T08:25:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.847+0000 7fcd1d400640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcd040b3c10 0x7fcd040b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:49.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.848+0000 7fcd1d400640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcd040b3c10 0x7fcd040b6120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcd0802aa00 tx=0x7fcd08005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:49.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:49.851+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcd100ab4a0 con 0x7fcd201033e0 2024-07-26T08:25:50.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.057+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fcd2019d130 con 0x7fcd201033e0 2024-07-26T08:25:50.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.067+0000 7fcd16a00640 1 -- 172.21.15.44:0/1788521324 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v22)=0 v22) ==== 138+0+0 (secure 0 0 0) 0x7fcd100aada0 con 0x7fcd201033e0 2024-07-26T08:25:50.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.070+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcd040b3c10 msgr2=0x7fcd040b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.070+0000 7fcd1f200640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcd040b3c10 0x7fcd040b6120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcd0802aa00 tx=0x7fcd08005a20 comp rx=0 tx=0).stop 2024-07-26T08:25:50.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.070+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 msgr2=0x7fcd2019c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.070+0000 7fcd1f200640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 0x7fcd2019c000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000ea60 tx=0x7fcd1000ef20 comp rx=0 tx=0).stop 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 shutdown_connections 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcd040b3c10 0x7fcd040b6120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd20109670 0x7fcd201a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20108d20 0x7fcd2019c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 --2- 172.21.15.44:0/1788521324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd201033e0 0x7fcd2019c000 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 >> 172.21.15.44:0/1788521324 conn(0x7fcd200fee70 msgr2=0x7fcd201083a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 shutdown_connections 2024-07-26T08:25:50.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.071+0000 7fcd1f200640 1 -- 172.21.15.44:0/1788521324 wait complete. 2024-07-26T08:25:50.455 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:50 smithi044 ceph-mon[32404]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:50.455 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:50 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:50.455 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:50.455 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:50.455 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:50.456 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:50 smithi116 ceph-mon[37097]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:50 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:50 smithi171 ceph-mon[37789]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:50 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:50.609 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:50.610 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:50.611 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.551+0000 7f8b34a00640 1 -- 172.21.15.44:0/571291068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b301033c0 msgr2=0x7f8b301037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.551+0000 7f8b34a00640 1 --2- 172.21.15.44:0/571291068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b301033c0 0x7f8b301037d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009ca0 tx=0x7f8b2002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 -- 172.21.15.44:0/571291068 shutdown_connections 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 --2- 172.21.15.44:0/571291068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30109650 0x7f8b3010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 --2- 172.21.15.44:0/571291068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b30108d00 0x7f8b30109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 --2- 172.21.15.44:0/571291068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b301033c0 0x7f8b301037d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 -- 172.21.15.44:0/571291068 >> 172.21.15.44:0/571291068 conn(0x7f8b300fee10 msgr2=0x7f8b30101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 -- 172.21.15.44:0/571291068 shutdown_connections 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.552+0000 7f8b34a00640 1 -- 172.21.15.44:0/571291068 wait complete. 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.553+0000 7f8b34a00640 1 Processor -- start 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.554+0000 7f8b34a00640 1 -- start start 2024-07-26T08:25:50.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.554+0000 7f8b34a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 0x7f8b30075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.554+0000 7f8b34a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30108d00 0x7f8b30075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b34a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b30109650 0x7f8b30075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b34a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8b30113370 con 0x7f8b30108d00 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b34a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b301131f0 con 0x7f8b301033c0 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b34a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8b301134f0 con 0x7f8b30109650 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 0x7f8b30075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 0x7f8b30075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:37408/0 (socket says 172.21.15.44:37408) 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 -- 172.21.15.44:0/1557479760 learned_addr learned my addr 172.21.15.44:0/1557479760 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2f200640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b30109650 0x7f8b30075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b25e00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30108d00 0x7f8b30075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 -- 172.21.15.44:0/1557479760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b30109650 msgr2=0x7f8b30075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b30109650 0x7f8b30075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 -- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30108d00 msgr2=0x7f8b30075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30108d00 0x7f8b30075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b2e800640 1 -- 172.21.15.44:0/1557479760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b301a8060 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.555+0000 7f8b25e00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30108d00 0x7f8b30075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.556+0000 7f8b2e800640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 0x7f8b30075140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b20005e30 tx=0x7f8b20031840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.556+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b20003b70 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.556+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b301a82f0 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.556+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b20003d10 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.556+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2002fa50 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.556+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b301a8880 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.557+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b301037d0 con 0x7f8b301033c0 2024-07-26T08:25:50.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.560+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f8b2002fbf0 con 0x7f8b301033c0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.563+0000 7f8b27e00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8b140b3b90 0x7f8b140b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.563+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8b20103620 con 0x7f8b301033c0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.563+0000 7f8b25e00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8b140b3b90 0x7f8b140b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.564+0000 7f8b25e00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8b140b3b90 0x7f8b140b60a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b30108850 tx=0x7f8b18005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.570+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8b200cc8e0 con 0x7f8b301033c0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.827+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f8b30076440 con 0x7f8b301033c0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.838+0000 7f8b27e00640 1 -- 172.21.15.44:0/1557479760 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v23)=0 v23) ==== 108+0+0 (secure 0 0 0) 0x7f8b200cc1e0 con 0x7f8b301033c0 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8b140b3b90 msgr2=0x7f8b140b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8b140b3b90 0x7f8b140b60a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8b30108850 tx=0x7f8b18005f90 comp rx=0 tx=0).stop 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 msgr2=0x7f8b30075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 0x7f8b30075140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b20005e30 tx=0x7f8b20031840 comp rx=0 tx=0).stop 2024-07-26T08:25:50.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 shutdown_connections 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8b140b3b90 0x7f8b140b60a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b30109650 0x7f8b30075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b30108d00 0x7f8b30075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.841+0000 7f8b34a00640 1 --2- 172.21.15.44:0/1557479760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b301033c0 0x7f8b30075140 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.842+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 >> 172.21.15.44:0/1557479760 conn(0x7f8b300fee10 msgr2=0x7f8b30101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.842+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 shutdown_connections 2024-07-26T08:25:50.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:50.842+0000 7f8b34a00640 1 -- 172.21.15.44:0/1557479760 wait complete. 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:25:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:25:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:25:52.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 -- 172.21.15.44:0/3351312126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 msgr2=0x7f8fdc077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:52.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 --2- 172.21.15.44:0/3351312126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc077780 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009900 tx=0x7f8fcc02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 -- 172.21.15.44:0/3351312126 shutdown_connections 2024-07-26T08:25:52.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 --2- 172.21.15.44:0/3351312126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc1141a0 0x7f8fdc1165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 --2- 172.21.15.44:0/3351312126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 --2- 172.21.15.44:0/3351312126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc077780 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 -- 172.21.15.44:0/3351312126 >> 172.21.15.44:0/3351312126 conn(0x7f8fdc1011f0 msgr2=0x7f8fdc103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 -- 172.21.15.44:0/3351312126 shutdown_connections 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.466+0000 7f8fe2400640 1 -- 172.21.15.44:0/3351312126 wait complete. 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.467+0000 7f8fe2400640 1 Processor -- start 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 -- start start 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc1a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc1a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc1141a0 0x7f8fdc1a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc119b20 con 0x7f8fdc1141a0 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc1199a0 con 0x7f8fdc077370 2024-07-26T08:25:52.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fe2400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc119ca0 con 0x7f8fdc077cc0 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fdbe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc1141a0 0x7f8fdc1a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fdaa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc1a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fdaa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc1a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:49040/0 (socket says 172.21.15.44:49040) 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.468+0000 7f8fdaa00640 1 -- 172.21.15.44:0/1497889644 learned_addr learned my addr 172.21.15.44:0/1497889644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdb400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc1a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdaa00640 1 -- 172.21.15.44:0/1497889644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 msgr2=0x7f8fdc1a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdaa00640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc1a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdaa00640 1 -- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc1141a0 msgr2=0x7f8fdc1a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdaa00640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc1141a0 0x7f8fdc1a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdaa00640 1 -- 172.21.15.44:0/1497889644 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc1a6480 con 0x7f8fdc077cc0 2024-07-26T08:25:52.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdb400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc1a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdbe00640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc1141a0 0x7f8fdc1a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.469+0000 7f8fdaa00640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc1a0ab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e9f0 tx=0x7f8fc800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.470+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800ccc0 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.470+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc80108e0 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.470+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fdc1a6770 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.470+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8018860 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.470+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fdc1aced0 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.471+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fdc077780 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.474+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f8fc8010430 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.476+0000 7f8fd3e00640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8fc00b3c10 0x7f8fc00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.476+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8fc80e2610 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.476+0000 7f8fdb400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8fc00b3c10 0x7f8fc00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.477+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8fc80ab7c0 con 0x7f8fdc077cc0 2024-07-26T08:25:52.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.477+0000 7f8fdb400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8fc00b3c10 0x7f8fc00b6120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc031440 tx=0x7f8fcc009680 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.685+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7f8fdc1a1640 con 0x7f8fdc077cc0 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.697+0000 7f8fd3e00640 1 -- 172.21.15.44:0/1497889644 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v24)=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f8fdc1a1640 con 0x7f8fdc077cc0 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8fc00b3c10 msgr2=0x7f8fc00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8fc00b3c10 0x7f8fc00b6120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc031440 tx=0x7f8fcc009680 comp rx=0 tx=0).stop 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 msgr2=0x7f8fdc1a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc1a0ab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e9f0 tx=0x7f8fc800eeb0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 shutdown_connections 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8fc00b3c10 0x7f8fc00b6120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fdc077cc0 0x7f8fdc1a0ab0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8fdc1141a0 0x7f8fdc1a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 --2- 172.21.15.44:0/1497889644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fdc077370 0x7f8fdc1a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.701+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 >> 172.21.15.44:0/1497889644 conn(0x7f8fdc1011f0 msgr2=0x7f8fdc103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:52.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.702+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 shutdown_connections 2024-07-26T08:25:52.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:51.702+0000 7f8fe2400640 1 -- 172.21.15.44:0/1497889644 wait complete. 2024-07-26T08:25:54.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:53 smithi044 ceph-mon[32404]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:53 smithi116 ceph-mon[37097]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:53 smithi171 ceph-mon[37789]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:56.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:55 smithi044 ceph-mon[32404]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:55 smithi116 ceph-mon[37097]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:55 smithi171 ceph-mon[37789]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:58.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:57 smithi044 ceph-mon[32404]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:57 smithi116 ceph-mon[37097]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:57 smithi171 ceph-mon[37789]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:25:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:25:59 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3810190579' entity='client.admin' 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 -- 172.21.15.44:0/3982115538 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 msgr2=0x7f8d681122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3982115538 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 0x7f8d681122a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009be0 tx=0x7f8d6002a6c0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 -- 172.21.15.44:0/3982115538 shutdown_connections 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3982115538 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 0x7f8d681122a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3982115538 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d6810b120 0x7f8d6810b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3982115538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d681057e0 0x7f8d68105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.938+0000 7f8d6d400640 1 -- 172.21.15.44:0/3982115538 >> 172.21.15.44:0/3982115538 conn(0x7f8d68101210 msgr2=0x7f8d68103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.939+0000 7f8d6d400640 1 -- 172.21.15.44:0/3982115538 shutdown_connections 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.939+0000 7f8d6d400640 1 -- 172.21.15.44:0/3982115538 wait complete. 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 Processor -- start 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 -- start start 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 0x7f8d6819c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d6810b120 0x7f8d6819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 0x7f8d681a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8d68115740 con 0x7f8d681057e0 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d681155c0 con 0x7f8d6810b120 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.940+0000 7f8d6d400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8d681158c0 con 0x7f8d6810ba70 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 0x7f8d6819c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 0x7f8d6819c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32946/0 (socket says 172.21.15.44:32946) 2024-07-26T08:25:59.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 -- 172.21.15.44:0/3810190579 learned_addr learned my addr 172.21.15.44:0/3810190579 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d67200640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 0x7f8d681a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d65e00640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d6810b120 0x7f8d6819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 -- 172.21.15.44:0/3810190579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 msgr2=0x7f8d681a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 0x7f8d681a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 -- 172.21.15.44:0/3810190579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d6810b120 msgr2=0x7f8d6819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d6810b120 0x7f8d6819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 -- 172.21.15.44:0/3810190579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d681a2060 con 0x7f8d681057e0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.941+0000 7f8d66800640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 0x7f8d6819c0f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800e9f0 tx=0x7f8d5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.942+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5800cd80 con 0x7f8d681057e0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.942+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d580054d0 con 0x7f8d681057e0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.942+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d681a2350 con 0x7f8d681057e0 2024-07-26T08:25:59.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.942+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58010690 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.942+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d681a8ab0 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.943+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d68105bf0 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.946+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f8d58004e30 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.947+0000 7f8d5f400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d4c0b3b90 0x7f8d4c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.947+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8d580e2650 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.948+0000 7f8d65e00640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d4c0b3b90 0x7f8d4c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.948+0000 7f8d65e00640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d4c0b3b90 0x7f8d4c0b60a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8d681049f0 tx=0x7f8d54009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:57.950+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8d58055600 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.168+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f8d6819d220 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.178+0000 7f8d5f400640 1 -- 172.21.15.44:0/3810190579 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v25)=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f8d580ab800 con 0x7f8d681057e0 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d4c0b3b90 msgr2=0x7f8d4c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:59.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d4c0b3b90 0x7f8d4c0b60a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8d681049f0 tx=0x7f8d54009260 comp rx=0 tx=0).stop 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 msgr2=0x7f8d6819c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 0x7f8d6819c0f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800e9f0 tx=0x7f8d5800edf0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 shutdown_connections 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d4c0b3b90 0x7f8d4c0b60a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d6810ba70 0x7f8d681a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d6810b120 0x7f8d6819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 --2- 172.21.15.44:0/3810190579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d681057e0 0x7f8d6819c0f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.181+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 >> 172.21.15.44:0/3810190579 conn(0x7f8d68101210 msgr2=0x7f8d68108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.182+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 shutdown_connections 2024-07-26T08:25:59.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:25:58.182+0000 7f8d6d400640 1 -- 172.21.15.44:0/3810190579 wait complete. 2024-07-26T08:25:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:25:59 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3810190579' entity='client.admin' 2024-07-26T08:25:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:25:59 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3810190579' entity='client.admin' 2024-07-26T08:25:59.678 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:26:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:00 smithi044 ceph-mon[32404]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:00.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 -- 172.21.15.44:0/2810867073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 msgr2=0x7fa62c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 --2- 172.21.15.44:0/2810867073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c1092f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa618009860 tx=0x7fa61802a340 comp rx=0 tx=0).stop 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 -- 172.21.15.44:0/2810867073 shutdown_connections 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 --2- 172.21.15.44:0/2810867073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c1092f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 --2- 172.21.15.44:0/2810867073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 0x7fa62c110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 --2- 172.21.15.44:0/2810867073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 0x7fa62c1039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 -- 172.21.15.44:0/2810867073 >> 172.21.15.44:0/2810867073 conn(0x7fa62c0ff010 msgr2=0x7fa62c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.486+0000 7fa634400640 1 -- 172.21.15.44:0/2810867073 shutdown_connections 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.487+0000 7fa634400640 1 -- 172.21.15.44:0/2810867073 wait complete. 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 Processor -- start 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 -- start start 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 0x7fa62c19c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 0x7fa62c1a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1134a0 con 0x7fa62c1035a0 2024-07-26T08:26:00.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c113320 con 0x7fa62c109830 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.488+0000 7fa634400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c113620 con 0x7fa62c108ee0 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa631a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 0x7fa62c19c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa631a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 0x7fa62c19c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:32958/0 (socket says 172.21.15.44:32958) 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa631000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa631a00640 1 -- 172.21.15.44:0/2308027418 learned_addr learned my addr 172.21.15.44:0/2308027418 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa632400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 0x7fa62c1a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa632400640 1 -- 172.21.15.44:0/2308027418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 msgr2=0x7fa62c19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa632400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa632400640 1 -- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 msgr2=0x7fa62c19c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa632400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 0x7fa62c19c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa632400640 1 -- 172.21.15.44:0/2308027418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c1a20a0 con 0x7fa62c109830 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.489+0000 7fa631000640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:26:00.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.490+0000 7fa632400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 0x7fa62c1a1970 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa62400ec50 tx=0x7fa62400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.490+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624003b70 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.490+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c1a2390 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.490+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa624003d10 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.490+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624010830 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.490+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1a8af0 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.491+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c1039b0 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.494+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fa624003fa0 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.495+0000 7fa62a000640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6140b3c10 0x7fa6140b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.495+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa6240e2620 con 0x7fa62c109830 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.496+0000 7fa631a00640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6140b3c10 0x7fa6140b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.496+0000 7fa631a00640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6140b3c10 0x7fa6140b6120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa620005c80 tx=0x7fa620009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:00.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.499+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa6240ab7d0 con 0x7fa62c109830 2024-07-26T08:26:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:00 smithi116 ceph-mon[37097]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:00 smithi171 ceph-mon[37789]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:00.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.712+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7fa62c1a8dd0 con 0x7fa62c109830 2024-07-26T08:26:00.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.721+0000 7fa62a000640 1 -- 172.21.15.44:0/2308027418 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v26)=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7fa6240ab0d0 con 0x7fa62c109830 2024-07-26T08:26:00.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6140b3c10 msgr2=0x7fa6140b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6140b3c10 0x7fa6140b6120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa620005c80 tx=0x7fa620009260 comp rx=0 tx=0).stop 2024-07-26T08:26:00.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 msgr2=0x7fa62c1a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 0x7fa62c1a1970 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa62400ec50 tx=0x7fa62400c6a0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 shutdown_connections 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6140b3c10 0x7fa6140b6120 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa62c108ee0 0x7fa62c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c109830 0x7fa62c1a1970 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 --2- 172.21.15.44:0/2308027418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa62c1035a0 0x7fa62c19c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 >> 172.21.15.44:0/2308027418 conn(0x7fa62c0ff010 msgr2=0x7fa62c1085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 shutdown_connections 2024-07-26T08:26:00.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.724+0000 7fa634400640 1 -- 172.21.15.44:0/2308027418 wait complete. 2024-07-26T08:26:00.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.585+0000 7f8eca000640 1 -- 172.21.15.44:0/4191306776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec4104230 msgr2=0x7f8ec4104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.585+0000 7f8eca000640 1 --2- 172.21.15.44:0/4191306776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec4104230 0x7f8ec4104640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009900 tx=0x7f8eb802a400 comp rx=0 tx=0).stop 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.585+0000 7f8eca000640 1 -- 172.21.15.44:0/4191306776 shutdown_connections 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.585+0000 7f8eca000640 1 --2- 172.21.15.44:0/4191306776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec410d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.586+0000 7f8eca000640 1 --2- 172.21.15.44:0/4191306776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec410d690 0x7f8ec410fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.586+0000 7f8eca000640 1 --2- 172.21.15.44:0/4191306776 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec4104230 0x7f8ec4104640 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.586+0000 7f8eca000640 1 -- 172.21.15.44:0/4191306776 >> 172.21.15.44:0/4191306776 conn(0x7f8ec40fdd50 msgr2=0x7f8ec41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.586+0000 7f8eca000640 1 -- 172.21.15.44:0/4191306776 shutdown_connections 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.586+0000 7f8eca000640 1 -- 172.21.15.44:0/4191306776 wait complete. 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.587+0000 7f8eca000640 1 Processor -- start 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.587+0000 7f8eca000640 1 -- start start 2024-07-26T08:26:00.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8eca000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec4104230 0x7f8ec4101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8eca000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec41024b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8eca000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec410d690 0x7f8ec41029f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8eca000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec4113300 con 0x7f8ec410d690 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8eca000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec4113180 con 0x7f8ec4104230 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8eca000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec4113480 con 0x7f8ec4104b80 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8ec3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec41024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8ec3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec41024b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46230/0 (socket says 172.21.15.44:46230) 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8ec8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec4104230 0x7f8ec4101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8ec3e00640 1 -- 172.21.15.44:0/2006367486 learned_addr learned my addr 172.21.15.44:0/2006367486 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.588+0000 7f8ec9600640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec410d690 0x7f8ec41029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec3e00640 1 -- 172.21.15.44:0/2006367486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec4104230 msgr2=0x7f8ec4101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec3e00640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec4104230 0x7f8ec4101f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec3e00640 1 -- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec410d690 msgr2=0x7f8ec41029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec3e00640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec410d690 0x7f8ec41029f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec3e00640 1 -- 172.21.15.44:0/2006367486 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec41a8090 con 0x7f8ec4104b80 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec8c00640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec4104230 0x7f8ec4101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec3e00640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec41024b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000e970 tx=0x7f8eb000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000cd80 con 0x7f8ec4104b80 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eb00035f0 con 0x7f8ec4104b80 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec41a8380 con 0x7f8ec4104b80 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0010690 con 0x7f8ec4104b80 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.589+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec41a8910 con 0x7f8ec4104b80 2024-07-26T08:26:00.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.593+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec4104640 con 0x7f8ec4104b80 2024-07-26T08:26:00.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.595+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f8eb0004020 con 0x7f8ec4104b80 2024-07-26T08:26:00.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.596+0000 7f8ec1600640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8eac0b3c10 0x7f8eac0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:00.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.596+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8eb00e22c0 con 0x7f8ec4104b80 2024-07-26T08:26:00.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.597+0000 7f8ec8c00640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8eac0b3c10 0x7f8eac0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:00.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.597+0000 7f8ec8c00640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8eac0b3c10 0x7f8eac0b6120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000c00 tx=0x7f8eb8031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:00.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.601+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8eb00ab470 con 0x7f8ec4104b80 2024-07-26T08:26:00.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.824+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f8ec41a8ca0 con 0x7f8ec4104b80 2024-07-26T08:26:00.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.833+0000 7f8ec1600640 1 -- 172.21.15.44:0/2006367486 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v27)=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7f8eb00aad70 con 0x7f8ec4104b80 2024-07-26T08:26:00.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.836+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8eac0b3c10 msgr2=0x7f8eac0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.836+0000 7f8eca000640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8eac0b3c10 0x7f8eac0b6120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000c00 tx=0x7f8eb8031400 comp rx=0 tx=0).stop 2024-07-26T08:26:00.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.836+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 msgr2=0x7f8ec41024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.836+0000 7f8eca000640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec41024b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000e970 tx=0x7f8eb000ee30 comp rx=0 tx=0).stop 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.837+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 shutdown_connections 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.837+0000 7f8eca000640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8eac0b3c10 0x7f8eac0b6120 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.837+0000 7f8eca000640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ec4104b80 0x7f8ec41024b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.837+0000 7f8eca000640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ec410d690 0x7f8ec41029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.837+0000 7f8eca000640 1 --2- 172.21.15.44:0/2006367486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ec4104230 0x7f8ec4101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.837+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 >> 172.21.15.44:0/2006367486 conn(0x7f8ec40fdd50 msgr2=0x7f8ec41071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:00.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.838+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 shutdown_connections 2024-07-26T08:26:00.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:00.838+0000 7f8eca000640 1 -- 172.21.15.44:0/2006367486 wait complete. 2024-07-26T08:26:01.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:01 smithi044 ceph-mon[32404]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:01.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:01 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:01.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:01 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:01 smithi116 ceph-mon[37097]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:01 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:01 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:01 smithi171 ceph-mon[37789]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:01 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:01 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.499+0000 7f8322000640 1 -- 172.21.15.44:0/940405365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 msgr2=0x7f831c111d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.499+0000 7f8322000640 1 --2- 172.21.15.44:0/940405365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 0x7f831c111d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8314009900 tx=0x7f831402a410 comp rx=0 tx=0).stop 2024-07-26T08:26:02.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 -- 172.21.15.44:0/940405365 shutdown_connections 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 --2- 172.21.15.44:0/940405365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c075b30 0x7f831c10f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 --2- 172.21.15.44:0/940405365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 0x7f831c111d60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 --2- 172.21.15.44:0/940405365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f831c0751e0 0x7f831c0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 -- 172.21.15.44:0/940405365 >> 172.21.15.44:0/940405365 conn(0x7f831c0fee10 msgr2=0x7f831c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 -- 172.21.15.44:0/940405365 shutdown_connections 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.500+0000 7f8322000640 1 -- 172.21.15.44:0/940405365 wait complete. 2024-07-26T08:26:02.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.501+0000 7f8322000640 1 Processor -- start 2024-07-26T08:26:02.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 -- start start 2024-07-26T08:26:02.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 0x7f831c19be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:02.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f831c075b30 0x7f831c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 0x7f831c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f831c115500 con 0x7f831c10f920 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f831c115380 con 0x7f831c075b30 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.502+0000 7f8322000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f831c115680 con 0x7f831c0751e0 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.503+0000 7f8320c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 0x7f831c19be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.503+0000 7f8320c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 0x7f831c19be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:43330/0 (socket says 172.21.15.44:43330) 2024-07-26T08:26:02.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.503+0000 7f8320c00640 1 -- 172.21.15.44:0/603723998 learned_addr learned my addr 172.21.15.44:0/603723998 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:02.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.503+0000 7f831be00640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f831c075b30 0x7f831c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:02.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8321600640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 0x7f831c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:02.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8320c00640 1 -- 172.21.15.44:0/603723998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f831c075b30 msgr2=0x7f831c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:02.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8320c00640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f831c075b30 0x7f831c19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8320c00640 1 -- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 msgr2=0x7f831c1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:02.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8320c00640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 0x7f831c1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8320c00640 1 -- 172.21.15.44:0/603723998 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831c1a1de0 con 0x7f831c0751e0 2024-07-26T08:26:02.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.504+0000 7f8320c00640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 0x7f831c19be70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f830c00e9f0 tx=0x7f830c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:02.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.505+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c00cd80 con 0x7f831c0751e0 2024-07-26T08:26:02.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.505+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830c0108e0 con 0x7f831c0751e0 2024-07-26T08:26:02.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.505+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c018980 con 0x7f831c0751e0 2024-07-26T08:26:02.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.505+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831c1a20d0 con 0x7f831c0751e0 2024-07-26T08:26:02.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.505+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f831c1a8830 con 0x7f831c0751e0 2024-07-26T08:26:02.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.506+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f831c0755f0 con 0x7f831c0751e0 2024-07-26T08:26:02.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.508+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f830c010430 con 0x7f831c0751e0 2024-07-26T08:26:02.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.510+0000 7f8319600640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83000b3c10 0x7f83000b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:02.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.510+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f830c0e2660 con 0x7f831c0751e0 2024-07-26T08:26:02.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.510+0000 7f831be00640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83000b3c10 0x7f83000b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:02.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.511+0000 7f831be00640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83000b3c10 0x7f83000b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f831c111b70 tx=0x7f8304009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:02.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.512+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f830c0ab810 con 0x7f831c0751e0 2024-07-26T08:26:02.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.718+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f831c19cfa0 con 0x7f831c0751e0 2024-07-26T08:26:02.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.728+0000 7f8319600640 1 -- 172.21.15.44:0/603723998 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v28)=0 v28) ==== 129+0+0 (secure 0 0 0) 0x7f830c0ab110 con 0x7f831c0751e0 2024-07-26T08:26:02.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83000b3c10 msgr2=0x7f83000b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83000b3c10 0x7f83000b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f831c111b70 tx=0x7f8304009500 comp rx=0 tx=0).stop 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 msgr2=0x7f831c19be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 0x7f831c19be70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f830c00e9f0 tx=0x7f830c00eeb0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 shutdown_connections 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83000b3c10 0x7f83000b6120 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f831c10f920 0x7f831c1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f831c075b30 0x7f831c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 --2- 172.21.15.44:0/603723998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f831c0751e0 0x7f831c19be70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:02.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.732+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 >> 172.21.15.44:0/603723998 conn(0x7f831c0fee10 msgr2=0x7f831c109430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:02.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.733+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 shutdown_connections 2024-07-26T08:26:02.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:02.733+0000 7f8322000640 1 -- 172.21.15.44:0/603723998 wait complete. 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 -- 172.21.15.44:0/3670922589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134108d20 msgr2=0x7fa134109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 --2- 172.21.15.44:0/3670922589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134108d20 0x7fa134109130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa120009860 tx=0x7fa12002a340 comp rx=0 tx=0).stop 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 -- 172.21.15.44:0/3670922589 shutdown_connections 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 --2- 172.21.15.44:0/3670922589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134108d20 0x7fa134109130 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 --2- 172.21.15.44:0/3670922589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa134109670 0x7fa13410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 --2- 172.21.15.44:0/3670922589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1341033e0 0x7fa1341037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 -- 172.21.15.44:0/3670922589 >> 172.21.15.44:0/3670922589 conn(0x7fa1340fee30 msgr2=0x7fa134101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 -- 172.21.15.44:0/3670922589 shutdown_connections 2024-07-26T08:26:03.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.750+0000 7fa13aa00640 1 -- 172.21.15.44:0/3670922589 wait complete. 2024-07-26T08:26:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.752+0000 7fa13aa00640 1 Processor -- start 2024-07-26T08:26:03.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.752+0000 7fa13aa00640 1 -- start start 2024-07-26T08:26:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.752+0000 7fa13aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 0x7fa13419c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.752+0000 7fa13aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa134108d20 0x7fa13419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa13aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134109670 0x7fa1341a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:03.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa13aa00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa134113300 con 0x7fa1341033e0 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa13aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa134113180 con 0x7fa134108d20 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa13aa00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa134113480 con 0x7fa134109670 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 0x7fa13419c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 0x7fa13419c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43246/0 (socket says 172.21.15.44:43246) 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 -- 172.21.15.44:0/2893851407 learned_addr learned my addr 172.21.15.44:0/2893851407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa138c00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa134108d20 0x7fa13419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa13a000640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134109670 0x7fa1341a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:03.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 -- 172.21.15.44:0/2893851407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134109670 msgr2=0x7fa1341a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134109670 0x7fa1341a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 -- 172.21.15.44:0/2893851407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa134108d20 msgr2=0x7fa13419c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa134108d20 0x7fa13419c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa139600640 1 -- 172.21.15.44:0/2893851407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1341a1f80 con 0x7fa1341033e0 2024-07-26T08:26:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.753+0000 7fa138c00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa134108d20 0x7fa13419c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:26:03.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.754+0000 7fa139600640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 0x7fa13419c010 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa12400eb90 tx=0x7fa12400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.754+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124004d20 con 0x7fa1341033e0 2024-07-26T08:26:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.754+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12400ee90 con 0x7fa1341033e0 2024-07-26T08:26:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.754+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1341a2270 con 0x7fa1341033e0 2024-07-26T08:26:03.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.754+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124010670 con 0x7fa1341033e0 2024-07-26T08:26:03.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.754+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1341a89d0 con 0x7fa1341033e0 2024-07-26T08:26:03.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.756+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1341037f0 con 0x7fa1341033e0 2024-07-26T08:26:03.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.758+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fa124005020 con 0x7fa1341033e0 2024-07-26T08:26:03.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.760+0000 7fa132000640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa1180b3c10 0x7fa1180b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:03.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.760+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa1240e26e0 con 0x7fa1341033e0 2024-07-26T08:26:03.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.760+0000 7fa138c00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa1180b3c10 0x7fa1180b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:03.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.761+0000 7fa138c00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa1180b3c10 0x7fa1180b6120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa120035040 tx=0x7fa120030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:03.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.764+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa1240ab890 con 0x7fa1341033e0 2024-07-26T08:26:03.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.970+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fa1341a8d40 con 0x7fa1341033e0 2024-07-26T08:26:03.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:03 smithi044 ceph-mon[32404]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:03.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:03 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:03.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:03.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:03.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:03.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:03.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.981+0000 7fa132000640 1 -- 172.21.15.44:0/2893851407 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v29)=0 v29) ==== 120+0+0 (secure 0 0 0) 0x7fa1240ab190 con 0x7fa1341033e0 2024-07-26T08:26:03.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.983+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa1180b3c10 msgr2=0x7fa1180b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:03.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.983+0000 7fa13aa00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa1180b3c10 0x7fa1180b6120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa120035040 tx=0x7fa120030340 comp rx=0 tx=0).stop 2024-07-26T08:26:03.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 msgr2=0x7fa13419c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:03.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 0x7fa13419c010 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa12400eb90 tx=0x7fa12400c6a0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 shutdown_connections 2024-07-26T08:26:03.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa1180b3c10 0x7fa1180b6120 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa134109670 0x7fa1341a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa134108d20 0x7fa13419c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 --2- 172.21.15.44:0/2893851407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa1341033e0 0x7fa13419c010 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:03.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 >> 172.21.15.44:0/2893851407 conn(0x7fa1340fee30 msgr2=0x7fa1341063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:03.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 shutdown_connections 2024-07-26T08:26:03.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:03.984+0000 7fa13aa00640 1 -- 172.21.15.44:0/2893851407 wait complete. 2024-07-26T08:26:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:03 smithi116 ceph-mon[37097]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:03 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:03 smithi171 ceph-mon[37789]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:03 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:05.216 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:04 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2893851407' entity='client.admin' 2024-07-26T08:26:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:04 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2893851407' entity='client.admin' 2024-07-26T08:26:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:04 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2893851407' entity='client.admin' 2024-07-26T08:26:06.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:05 smithi044 ceph-mon[32404]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:05 smithi116 ceph-mon[37097]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:05 smithi171 ceph-mon[37789]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 -- 172.21.15.44:0/2338736216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 msgr2=0x7f7488109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 --2- 172.21.15.44:0/2338736216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 0x7f7488109310 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7470009be0 tx=0x7f747002a6c0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 -- 172.21.15.44:0/2338736216 shutdown_connections 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 --2- 172.21.15.44:0/2338736216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7488109850 0x7f7488110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 --2- 172.21.15.44:0/2338736216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 0x7f7488109310 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 --2- 172.21.15.44:0/2338736216 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74881035c0 0x7f74881039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 -- 172.21.15.44:0/2338736216 >> 172.21.15.44:0/2338736216 conn(0x7f74880ff010 msgr2=0x7f7488101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 -- 172.21.15.44:0/2338736216 shutdown_connections 2024-07-26T08:26:07.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.045+0000 7f748e000640 1 -- 172.21.15.44:0/2338736216 wait complete. 2024-07-26T08:26:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.046+0000 7f748e000640 1 Processor -- start 2024-07-26T08:26:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 -- start start 2024-07-26T08:26:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 0x7f748819c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 0x7f748819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7488109850 0x7f74881a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f74881134c0 con 0x7f74881035c0 2024-07-26T08:26:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7488113340 con 0x7f7488108f00 2024-07-26T08:26:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f748e000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7488113640 con 0x7f7488109850 2024-07-26T08:26:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f7487400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 0x7f748819c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.047+0000 7f7487400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 0x7f748819c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43264/0 (socket says 172.21.15.44:43264) 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 -- 172.21.15.44:0/3095379309 learned_addr learned my addr 172.21.15.44:0/3095379309 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7486a00640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 0x7f748819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487e00640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7488109850 0x7f74881a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 -- 172.21.15.44:0/3095379309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7488109850 msgr2=0x7f74881a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7488109850 0x7f74881a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 -- 172.21.15.44:0/3095379309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 msgr2=0x7f748819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 0x7f748819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 -- 172.21.15.44:0/3095379309 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74881a2060 con 0x7f74881035c0 2024-07-26T08:26:07.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.048+0000 7f7487400640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 0x7f748819c0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f747800e9c0 tx=0x7f747800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.049+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747800cde0 con 0x7f74881035c0 2024-07-26T08:26:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.049+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74780035f0 con 0x7f74881035c0 2024-07-26T08:26:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.049+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478010690 con 0x7f74881035c0 2024-07-26T08:26:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.049+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74881a2350 con 0x7f74881035c0 2024-07-26T08:26:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.049+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74881a8ab0 con 0x7f74881035c0 2024-07-26T08:26:07.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.050+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74881039d0 con 0x7f74881035c0 2024-07-26T08:26:07.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.055+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f7478004000 con 0x7f74881035c0 2024-07-26T08:26:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.057+0000 7f747fe00640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f74680b3b90 0x7f74680b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.057+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f74780e2480 con 0x7f74881035c0 2024-07-26T08:26:07.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.057+0000 7f7486a00640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f74680b3b90 0x7f74680b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.058+0000 7f7486a00640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f74680b3b90 0x7f74680b60a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f747002abe0 tx=0x7f7470005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:07.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.059+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f74780ab6c0 con 0x7f74881035c0 2024-07-26T08:26:07.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.270+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f748819d220 con 0x7f74881035c0 2024-07-26T08:26:07.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.281+0000 7f747fe00640 1 -- 172.21.15.44:0/3095379309 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f74780aafc0 con 0x7f74881035c0 2024-07-26T08:26:07.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.285+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f74680b3b90 msgr2=0x7f74680b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.285+0000 7f748e000640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f74680b3b90 0x7f74680b60a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f747002abe0 tx=0x7f7470005fd0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.285+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 msgr2=0x7f748819c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.285+0000 7f748e000640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 0x7f748819c0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f747800e9c0 tx=0x7f747800ee80 comp rx=0 tx=0).stop 2024-07-26T08:26:07.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 shutdown_connections 2024-07-26T08:26:07.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f74680b3b90 0x7f74680b60a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7488109850 0x7f74881a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7488108f00 0x7f748819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 --2- 172.21.15.44:0/3095379309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f74881035c0 0x7f748819c0f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 >> 172.21.15.44:0/3095379309 conn(0x7f74880ff010 msgr2=0x7f7488107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:07.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 shutdown_connections 2024-07-26T08:26:07.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.286+0000 7f748e000640 1 -- 172.21.15.44:0/3095379309 wait complete. 2024-07-26T08:26:07.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 -- 172.21.15.44:0/3590318177 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 msgr2=0x7fabe0111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 --2- 172.21.15.44:0/3590318177 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe0111d80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009860 tx=0x7fabd802a340 comp rx=0 tx=0).stop 2024-07-26T08:26:07.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 -- 172.21.15.44:0/3590318177 shutdown_connections 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 --2- 172.21.15.44:0/3590318177 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe0111d80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 --2- 172.21.15.44:0/3590318177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe0075b30 0x7fabe010f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 --2- 172.21.15.44:0/3590318177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe00751e0 0x7fabe00755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 -- 172.21.15.44:0/3590318177 >> 172.21.15.44:0/3590318177 conn(0x7fabe00fee70 msgr2=0x7fabe01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 -- 172.21.15.44:0/3590318177 shutdown_connections 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.132+0000 7fabe4a00640 1 -- 172.21.15.44:0/3590318177 wait complete. 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.133+0000 7fabe4a00640 1 Processor -- start 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.133+0000 7fabe4a00640 1 -- start start 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.133+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe00751e0 0x7fabe019bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 0x7fabe019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe01a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabe4a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01154d0 con 0x7fabe00751e0 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabe4a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0115350 con 0x7fabe0075b30 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabe4a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0115650 con 0x7fabe010f940 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 0x7fabe019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabde800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe00751e0 0x7fabe019bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 0x7fabe019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:41854/0 (socket says 172.21.15.44:41854) 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 -- 172.21.15.44:0/424726629 learned_addr learned my addr 172.21.15.44:0/424726629 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdf200640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe01a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 -- 172.21.15.44:0/424726629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 msgr2=0x7fabe01a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe01a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 -- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe00751e0 msgr2=0x7fabe019bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe00751e0 0x7fabe019bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdde00640 1 -- 172.21.15.44:0/424726629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe01a1e10 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.134+0000 7fabdf200640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe01a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.135+0000 7fabdde00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 0x7fabe019c440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9a0 tx=0x7fabcc00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.135+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00cd80 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.135+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabcc0108e0 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.135+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe01a2100 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.135+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc018960 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.135+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe01a8860 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.137+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe00755f0 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.140+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fabcc010a90 con 0x7fabe0075b30 2024-07-26T08:26:07.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.143+0000 7fabd7400640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fabc40b3b90 0x7fabc40b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.143+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fabcc0e2490 con 0x7fabe0075b30 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.143+0000 7fabde800640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fabc40b3b90 0x7fabc40b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.144+0000 7fabde800640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fabc40b3b90 0x7fabc40b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabd00099a0 tx=0x7fabd0009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.146+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fabcc055600 con 0x7fabe0075b30 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.369+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7fabe019cfd0 con 0x7fabe0075b30 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.375+0000 7fabd7400640 1 -- 172.21.15.44:0/424726629 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v32)=0 v32) ==== 121+0+0 (secure 0 0 0) 0x7fabcc0ab640 con 0x7fabe0075b30 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fabc40b3b90 msgr2=0x7fabc40b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fabc40b3b90 0x7fabc40b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabd00099a0 tx=0x7fabd0009500 comp rx=0 tx=0).stop 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 msgr2=0x7fabe019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 0x7fabe019c440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00e9a0 tx=0x7fabcc00ee60 comp rx=0 tx=0).stop 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 shutdown_connections 2024-07-26T08:26:07.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fabc40b3b90 0x7fabc40b60a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabe010f940 0x7fabe01a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabe0075b30 0x7fabe019c440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 --2- 172.21.15.44:0/424726629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe00751e0 0x7fabe019bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 >> 172.21.15.44:0/424726629 conn(0x7fabe00fee70 msgr2=0x7fabe01094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 shutdown_connections 2024-07-26T08:26:07.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:07.378+0000 7fabe4a00640 1 -- 172.21.15.44:0/424726629 wait complete. 2024-07-26T08:26:07.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:07.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:07.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3095379309' entity='client.admin' 2024-07-26T08:26:07.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:07.701 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:07.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:07.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:07.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:07 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3095379309' entity='client.admin' 2024-07-26T08:26:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:07 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3095379309' entity='client.admin' 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:07 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:08.219 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi044') 2024-07-26T08:26:08.220 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi044') 2024-07-26T08:26:08.220 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi044') 2024-07-26T08:26:08.220 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi044') 2024-07-26T08:26:08.220 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi044') 2024-07-26T08:26:08.220 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi044') 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.253+0000 7f4a12200640 1 -- 172.21.15.44:0/3653537696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 msgr2=0x7f4a0c077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.253+0000 7f4a12200640 1 --2- 172.21.15.44:0/3653537696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c077780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009860 tx=0x7f4a0002a340 comp rx=0 tx=0).stop 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 -- 172.21.15.44:0/3653537696 shutdown_connections 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 --2- 172.21.15.44:0/3653537696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c113fa0 0x7f4a0c1163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 --2- 172.21.15.44:0/3653537696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0c077cc0 0x7f4a0c113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 --2- 172.21.15.44:0/3653537696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c077780 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 -- 172.21.15.44:0/3653537696 >> 172.21.15.44:0/3653537696 conn(0x7f4a0c1010b0 msgr2=0x7f4a0c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 -- 172.21.15.44:0/3653537696 shutdown_connections 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.254+0000 7f4a12200640 1 -- 172.21.15.44:0/3653537696 wait complete. 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 Processor -- start 2024-07-26T08:26:08.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 -- start start 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c1a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c077cc0 0x7f4a0c1a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0c113fa0 0x7f4a0c1a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c119900 con 0x7f4a0c077cc0 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c119780 con 0x7f4a0c113fa0 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.256+0000 7f4a12200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c119a80 con 0x7f4a0c077370 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c1a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c1a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:43378/0 (socket says 172.21.15.44:43378) 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 -- 172.21.15.44:0/1344584226 learned_addr learned my addr 172.21.15.44:0/1344584226 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a0be00640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c077cc0 0x7f4a0c1a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a11800640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0c113fa0 0x7f4a0c1a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:08.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 -- 172.21.15.44:0/1344584226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0c113fa0 msgr2=0x7f4a0c1a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0c113fa0 0x7f4a0c1a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 -- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c077cc0 msgr2=0x7f4a0c1a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c077cc0 0x7f4a0c1a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 -- 172.21.15.44:0/1344584226 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c1a6340 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.257+0000 7f4a10e00640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c1a03d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a000303f0 tx=0x7f4a00032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.258+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00040020 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.258+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c1a65d0 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.258+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a00030dd0 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.258+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00037040 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.258+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c1acd90 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.259+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0c077780 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.262+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f4a00043020 con 0x7f4a0c077370 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.263+0000 7f4a09600640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f49f40b3b90 0x7f49f40b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:08.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.263+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4a00101ff0 con 0x7f4a0c077370 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.263+0000 7f4a0be00640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f49f40b3b90 0x7f49f40b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.264+0000 7f4a0be00640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f49f40b3b90 0x7f49f40b60a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c079af0 tx=0x7f49f8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.265+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4a0004a020 con 0x7f4a0c077370 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.477+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f4a0c1a1500 con 0x7f4a0c077370 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.488+0000 7f4a09600640 1 -- 172.21.15.44:0/1344584226 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v30)=0 v30) ==== 119+0+0 (secure 0 0 0) 0x7f4a000cb220 con 0x7f4a0c077370 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f49f40b3b90 msgr2=0x7f49f40b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f49f40b3b90 0x7f49f40b60a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c079af0 tx=0x7f49f8009260 comp rx=0 tx=0).stop 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 msgr2=0x7f4a0c1a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c1a03d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a000303f0 tx=0x7f4a00032da0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 shutdown_connections 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f49f40b3b90 0x7f49f40b60a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a0c113fa0 0x7f4a0c1a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a0c077cc0 0x7f4a0c1a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 --2- 172.21.15.44:0/1344584226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c077370 0x7f4a0c1a03d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 >> 172.21.15.44:0/1344584226 conn(0x7f4a0c1010b0 msgr2=0x7f4a0c116070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 shutdown_connections 2024-07-26T08:26:08.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:06.492+0000 7f4a12200640 1 -- 172.21.15.44:0/1344584226 wait complete. 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 -- 172.21.15.44:0/321712019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 msgr2=0x7f1170105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 --2- 172.21.15.44:0/321712019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f1170105bf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1160009900 tx=0x7f116002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 -- 172.21.15.44:0/321712019 shutdown_connections 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 --2- 172.21.15.44:0/321712019 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 0x7f11701122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 --2- 172.21.15.44:0/321712019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f117010b120 0x7f117010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 --2- 172.21.15.44:0/321712019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f1170105bf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 -- 172.21.15.44:0/321712019 >> 172.21.15.44:0/321712019 conn(0x7f11701011f0 msgr2=0x7f1170103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:09.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 -- 172.21.15.44:0/321712019 shutdown_connections 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.797+0000 7f1177200640 1 -- 172.21.15.44:0/321712019 wait complete. 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.798+0000 7f1177200640 1 Processor -- start 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 -- start start 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f117019c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f117010b120 0x7f117019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 0x7f11701a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1170115740 con 0x7f11701057e0 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11701155c0 con 0x7f117010b120 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1177200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f11701158c0 con 0x7f117010ba70 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f116fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f117019c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f116fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f117019c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43306/0 (socket says 172.21.15.44:43306) 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f116fe00640 1 -- 172.21.15.44:0/4182526056 learned_addr learned my addr 172.21.15.44:0/4182526056 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.799+0000 7f1175200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 0x7f11701a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116f400640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f117010b120 0x7f117019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116fe00640 1 -- 172.21.15.44:0/4182526056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 msgr2=0x7f11701a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116fe00640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 0x7f11701a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116fe00640 1 -- 172.21.15.44:0/4182526056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f117010b120 msgr2=0x7f117019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116fe00640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f117010b120 0x7f117019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116fe00640 1 -- 172.21.15.44:0/4182526056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11701a20c0 con 0x7f11701057e0 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f1175200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 0x7f11701a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.800+0000 7f116fe00640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f117019c150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1160005940 tx=0x7f11600040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.801+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116003f030 con 0x7f11701057e0 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.801+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11600044c0 con 0x7f11701057e0 2024-07-26T08:26:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.801+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160035070 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.801+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11701a2350 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.801+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11701a8b10 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.802+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1170106a60 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.804+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f1160042030 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.806+0000 7f116cc00640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f11540b3c10 0x7f11540b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.806+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1160102670 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.806+0000 7f116f400640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f11540b3c10 0x7f11540b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.807+0000 7f116f400640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f11540b3c10 0x7f11540b6120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f11580095f0 tx=0x7f1158005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:08.808+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f11600cb1d0 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.012+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f11701a8ce0 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.018+0000 7f116cc00640 1 -- 172.21.15.44:0/4182526056 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v33)=0 v33) ==== 121+0+0 (secure 0 0 0) 0x7f11600cf720 con 0x7f11701057e0 2024-07-26T08:26:09.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f11540b3c10 msgr2=0x7f11540b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f11540b3c10 0x7f11540b6120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f11580095f0 tx=0x7f1158005f90 comp rx=0 tx=0).stop 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 msgr2=0x7f117019c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f117019c150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1160005940 tx=0x7f11600040c0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 shutdown_connections 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f11540b3c10 0x7f11540b6120 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f117010ba70 0x7f11701a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f117010b120 0x7f117019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 --2- 172.21.15.44:0/4182526056 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11701057e0 0x7f117019c150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 >> 172.21.15.44:0/4182526056 conn(0x7f11701011f0 msgr2=0x7f11701098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 shutdown_connections 2024-07-26T08:26:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.019+0000 7f1177200640 1 -- 172.21.15.44:0/4182526056 wait complete. 2024-07-26T08:26:10.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:10 smithi044 ceph-mon[32404]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:10.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:10 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4182526056' entity='client.admin' 2024-07-26T08:26:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:10 smithi116 ceph-mon[37097]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:10 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4182526056' entity='client.admin' 2024-07-26T08:26:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:10 smithi171 ceph-mon[37789]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:10 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4182526056' entity='client.admin' 2024-07-26T08:26:10.842 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi044') 2024-07-26T08:26:10.842 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi044') 2024-07-26T08:26:10.842 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi044') 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 -- 172.21.15.44:0/1895964885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0108ed0 msgr2=0x7fcfe01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/1895964885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0108ed0 0x7fcfe01092e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009900 tx=0x7fcfc802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 -- 172.21.15.44:0/1895964885 shutdown_connections 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/1895964885 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfe0109820 0x7fcfe0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/1895964885 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0108ed0 0x7fcfe01092e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/1895964885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.802+0000 7fcfe7c00640 1 -- 172.21.15.44:0/1895964885 >> 172.21.15.44:0/1895964885 conn(0x7fcfe007b280 msgr2=0x7fcfe007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.803+0000 7fcfe7c00640 1 -- 172.21.15.44:0/1895964885 shutdown_connections 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.803+0000 7fcfe7c00640 1 -- 172.21.15.44:0/1895964885 wait complete. 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 Processor -- start 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 -- start start 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe0075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfe0108ed0 0x7fcfe0075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0109820 0x7fcfe0075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0113500 con 0x7fcfe0109820 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0113380 con 0x7fcfe0103590 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.804+0000 7fcfe7c00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0113680 con 0x7fcfe0108ed0 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe0075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0109820 0x7fcfe0075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:11.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe0075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:41876/0 (socket says 172.21.15.44:41876) 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 -- 172.21.15.44:0/3246258408 learned_addr learned my addr 172.21.15.44:0/3246258408 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfd7e00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfe0108ed0 0x7fcfe0075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 -- 172.21.15.44:0/3246258408 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfe0108ed0 msgr2=0x7fcfe0075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfe0108ed0 0x7fcfe0075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 -- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0109820 msgr2=0x7fcfe0075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0109820 0x7fcfe0075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 -- 172.21.15.44:0/3246258408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe01a81b0 con 0x7fcfe0103590 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.805+0000 7fcfe5200640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe0075140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000e9f0 tx=0x7fcfd000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.806+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000cd80 con 0x7fcfe0103590 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.806+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd00108e0 con 0x7fcfe0103590 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.806+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0018980 con 0x7fcfe0103590 2024-07-26T08:26:11.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.806+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe01a84a0 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.806+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe01a8a00 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.807+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe01039a0 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.810+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fcfd0010430 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.811+0000 7fcfde000640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcfc40b3c90 0x7fcfc40b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.811+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fcfd00e2810 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.812+0000 7fcfd7e00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcfc40b3c90 0x7fcfc40b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.812+0000 7fcfd7e00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcfc40b3c90 0x7fcfc40b61a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcfc802a9e0 tx=0x7fcfc8005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:09.813+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcfd00ab370 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.041+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fcfe002e0c0 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.051+0000 7fcfde000640 1 -- 172.21.15.44:0/3246258408 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v34)=0 v34) ==== 125+0+0 (secure 0 0 0) 0x7fcfd00af8c0 con 0x7fcfe0103590 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.054+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcfc40b3c90 msgr2=0x7fcfc40b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:11.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.054+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcfc40b3c90 0x7fcfc40b61a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcfc802a9e0 tx=0x7fcfc8005a20 comp rx=0 tx=0).stop 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.054+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 msgr2=0x7fcfe0075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.054+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe0075140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000e9f0 tx=0x7fcfd000eeb0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 shutdown_connections 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcfc40b3c90 0x7fcfc40b61a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfe0108ed0 0x7fcfe0075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfe0109820 0x7fcfe0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 --2- 172.21.15.44:0/3246258408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0103590 0x7fcfe0075140 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 >> 172.21.15.44:0/3246258408 conn(0x7fcfe007b280 msgr2=0x7fcfe0106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.055+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 shutdown_connections 2024-07-26T08:26:11.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:10.056+0000 7fcfe7c00640 1 -- 172.21.15.44:0/3246258408 wait complete. 2024-07-26T08:26:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:11 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:11.384 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:11 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:11 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:12.083 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi044') 2024-07-26T08:26:12.084 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi044') 2024-07-26T08:26:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:12 smithi044 ceph-mon[32404]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:12 smithi171 ceph-mon[37789]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:12 smithi116 ceph-mon[37097]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:13.118 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi044') 2024-07-26T08:26:14.088 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:14 smithi044 ceph-mon[32404]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:14 smithi171 ceph-mon[37789]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 -- 172.21.15.44:0/643217049 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23881057b0 msgr2=0x7f2388105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 --2- 172.21.15.44:0/643217049 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23881057b0 0x7f2388105bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2378009860 tx=0x7f237802a340 comp rx=0 tx=0).stop 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 -- 172.21.15.44:0/643217049 shutdown_connections 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 --2- 172.21.15.44:0/643217049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810ba40 0x7f2388112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 --2- 172.21.15.44:0/643217049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f238810b0f0 0x7f238810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 --2- 172.21.15.44:0/643217049 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23881057b0 0x7f2388105bc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 -- 172.21.15.44:0/643217049 >> 172.21.15.44:0/643217049 conn(0x7f2388101240 msgr2=0x7f2388103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:14.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.385+0000 7f238dc00640 1 -- 172.21.15.44:0/643217049 shutdown_connections 2024-07-26T08:26:14.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.386+0000 7f238dc00640 1 -- 172.21.15.44:0/643217049 wait complete. 2024-07-26T08:26:14.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.387+0000 7f238dc00640 1 Processor -- start 2024-07-26T08:26:14.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 -- start start 2024-07-26T08:26:14.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 0x7f238819c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:14.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810b0f0 0x7f238819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:14.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f238810ba40 0x7f23881a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:14.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f23881156f0 con 0x7f238810b0f0 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2388115570 con 0x7f23881057b0 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.388+0000 7f238dc00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2388115870 con 0x7f238810ba40 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f237fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810b0f0 0x7f238819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 0x7f238819c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 0x7f238819c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:55830/0 (socket says 172.21.15.44:55830) 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f237fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810b0f0 0x7f238819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49058/0 (socket says 172.21.15.44:49058) 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 -- 172.21.15.44:0/119259351 learned_addr learned my addr 172.21.15.44:0/119259351 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:14.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2387400640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f238810ba40 0x7f23881a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:14.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 -- 172.21.15.44:0/119259351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f238810ba40 msgr2=0x7f23881a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:14.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f238810ba40 0x7f23881a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 -- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810b0f0 msgr2=0x7f238819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810b0f0 0x7f238819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.389+0000 7f2386a00640 1 -- 172.21.15.44:0/119259351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23881a2070 con 0x7f23881057b0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.390+0000 7f2386a00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 0x7f238819c100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2378035040 tx=0x7f2378032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.390+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378037040 con 0x7f23881057b0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.390+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2378041070 con 0x7f23881057b0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.390+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23881a2300 con 0x7f23881057b0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.390+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378003850 con 0x7f23881057b0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.390+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23881a2780 con 0x7f23881057b0 2024-07-26T08:26:14.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.392+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2388105bc0 con 0x7f23881057b0 2024-07-26T08:26:14.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.394+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f23780039f0 con 0x7f23881057b0 2024-07-26T08:26:14.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.395+0000 7f237f400640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f236c0b3b90 0x7f236c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:14.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.395+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2378101bc0 con 0x7f23881057b0 2024-07-26T08:26:14.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.395+0000 7f237fe00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f236c0b3b90 0x7f236c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:14.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.396+0000 7f237fe00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f236c0b3b90 0x7f236c0b60a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f238810ad30 tx=0x7f2370005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.399+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f23780cad70 con 0x7f23881057b0 2024-07-26T08:26:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:14 smithi116 ceph-mon[37097]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:14.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.602+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f238819d230 con 0x7f23881057b0 2024-07-26T08:26:14.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.612+0000 7f237f400640 1 -- 172.21.15.44:0/119259351 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v35)=0 v35) ==== 120+0+0 (secure 0 0 0) 0x7f23780ca670 con 0x7f23881057b0 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f236c0b3b90 msgr2=0x7f236c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f236c0b3b90 0x7f236c0b60a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f238810ad30 tx=0x7f2370005f90 comp rx=0 tx=0).stop 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 msgr2=0x7f238819c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 0x7f238819c100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2378035040 tx=0x7f2378032d50 comp rx=0 tx=0).stop 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 shutdown_connections 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f236c0b3b90 0x7f236c0b60a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f238810ba40 0x7f23881a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f238810b0f0 0x7f238819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 --2- 172.21.15.44:0/119259351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23881057b0 0x7f238819c100 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:14.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.616+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 >> 172.21.15.44:0/119259351 conn(0x7f2388101240 msgr2=0x7f2388103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:14.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.617+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 shutdown_connections 2024-07-26T08:26:14.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:14.617+0000 7f238dc00640 1 -- 172.21.15.44:0/119259351 wait complete. 2024-07-26T08:26:15.469 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.471 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '10471e4e-4b28-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi044') 2024-07-26T08:26:15.471 INFO:tasks.cephadm:Deploying OSDs... 2024-07-26T08:26:15.471 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:26:15.471 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T08:26:15.532 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T08:26:15.532 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-07-26 08:24:05.196175925 +0000 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-07-26 08:17:47.825361377 +0000 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-07-26 08:17:47.825361377 +0000 2024-07-26T08:26:15.590 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-07-26 08:17:47.825361377 +0000 2024-07-26T08:26:15.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T08:26:15.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:15 smithi044 ceph-mon[32404]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:15.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:15 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' 2024-07-26T08:26:15.679 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-07-26T08:26:15.679 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-07-26T08:26:15.679 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000519088 s, 986 kB/s 2024-07-26T08:26:15.683 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T08:26:15.744 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-07-26 08:24:05.319174009 +0000 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-07-26 08:17:48.455351632 +0000 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-07-26 08:17:48.455351632 +0000 2024-07-26T08:26:15.796 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-07-26 08:17:48.455351632 +0000 2024-07-26T08:26:15.797 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T08:26:15.878 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-07-26T08:26:15.879 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-07-26T08:26:15.879 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000400689 s, 1.3 MB/s 2024-07-26T08:26:15.883 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T08:26:15.944 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-07-26 08:24:05.433172234 +0000 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-07-26 08:17:49.060342274 +0000 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-07-26 08:17:49.060342274 +0000 2024-07-26T08:26:15.999 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-07-26 08:17:49.060342274 +0000 2024-07-26T08:26:16.000 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T08:26:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:15 smithi116 ceph-mon[37097]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:15 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' 2024-07-26T08:26:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-07-26T08:26:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-07-26T08:26:16.085 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000382727 s, 1.3 MB/s 2024-07-26T08:26:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:15 smithi171 ceph-mon[37789]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:15 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' 2024-07-26T08:26:16.089 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T08:26:16.146 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-07-26T08:26:16.200 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-07-26 08:24:05.522170848 +0000 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-07-26 08:17:49.683332638 +0000 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-07-26 08:17:49.683332638 +0000 2024-07-26T08:26:16.201 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-07-26 08:17:49.683332638 +0000 2024-07-26T08:26:16.201 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T08:26:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-07-26T08:26:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-07-26T08:26:16.281 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000339843 s, 1.5 MB/s 2024-07-26T08:26:16.285 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T08:26:16.342 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-07-26T08:26:16.342 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T08:26:16.361 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T08:26:16.361 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-07-26T08:26:16.419 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T08:26:16.419 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:16.419 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-07-26T08:26:16.419 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:16.420 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:16.420 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-07-26 08:25:24.256906597 +0000 2024-07-26T08:26:16.420 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-07-26 08:15:33.911143017 +0000 2024-07-26T08:26:16.420 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-07-26 08:15:33.911143017 +0000 2024-07-26T08:26:16.420 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-07-26 08:15:33.911143017 +0000 2024-07-26T08:26:16.420 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T08:26:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-07-26T08:26:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-07-26T08:26:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000236125 s, 2.2 MB/s 2024-07-26T08:26:16.490 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T08:26:16.548 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-07-26T08:26:16.605 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T08:26:16.605 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:16.605 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-07-26T08:26:16.605 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:16.606 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:16.606 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-07-26 08:25:24.256906597 +0000 2024-07-26T08:26:16.606 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-07-26 08:15:34.244137126 +0000 2024-07-26T08:26:16.606 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-07-26 08:15:34.244137126 +0000 2024-07-26T08:26:16.606 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-07-26 08:15:34.244137126 +0000 2024-07-26T08:26:16.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T08:26:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-07-26T08:26:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-07-26T08:26:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000238832 s, 2.1 MB/s 2024-07-26T08:26:16.674 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T08:26:16.733 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-07-26T08:26:16.790 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T08:26:16.790 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:16.790 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-07-26T08:26:16.790 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:16.790 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:16.791 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-07-26 08:25:24.256906597 +0000 2024-07-26T08:26:16.791 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-07-26 08:15:34.556131608 +0000 2024-07-26T08:26:16.791 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-07-26 08:15:34.556131608 +0000 2024-07-26T08:26:16.791 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-07-26 08:15:34.556131608 +0000 2024-07-26T08:26:16.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T08:26:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-07-26T08:26:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-07-26T08:26:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000236952 s, 2.2 MB/s 2024-07-26T08:26:16.857 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T08:26:16.914 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-07-26T08:26:16.972 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T08:26:16.972 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-07-26 08:25:24.257906579 +0000 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-07-26 08:15:34.884125806 +0000 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-07-26 08:15:34.884125806 +0000 2024-07-26T08:26:16.973 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-07-26 08:15:34.884125806 +0000 2024-07-26T08:26:16.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T08:26:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-07-26T08:26:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-07-26T08:26:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000231064 s, 2.2 MB/s 2024-07-26T08:26:17.041 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T08:26:17.100 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-07-26T08:26:17.100 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T08:26:17.119 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T08:26:17.120 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-07-26 08:25:21.814744438 +0000 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-07-26 08:15:33.902744721 +0000 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-07-26 08:15:33.902744721 +0000 2024-07-26T08:26:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-07-26 08:15:33.902744721 +0000 2024-07-26T08:26:17.179 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T08:26:17.248 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-07-26T08:26:17.248 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-07-26T08:26:17.248 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000219427 s, 2.3 MB/s 2024-07-26T08:26:17.249 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T08:26:17.307 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-07-26T08:26:17.364 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-07-26 08:25:21.814744438 +0000 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-07-26 08:15:34.230738190 +0000 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-07-26 08:15:34.230738190 +0000 2024-07-26T08:26:17.365 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-07-26 08:15:34.230738190 +0000 2024-07-26T08:26:17.366 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T08:26:17.430 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-07-26T08:26:17.430 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-07-26T08:26:17.430 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.00021439 s, 2.4 MB/s 2024-07-26T08:26:17.431 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T08:26:17.489 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-07-26T08:26:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-07-26 08:25:21.815744419 +0000 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-07-26 08:15:34.552731778 +0000 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-07-26 08:15:34.552731778 +0000 2024-07-26T08:26:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-07-26 08:15:34.552731778 +0000 2024-07-26T08:26:17.545 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T08:26:17.608 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-07-26T08:26:17.608 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-07-26T08:26:17.608 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000206656 s, 2.5 MB/s 2024-07-26T08:26:17.609 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T08:26:17.666 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-07-26 08:25:21.815744419 +0000 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-07-26 08:15:34.879725266 +0000 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-07-26 08:15:34.879725266 +0000 2024-07-26T08:26:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-07-26 08:15:34.879725266 +0000 2024-07-26T08:26:17.722 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T08:26:17.783 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-07-26T08:26:17.783 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-07-26T08:26:17.784 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000236821 s, 2.2 MB/s 2024-07-26T08:26:17.785 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T08:26:17.840 INFO:tasks.cephadm:Deploying osd.0 on smithi044 with /dev/vg_nvme/lv_4... 2024-07-26T08:26:17.841 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T08:26:17.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:17 smithi044 ceph-mon[32404]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:17 smithi116 ceph-mon[37097]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:17 smithi171 ceph-mon[37789]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:18.303 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:26:19.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:19 smithi044 ceph-mon[32404]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:19 smithi116 ceph-mon[37097]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:19 smithi171 ceph-mon[37789]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:20.692 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T08:26:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T08:26:20.931 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-07-26T08:26:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-07-26T08:26:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.034983 s, 300 MB/s 2024-07-26T08:26:20.931 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-07-26T08:26:21.713 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_4 2024-07-26T08:26:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:21 smithi044 ceph-mon[32404]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:21 smithi116 ceph-mon[37097]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:21 smithi171 ceph-mon[37789]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:26:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:23 smithi044 ceph-mon[32404]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:23 smithi116 ceph-mon[37097]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:23 smithi171 ceph-mon[37789]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:24.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 -- 172.21.15.44:0/2553240162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9150108d20 msgr2=0x7f9150109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:24.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 --2- 172.21.15.44:0/2553240162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9150108d20 0x7f9150109130 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f913c009900 tx=0x7f913c02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:26:24.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 -- 172.21.15.44:0/2553240162 shutdown_connections 2024-07-26T08:26:24.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 --2- 172.21.15.44:0/2553240162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9150109670 0x7f915010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:24.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 --2- 172.21.15.44:0/2553240162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9150108d20 0x7f9150109130 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:24.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 --2- 172.21.15.44:0/2553240162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91501033e0 0x7f91501037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:24.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 -- 172.21.15.44:0/2553240162 >> 172.21.15.44:0/2553240162 conn(0x7f91500fee70 msgr2=0x7f91501012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:24.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 -- 172.21.15.44:0/2553240162 shutdown_connections 2024-07-26T08:26:24.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.261+0000 7f9156400640 1 -- 172.21.15.44:0/2553240162 wait complete. 2024-07-26T08:26:24.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 Processor -- start 2024-07-26T08:26:24.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 -- start start 2024-07-26T08:26:24.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 0x7f915019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:24.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9150108d20 0x7f915019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9150109670 0x7f91501a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9150113330 con 0x7f91501033e0 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91501131b0 con 0x7f9150108d20 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9156400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f91501134b0 con 0x7f9150109670 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.263+0000 7f9155000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 0x7f915019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 0x7f915019bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34434/0 (socket says 172.21.15.44:34434) 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 -- 172.21.15.44:0/1585053796 learned_addr learned my addr 172.21.15.44:0/1585053796 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f914fe00640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9150108d20 0x7f915019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155a00640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9150109670 0x7f91501a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 -- 172.21.15.44:0/1585053796 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9150109670 msgr2=0x7f91501a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9150109670 0x7f91501a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:24.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 -- 172.21.15.44:0/1585053796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9150108d20 msgr2=0x7f915019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:24.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9150108d20 0x7f915019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:24.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.264+0000 7f9155000640 1 -- 172.21.15.44:0/1585053796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91501a1f30 con 0x7f91501033e0 2024-07-26T08:26:24.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.265+0000 7f9155000640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 0x7f915019bfc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f914000ebc0 tx=0x7f914000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:24.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.265+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140010630 con 0x7f91501033e0 2024-07-26T08:26:24.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.265+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91501a2220 con 0x7f91501033e0 2024-07-26T08:26:24.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.265+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9140010c80 con 0x7f91501033e0 2024-07-26T08:26:24.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.265+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140018b60 con 0x7f91501033e0 2024-07-26T08:26:24.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.265+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91501a8980 con 0x7f91501033e0 2024-07-26T08:26:24.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.266+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91501037f0 con 0x7f91501033e0 2024-07-26T08:26:24.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.269+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f9140018d00 con 0x7f91501033e0 2024-07-26T08:26:24.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.270+0000 7f914d600640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91340b3b90 0x7f91340b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:24.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.270+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f91400e2680 con 0x7f91501033e0 2024-07-26T08:26:24.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.271+0000 7f914fe00640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91340b3b90 0x7f91340b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:24.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.272+0000 7f914fe00640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91340b3b90 0x7f91340b60a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f913c02a9e0 tx=0x7f913c005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:24.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.275+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f91400ab1e0 con 0x7f91501033e0 2024-07-26T08:26:24.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:24.487+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f915010f170 con 0x7f91340b3b90 2024-07-26T08:26:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:26:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:26:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:26:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:26:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:26:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:26:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:26.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:25 smithi044 ceph-mon[32404]: from='client.14385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:26:26.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:25 smithi044 ceph-mon[32404]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:25 smithi116 ceph-mon[37097]: from='client.14385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:26:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:25 smithi116 ceph-mon[37097]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:25 smithi171 ceph-mon[37789]: from='client.14385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:26:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:25 smithi171 ceph-mon[37789]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:27 smithi044 ceph-mon[32404]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:27 smithi116 ceph-mon[37097]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:27 smithi171 ceph-mon[37789]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:28 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3268920999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ac5961-fb53-49d7-925f-8be1ece073f7"}]: dispatch 2024-07-26T08:26:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:28 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3268920999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96ac5961-fb53-49d7-925f-8be1ece073f7"}]': finished 2024-07-26T08:26:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:28 smithi044 ceph-mon[32404]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T08:26:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:28 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3268920999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ac5961-fb53-49d7-925f-8be1ece073f7"}]: dispatch 2024-07-26T08:26:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:28 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3268920999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96ac5961-fb53-49d7-925f-8be1ece073f7"}]': finished 2024-07-26T08:26:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:28 smithi116 ceph-mon[37097]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T08:26:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:28 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3268920999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ac5961-fb53-49d7-925f-8be1ece073f7"}]: dispatch 2024-07-26T08:26:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:28 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3268920999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96ac5961-fb53-49d7-925f-8be1ece073f7"}]': finished 2024-07-26T08:26:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:28 smithi171 ceph-mon[37789]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T08:26:29.959 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:29 smithi044 ceph-mon[32404]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:29.959 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:29 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1576417423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:26:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:29 smithi116 ceph-mon[37097]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:29 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1576417423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:26:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:29 smithi171 ceph-mon[37789]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:29 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1576417423' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:26:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:31 smithi044 ceph-mon[32404]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:31 smithi116 ceph-mon[37097]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:31 smithi171 ceph-mon[37789]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:33 smithi044 ceph-mon[32404]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:33 smithi116 ceph-mon[37097]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:33 smithi171 ceph-mon[37789]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:35.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:35 smithi044 ceph-mon[32404]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:35 smithi116 ceph-mon[37097]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:35 smithi171 ceph-mon[37789]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:37.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:37 smithi044 ceph-mon[32404]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:37.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T08:26:37.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:37.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:37 smithi044 ceph-mon[32404]: Deploying daemon osd.0 on smithi044 2024-07-26T08:26:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:37 smithi116 ceph-mon[37097]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T08:26:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:37 smithi116 ceph-mon[37097]: Deploying daemon osd.0 on smithi044 2024-07-26T08:26:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:37 smithi171 ceph-mon[37789]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T08:26:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:37 smithi171 ceph-mon[37789]: Deploying daemon osd.0 on smithi044 2024-07-26T08:26:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:39 smithi044 ceph-mon[32404]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:39 smithi116 ceph-mon[37097]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:39 smithi171 ceph-mon[37789]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:41 smithi044 ceph-mon[32404]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:41 smithi116 ceph-mon[37097]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:41 smithi171 ceph-mon[37789]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:43 smithi044 ceph-mon[32404]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:43 smithi116 ceph-mon[37097]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:43 smithi171 ceph-mon[37789]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:45 smithi044 ceph-mon[32404]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:45 smithi116 ceph-mon[37097]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:45 smithi171 ceph-mon[37789]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:47 smithi044 ceph-mon[32404]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:47 smithi116 ceph-mon[37097]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:47 smithi171 ceph-mon[37789]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:49 smithi044 ceph-mon[32404]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:49 smithi171 ceph-mon[37789]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T08:26:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:49 smithi116 ceph-mon[37097]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T08:26:50.523 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 0 on host 'smithi044' 2024-07-26T08:26:50.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.520+0000 7f914d600640 1 -- 172.21.15.44:0/1585053796 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f915010f170 con 0x7f91340b3b90 2024-07-26T08:26:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91340b3b90 msgr2=0x7f91340b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91340b3b90 0x7f91340b60a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f913c02a9e0 tx=0x7f913c005a20 comp rx=0 tx=0).stop 2024-07-26T08:26:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 msgr2=0x7f915019bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 0x7f915019bfc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f914000ebc0 tx=0x7f914000c6a0 comp rx=0 tx=0).stop 2024-07-26T08:26:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 shutdown_connections 2024-07-26T08:26:50.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91340b3b90 0x7f91340b60a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9150109670 0x7f91501a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9150108d20 0x7f915019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 --2- 172.21.15.44:0/1585053796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91501033e0 0x7f915019bfc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.524+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 >> 172.21.15.44:0/1585053796 conn(0x7f91500fee70 msgr2=0x7f91501083f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.525+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 shutdown_connections 2024-07-26T08:26:50.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:26:50.525+0000 7f9156400640 1 -- 172.21.15.44:0/1585053796 wait complete. 2024-07-26T08:26:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T08:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T08:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:50 smithi171 ceph-mon[37789]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T08:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:50 smithi044 ceph-mon[32404]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T08:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:50 smithi116 ceph-mon[37097]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:26:51.381 DEBUG:teuthology.orchestra.run.smithi044:osd.0> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.0.service 2024-07-26T08:26:51.386 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/vg_nvme/lv_4... 2024-07-26T08:26:51.386 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T08:26:51.637 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:26:51.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-mon[32404]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:51.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-mon[32404]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:26:51.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:51.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-mon[32404]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T08:26:51.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:51.881 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-0[52011]: 2024-07-26T08:26:51.788+0000 7f252c600640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T08:26:51.881 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:26:51 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-0[52011]: 2024-07-26T08:26:51.796+0000 7f2527600640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:26:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:51 smithi116 ceph-mon[37097]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:51 smithi116 ceph-mon[37097]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:26:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:51 smithi116 ceph-mon[37097]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T08:26:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:51 smithi171 ceph-mon[37789]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:51 smithi171 ceph-mon[37789]: from='osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:51 smithi171 ceph-mon[37789]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T08:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:52.915 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:52 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:26:52.915 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:52 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:26:52.915 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:52.915 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:52 smithi116 ceph-mon[37097]: osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] boot 2024-07-26T08:26:52.915 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:52 smithi116 ceph-mon[37097]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T08:26:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:52 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:26:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:52 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:26:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:52 smithi044 ceph-mon[32404]: osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] boot 2024-07-26T08:26:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:52 smithi044 ceph-mon[32404]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T08:26:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:52 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:26:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:52 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:26:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T08:26:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:52 smithi171 ceph-mon[37789]: osd.0 [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] boot 2024-07-26T08:26:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:52 smithi171 ceph-mon[37789]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T08:26:53.251 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T08:26:53.338 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T08:26:53.353 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-07-26T08:26:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-07-26T08:26:53.353 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130177 s, 805 MB/s 2024-07-26T08:26:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-07-26T08:26:53.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-07-26T08:26:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:53 smithi116 ceph-mon[37097]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:53 smithi116 ceph-mon[37097]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T08:26:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:53 smithi171 ceph-mon[37789]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:53 smithi171 ceph-mon[37789]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T08:26:54.202 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:26:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:53 smithi044 ceph-mon[32404]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T08:26:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:53 smithi044 ceph-mon[32404]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T08:26:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 -- 172.21.15.116:0/2937876582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 msgr2=0x7f2664108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 --2- 172.21.15.116:0/2937876582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f2664108200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2658009900 tx=0x7f265802a410 comp rx=0 tx=0).stop 2024-07-26T08:26:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 -- 172.21.15.116:0/2937876582 shutdown_connections 2024-07-26T08:26:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 --2- 172.21.15.116:0/2937876582 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2664108740 0x7f266410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 --2- 172.21.15.116:0/2937876582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f2664108200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 --2- 172.21.15.116:0/2937876582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26641024b0 0x7f26641028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 -- 172.21.15.116:0/2937876582 >> 172.21.15.116:0/2937876582 conn(0x7f2664079170 msgr2=0x7f2664079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 -- 172.21.15.116:0/2937876582 shutdown_connections 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.754+0000 7f2669f59640 1 -- 172.21.15.116:0/2937876582 wait complete. 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.755+0000 7f2669f59640 1 Processor -- start 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.755+0000 7f2669f59640 1 -- start start 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26641024b0 0x7f266419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f266419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2664108740 0x7f266419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26641135c0 con 0x7f2664108740 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2664113440 con 0x7f2664107df0 2024-07-26T08:26:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2664113740 con 0x7f26641024b0 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f266419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f266419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59042/0 (socket says 172.21.15.116:59042) 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 -- 172.21.15.116:0/572326686 learned_addr learned my addr 172.21.15.116:0/572326686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f26637fe640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26641024b0 0x7f266419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 -- 172.21.15.116:0/572326686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26641024b0 msgr2=0x7f266419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26641024b0 0x7f266419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 -- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2664108740 msgr2=0x7f266419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2664108740 0x7f266419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 -- 172.21.15.116:0/572326686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266419ffc0 con 0x7f2664107df0 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2662ffd640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f266419a5f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2658031440 tx=0x7f2658003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265803f070 con 0x7f2664107df0 2024-07-26T08:26:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2658031d60 con 0x7f2664107df0 2024-07-26T08:26:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26641a0250 con 0x7f2664107df0 2024-07-26T08:26:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658035030 con 0x7f2664107df0 2024-07-26T08:26:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.756+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26641a6a10 con 0x7f2664107df0 2024-07-26T08:26:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.758+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2664103730 con 0x7f2664107df0 2024-07-26T08:26:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.759+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f2658042070 con 0x7f2664107df0 2024-07-26T08:26:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.760+0000 7f2660ff9640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f264c0b3c10 0x7f264c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:26:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.760+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7f2658102ff0 con 0x7f2664107df0 2024-07-26T08:26:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.760+0000 7f26637fe640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f264c0b3c10 0x7f264c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:26:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.761+0000 7f26637fe640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f264c0b3c10 0x7f264c0b6120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2654009890 tx=0x7f2654008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:26:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.763+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f26580cbfe0 con 0x7f2664107df0 2024-07-26T08:26:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:26:55.947+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2664063230 con 0x7f264c0b3c10 2024-07-26T08:26:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:55 smithi044 ceph-mon[32404]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:26:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:55 smithi171 ceph-mon[37789]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:26:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:55 smithi116 ceph-mon[37097]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:26:56.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:26:56.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi044 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:26:56.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:26:56.948 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 17094M 2024-07-26T08:26:56.948 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:56.948 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:56.948 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:56.948 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:26:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:26:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:26:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi044 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 17094M 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi044 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 17094M 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:26:58.474 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:58 smithi116 ceph-mon[37097]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:26:58.475 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:58 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/220884642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]: dispatch 2024-07-26T08:26:58.475 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:58 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]: dispatch 2024-07-26T08:26:58.475 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:58 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]': finished 2024-07-26T08:26:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:58 smithi044 ceph-mon[32404]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:26:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:58 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/220884642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]: dispatch 2024-07-26T08:26:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:58 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]: dispatch 2024-07-26T08:26:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:58 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]': finished 2024-07-26T08:26:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:58 smithi171 ceph-mon[37789]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:26:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:58 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/220884642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]: dispatch 2024-07-26T08:26:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:58 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]: dispatch 2024-07-26T08:26:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:58 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "853c8479-ad0a-4af4-996a-646245d17441"}]': finished 2024-07-26T08:26:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:26:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:59 smithi044 ceph-mon[32404]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T08:26:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:26:59 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/937352220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:26:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:26:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:59 smithi116 ceph-mon[37097]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T08:26:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:26:59 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/937352220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:26:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:26:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:59 smithi171 ceph-mon[37789]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T08:26:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:26:59 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/937352220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:00 smithi044 ceph-mon[32404]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:00 smithi116 ceph-mon[37097]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:00 smithi171 ceph-mon[37789]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:02.338 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:02 smithi116 ceph-mon[37097]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:02 smithi044 ceph-mon[32404]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:02 smithi171 ceph-mon[37789]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:04 smithi116 ceph-mon[37097]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:04 smithi044 ceph-mon[32404]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:04 smithi171 ceph-mon[37789]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:05.443 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T08:27:05.443 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T08:27:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T08:27:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:06.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:06 smithi116 ceph-mon[37097]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:06.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:06 smithi116 ceph-mon[37097]: Deploying daemon osd.1 on smithi116 2024-07-26T08:27:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:06 smithi044 ceph-mon[32404]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:06 smithi044 ceph-mon[32404]: Deploying daemon osd.1 on smithi116 2024-07-26T08:27:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:06 smithi171 ceph-mon[37789]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:06 smithi171 ceph-mon[37789]: Deploying daemon osd.1 on smithi116 2024-07-26T08:27:08.384 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:08 smithi116 ceph-mon[37097]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:08 smithi044 ceph-mon[32404]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:08 smithi171 ceph-mon[37789]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:10 smithi044 ceph-mon[32404]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:10 smithi116 ceph-mon[37097]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:10 smithi171 ceph-mon[37789]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:12 smithi116 ceph-mon[37097]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:12.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:12.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:12.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:12 smithi044 ceph-mon[32404]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:12 smithi171 ceph-mon[37789]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.494+0000 7f2660ff9640 1 -- 172.21.15.116:0/572326686 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2664063230 con 0x7f264c0b3c10 2024-07-26T08:27:13.495 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2024-07-26T08:27:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f264c0b3c10 msgr2=0x7f264c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f264c0b3c10 0x7f264c0b6120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2654009890 tx=0x7f2654008040 comp rx=0 tx=0).stop 2024-07-26T08:27:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 msgr2=0x7f266419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f266419a5f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2658031440 tx=0x7f2658003d50 comp rx=0 tx=0).stop 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 shutdown_connections 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f264c0b3c10 0x7f264c0b6120 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2664108740 0x7f266419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2664107df0 0x7f266419a5f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 --2- 172.21.15.116:0/572326686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26641024b0 0x7f266419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 >> 172.21.15.116:0/572326686 conn(0x7f2664079170 msgr2=0x7f2664104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 shutdown_connections 2024-07-26T08:27:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:27:13.496+0000 7f2669f59640 1 -- 172.21.15.116:0/572326686 wait complete. 2024-07-26T08:27:14.172 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.1.service 2024-07-26T08:27:14.175 INFO:tasks.cephadm:Deploying osd.2 on smithi171 with /dev/vg_nvme/lv_4... 2024-07-26T08:27:14.175 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T08:27:14.429 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:27:14.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:14 smithi116 ceph-mon[37097]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:14.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:14.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:14.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:14.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:14 smithi116 ceph-mon[37097]: from='osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T08:27:14.453 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:14 smithi116 ceph-mon[37097]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T08:27:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:14 smithi044 ceph-mon[32404]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:14 smithi044 ceph-mon[32404]: from='osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T08:27:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:14 smithi044 ceph-mon[32404]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T08:27:14.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:14 smithi171 ceph-mon[37789]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:14.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:14.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:14.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:14.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:14 smithi171 ceph-mon[37789]: from='osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T08:27:14.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:14 smithi171 ceph-mon[37789]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T08:27:15.676 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-mon[37097]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T08:27:15.676 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:15.676 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-mon[37097]: from='osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:27:15.676 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-mon[37097]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T08:27:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-mon[37097]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:27:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-mon[37097]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:15.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:15 smithi171 ceph-mon[37789]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T08:27:15.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:15.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:15 smithi171 ceph-mon[37789]: from='osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:27:15.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:15 smithi171 ceph-mon[37789]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T08:27:15.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:15 smithi171 ceph-mon[37789]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:27:15.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:15 smithi171 ceph-mon[37789]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:15 smithi044 ceph-mon[32404]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T08:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:15 smithi044 ceph-mon[32404]: from='osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:15 smithi044 ceph-mon[32404]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T08:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:15 smithi044 ceph-mon[32404]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:15 smithi044 ceph-mon[32404]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:16.058 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T08:27:16.144 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T08:27:16.159 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-07-26T08:27:16.159 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-07-26T08:27:16.159 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129247 s, 811 MB/s 2024-07-26T08:27:16.159 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-07-26T08:27:16.272 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-1[42961]: 2024-07-26T08:27:15.930+0000 7f5f75163640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T08:27:16.273 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:15 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-1[42961]: 2024-07-26T08:27:15.934+0000 7f5f6cf59640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:27:16.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:16 smithi116 ceph-mon[37097]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:27:16.700 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:16.700 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:16 smithi116 ceph-mon[37097]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T08:27:16.700 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:16.700 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:16 smithi116 ceph-mon[37097]: from='osd.1 ' entity='osd.1' 2024-07-26T08:27:16.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:16 smithi171 ceph-mon[37789]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:27:16.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:16.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:16 smithi171 ceph-mon[37789]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T08:27:16.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:16.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:16 smithi171 ceph-mon[37789]: from='osd.1 ' entity='osd.1' 2024-07-26T08:27:16.761 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_4 2024-07-26T08:27:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:16 smithi044 ceph-mon[32404]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:27:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:16 smithi044 ceph-mon[32404]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T08:27:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:16 smithi044 ceph-mon[32404]: from='osd.1 ' entity='osd.1' 2024-07-26T08:27:17.013 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:27:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:27:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] boot 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi116 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 17088M 2024-07-26T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] boot 2024-07-26T08:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi116 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 17088M 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:27:17.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:27:17.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:17.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T08:27:17.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: osd.1 [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] boot 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi116 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 17088M 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:17.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:18.527 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2669088100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18107eb0 msgr2=0x7f0f181082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:18.527 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2669088100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18107eb0 0x7f0f181082c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009900 tx=0x7f0f0802a410 comp rx=0 tx=0).stop 2024-07-26T08:27:18.527 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2669088100 shutdown_connections 2024-07-26T08:27:18.527 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2669088100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18107eb0 0x7f0f181082c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:18.527 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2669088100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f18108800 0x7f0f1810f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2669088100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f18102570 0x7f0f18102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2669088100 >> 172.21.15.171:0/2669088100 conn(0x7f0f18079180 msgr2=0x7f0f18079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2669088100 shutdown_connections 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.524+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2669088100 wait complete. 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 Processor -- start 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 -- start start 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f18102570 0x7f0f18199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f18107eb0 0x7f0f1819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 0x7f0f1819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f18113510 con 0x7f0f18102570 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f18113390 con 0x7f0f18107eb0 2024-07-26T08:27:18.528 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1f54c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0f18113690 con 0x7f0f18108800 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 0x7f0f1819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 0x7f0f1819f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:40434/0 (socket says 172.21.15.171:40434) 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 -- 172.21.15.171:0/2601409821 learned_addr learned my addr 172.21.15.171:0/2601409821 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1cae7640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f18107eb0 0x7f0f1819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 -- 172.21.15.171:0/2601409821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f18107eb0 msgr2=0x7f0f1819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f18107eb0 0x7f0f1819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 -- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f18102570 msgr2=0x7f0f18199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f18102570 0x7f0f18199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.525+0000 7f0f1dae9640 1 -- 172.21.15.171:0/2601409821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1819fe70 con 0x7f0f18108800 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.526+0000 7f0f1dae9640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 0x7f0f1819f740 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009fd0 tx=0x7f0f1000b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.526+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f10017070 con 0x7f0f18108800 2024-07-26T08:27:18.529 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.526+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f10003e20 con 0x7f0f18108800 2024-07-26T08:27:18.530 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.526+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f181a0160 con 0x7f0f18108800 2024-07-26T08:27:18.530 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.526+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f10010460 con 0x7f0f18108800 2024-07-26T08:27:18.530 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.526+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f181a68c0 con 0x7f0f18108800 2024-07-26T08:27:18.531 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.527+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f181037f0 con 0x7f0f18108800 2024-07-26T08:27:18.531 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.528+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f0f10003970 con 0x7f0f18108800 2024-07-26T08:27:18.531 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.529+0000 7f0f0e7fc640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0f040b3c10 0x7f0f040b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:18.531 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.529+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2524+0+0 (secure 0 0 0) 0x7f0f100e2610 con 0x7f0f18108800 2024-07-26T08:27:18.531 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.529+0000 7f0f1d2e8640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0f040b3c10 0x7f0f040b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:18.531 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.531+0000 7f0f1d2e8640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0f040b3c10 0x7f0f040b6120 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009710 tx=0x7f0f14006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:27:18.533 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.532+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0f100ab460 con 0x7f0f18108800 2024-07-26T08:27:18.715 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:18.713+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0f18063230 con 0x7f0f040b3c10 2024-07-26T08:27:19.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:19 smithi171 ceph-mon[37789]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T08:27:19.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:19 smithi171 ceph-mon[37789]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:19.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:19 smithi171 ceph-mon[37789]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:27:19.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:27:19.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:27:19.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:19 smithi044 ceph-mon[32404]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T08:27:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:19 smithi044 ceph-mon[32404]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:19 smithi044 ceph-mon[32404]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:27:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:27:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:27:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:19 smithi116 ceph-mon[37097]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T08:27:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:19 smithi116 ceph-mon[37097]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:19 smithi116 ceph-mon[37097]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:27:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:27:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:27:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/4099860262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]: dispatch 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]: dispatch 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]': finished 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T08:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:21 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/3503894013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/4099860262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]: dispatch 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]: dispatch 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]': finished 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T08:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:21 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/3503894013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/4099860262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]: dispatch 2024-07-26T08:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]: dispatch 2024-07-26T08:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "284a11b8-2736-4d1c-90b8-59bdb63c3311"}]': finished 2024-07-26T08:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T08:27:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:21 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/3503894013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:23 smithi044 ceph-mon[32404]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:23 smithi116 ceph-mon[37097]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:23 smithi171 ceph-mon[37789]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:25 smithi171 ceph-mon[37789]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:25 smithi044 ceph-mon[32404]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:25 smithi116 ceph-mon[37097]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:27 smithi044 ceph-mon[32404]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:27 smithi116 ceph-mon[37097]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:27 smithi171 ceph-mon[37789]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T08:27:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:28 smithi171 ceph-mon[37789]: Deploying daemon osd.2 on smithi171 2024-07-26T08:27:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T08:27:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:28 smithi044 ceph-mon[32404]: Deploying daemon osd.2 on smithi171 2024-07-26T08:27:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T08:27:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:28 smithi116 ceph-mon[37097]: Deploying daemon osd.2 on smithi171 2024-07-26T08:27:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:29 smithi044 ceph-mon[32404]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:29 smithi116 ceph-mon[37097]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:29 smithi171 ceph-mon[37789]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:32.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:31 smithi044 ceph-mon[32404]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:31 smithi116 ceph-mon[37097]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:32.072 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:31 smithi171 ceph-mon[37789]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:33.070 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:33.070 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:33.070 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:33 smithi044 ceph-mon[32404]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:33 smithi116 ceph-mon[37097]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:34.063 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:33 smithi171 ceph-mon[37789]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:36.071 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 2 on host 'smithi171' 2024-07-26T08:27:36.072 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.069+0000 7f0f0e7fc640 1 -- 172.21.15.171:0/2601409821 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0f18063230 con 0x7f0f040b3c10 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0f040b3c10 msgr2=0x7f0f040b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0f040b3c10 0x7f0f040b6120 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009710 tx=0x7f0f14006c90 comp rx=0 tx=0).stop 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 msgr2=0x7f0f1819f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 0x7f0f1819f740 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009fd0 tx=0x7f0f1000b260 comp rx=0 tx=0).stop 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 shutdown_connections 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0f040b3c10 0x7f0f040b6120 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f18108800 0x7f0f1819f740 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f18107eb0 0x7f0f1819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 --2- 172.21.15.171:0/2601409821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f18102570 0x7f0f18199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:36.075 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.072+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 >> 172.21.15.171:0/2601409821 conn(0x7f0f18079180 msgr2=0x7f0f18104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:27:36.075 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.073+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 shutdown_connections 2024-07-26T08:27:36.075 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:27:36.073+0000 7f0f1f54c640 1 -- 172.21.15.171:0/2601409821 wait complete. 2024-07-26T08:27:36.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:36.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:36.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:36.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:36.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:36.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:36.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:36.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:36.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:36.614 DEBUG:teuthology.orchestra.run.smithi171:osd.2> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.2.service 2024-07-26T08:27:36.617 INFO:tasks.cephadm:Deploying osd.3 on smithi044 with /dev/vg_nvme/lv_3... 2024-07-26T08:27:36.617 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T08:27:37.082 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:27:37.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:37.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:37 smithi171 ceph-mon[37789]: from='osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T08:27:37.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:37 smithi171 ceph-mon[37789]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T08:27:37.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:37.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:37 smithi044 ceph-mon[32404]: from='osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T08:27:37.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:37 smithi044 ceph-mon[32404]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T08:27:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:37 smithi116 ceph-mon[37097]: from='osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T08:27:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:37 smithi116 ceph-mon[37097]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T08:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-mon[37789]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-mon[37789]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T08:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-mon[37789]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T08:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-mon[37789]: from='osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-mon[37789]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:27:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:38 smithi044 ceph-mon[32404]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:38 smithi044 ceph-mon[32404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T08:27:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:38 smithi044 ceph-mon[32404]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T08:27:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:38 smithi044 ceph-mon[32404]: from='osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:27:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:38 smithi044 ceph-mon[32404]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:27:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:38 smithi116 ceph-mon[37097]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:38 smithi116 ceph-mon[37097]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T08:27:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:38 smithi116 ceph-mon[37097]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T08:27:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:38 smithi116 ceph-mon[37097]: from='osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:27:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:38 smithi116 ceph-mon[37097]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:27:38.826 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-2[44174]: 2024-07-26T08:27:38.517+0000 7f801d936640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T08:27:38.827 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:38 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-2[44174]: 2024-07-26T08:27:38.524+0000 7f801572c640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:27:39.246 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:39 smithi044 ceph-mon[32404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:27:39.246 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.246 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:39 smithi044 ceph-mon[32404]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T08:27:39.246 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.247 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:39 smithi044 ceph-mon[32404]: from='osd.2 ' entity='osd.2' 2024-07-26T08:27:39.247 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.468 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:39 smithi171 ceph-mon[37789]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:27:39.468 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.468 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:39 smithi171 ceph-mon[37789]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T08:27:39.468 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.468 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:39 smithi171 ceph-mon[37789]: from='osd.2 ' entity='osd.2' 2024-07-26T08:27:39.469 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:39 smithi116 ceph-mon[37097]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:27:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:39 smithi116 ceph-mon[37097]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T08:27:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:39 smithi116 ceph-mon[37097]: from='osd.2 ' entity='osd.2' 2024-07-26T08:27:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:39.769 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T08:27:39.988 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T08:27:40.028 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-07-26T08:27:40.028 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-07-26T08:27:40.028 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0333694 s, 314 MB/s 2024-07-26T08:27:40.028 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:40.442 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] boot 2024-07-26T08:27:40.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T08:27:40.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:40.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:40.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:27:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] boot 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: osd.2 [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] boot 2024-07-26T08:27:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T08:27:40.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:40.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:40.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:40.932 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_3 2024-07-26T08:27:41.404 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:27:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:41 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi171 2024-07-26T08:27:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:41 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 17088M 2024-07-26T08:27:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:41 smithi044 ceph-mon[32404]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T08:27:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:41 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi171 2024-07-26T08:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:41 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 17088M 2024-07-26T08:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:41 smithi116 ceph-mon[37097]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T08:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:41 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi171 2024-07-26T08:27:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:41 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 17088M 2024-07-26T08:27:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:41 smithi171 ceph-mon[37789]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T08:27:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:42 smithi044 ceph-mon[32404]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T08:27:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:42 smithi044 ceph-mon[32404]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T08:27:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:42 smithi116 ceph-mon[37097]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T08:27:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:42 smithi116 ceph-mon[37097]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T08:27:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:42 smithi171 ceph-mon[37789]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T08:27:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:42 smithi171 ceph-mon[37789]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T08:27:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T08:27:43.015 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56475]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56475]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56475]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56475]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56479]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56479]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56479]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.016 INFO:journalctl@ceph.osd.0.smithi044.stdout:Jul 26 08:27:42 smithi044 sudo[56479]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.164 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46640]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46640]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46640]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46640]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46643]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46643]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46643]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.165 INFO:journalctl@ceph.osd.1.smithi116.stdout:Jul 26 08:27:42 smithi116 sudo[46643]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.291 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56501]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T08:27:43.291 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56501]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.291 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56501]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56501]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56504]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T08:27:43.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56504]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56504]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 sudo[56504]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48247]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48247]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48247]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48247]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48250]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48250]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48250]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 sudo[48250]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.337 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48241]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T08:27:43.337 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48241]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.337 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48241]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.338 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48241]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.338 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48244]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T08:27:43.338 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48244]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.338 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48244]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.338 INFO:journalctl@ceph.osd.2.smithi171.stdout:Jul 26 08:27:42 smithi171 sudo[48244]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46646]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T08:27:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46646]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46646]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T08:27:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T08:27:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T08:27:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 sudo[46649]: pam_unix(sudo:session): session closed for user root 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 -- 172.21.15.44:0/329557459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 msgr2=0x7f1f7010a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 --2- 172.21.15.44:0/329557459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f7010a4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009900 tx=0x7f1f5c02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 -- 172.21.15.44:0/329557459 shutdown_connections 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 --2- 172.21.15.44:0/329557459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 0x7f1f7011a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 --2- 172.21.15.44:0/329557459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f7010a4b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 --2- 172.21.15.44:0/329557459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f7010c920 0x7f1f7010cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 -- 172.21.15.44:0/329557459 >> 172.21.15.44:0/329557459 conn(0x7f1f70074110 msgr2=0x7f1f70074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 -- 172.21.15.44:0/329557459 shutdown_connections 2024-07-26T08:27:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.625+0000 7f1f77a00640 1 -- 172.21.15.44:0/329557459 wait complete. 2024-07-26T08:27:43.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 Processor -- start 2024-07-26T08:27:43.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 -- start start 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f701a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 0x7f1f701a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f7010c920 0x7f1f701aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7011df50 con 0x7f1f7010ab50 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7011ddd0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.627+0000 7f1f77a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7011e0d0 con 0x7f1f7010c920 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f6fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 0x7f1f701a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:43.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f701a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:43.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f7010c920 0x7f1f701aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:43.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f6fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 0x7f1f701a4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50368/0 (socket says 172.21.15.44:50368) 2024-07-26T08:27:43.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f701a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:58632/0 (socket says 172.21.15.44:58632) 2024-07-26T08:27:43.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f6fe00640 1 -- 172.21.15.44:0/3078242217 learned_addr learned my addr 172.21.15.44:0/3078242217 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 -- 172.21.15.44:0/3078242217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f7010c920 msgr2=0x7f1f701aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f7010c920 0x7f1f701aa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 -- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 msgr2=0x7f1f701a4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 0x7f1f701a4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.628+0000 7f1f75000640 1 -- 172.21.15.44:0/3078242217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f701aa8b0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.629+0000 7f1f75000640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f701a4940 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400e9f0 tx=0x7f1f6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.629+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6400cd80 con 0x7f1f7010a0a0 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.629+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f640035f0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.629+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f701aaba0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.629+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64010670 con 0x7f1f7010a0a0 2024-07-26T08:27:43.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.629+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f701b1300 con 0x7f1f7010a0a0 2024-07-26T08:27:43.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.630+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7010a4b0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.632+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f1f640037a0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.634+0000 7f1f6d600640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1f580b3ca0 0x7f1f580b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:27:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.634+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 3452+0+0 (secure 0 0 0) 0x7f1f640e2be0 con 0x7f1f7010a0a0 2024-07-26T08:27:43.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.634+0000 7f1f6fe00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1f580b3ca0 0x7f1f580b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:27:43.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.635+0000 7f1f6fe00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1f580b3ca0 0x7f1f580b61b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c006010 tx=0x7f1f5c005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:27:43.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.638+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1f640ab490 con 0x7f1f7010a0a0 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:43.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:43.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:43.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:27:43.852+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1f7006f5d0 con 0x7f1f580b3ca0 2024-07-26T08:27:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T08:27:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T08:27:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:44.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:44.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:44.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:44.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:44.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:44.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:44.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:44.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T08:27:44.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T08:27:44.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:44 smithi044 ceph-mon[32404]: osdmap e23: 3 total, 3 up, 3 in 2024-07-26T08:27:44.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:44 smithi044 ceph-mon[32404]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T08:27:44.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:44 smithi044 ceph-mon[32404]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:27:44.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:27:44.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:27:44.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:44 smithi116 ceph-mon[37097]: osdmap e23: 3 total, 3 up, 3 in 2024-07-26T08:27:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:44 smithi116 ceph-mon[37097]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T08:27:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:44 smithi116 ceph-mon[37097]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:27:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:27:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:27:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:44 smithi171 ceph-mon[37789]: osdmap e23: 3 total, 3 up, 3 in 2024-07-26T08:27:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:44 smithi171 ceph-mon[37789]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T08:27:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:44 smithi171 ceph-mon[37789]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:27:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:27:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:27:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:45 smithi044 ceph-mon[32404]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:45 smithi116 ceph-mon[37097]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:45 smithi171 ceph-mon[37789]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:48.014 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:47 smithi044 ceph-mon[32404]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:47 smithi116 ceph-mon[37097]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:47 smithi171 ceph-mon[37789]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:48 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1141646311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]: dispatch 2024-07-26T08:27:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:48 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]: dispatch 2024-07-26T08:27:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:48 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]': finished 2024-07-26T08:27:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:48 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:27:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:48 smithi044 ceph-mon[32404]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T08:27:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:48 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1614988395' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:48 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1141646311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]: dispatch 2024-07-26T08:27:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:48 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]: dispatch 2024-07-26T08:27:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:48 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]': finished 2024-07-26T08:27:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:48 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:27:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:48 smithi116 ceph-mon[37097]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T08:27:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:48 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1614988395' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:48 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1141646311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]: dispatch 2024-07-26T08:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:48 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]: dispatch 2024-07-26T08:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:48 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "895978fc-b2e6-4251-bb79-9bcb77079e00"}]': finished 2024-07-26T08:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:48 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:48 smithi171 ceph-mon[37789]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T08:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:48 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1614988395' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:27:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:49 smithi044 ceph-mon[32404]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:49 smithi116 ceph-mon[37097]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:49 smithi171 ceph-mon[37789]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:51 smithi044 ceph-mon[32404]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:51 smithi116 ceph-mon[37097]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:51 smithi171 ceph-mon[37789]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:53 smithi044 ceph-mon[32404]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:53 smithi116 ceph-mon[37097]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:53 smithi171 ceph-mon[37789]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:55.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:55 smithi044 ceph-mon[32404]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:55 smithi116 ceph-mon[37097]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:55 smithi171 ceph-mon[37789]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:57 smithi044 ceph-mon[32404]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T08:27:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:57 smithi044 ceph-mon[32404]: Deploying daemon osd.3 on smithi044 2024-07-26T08:27:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:57 smithi116 ceph-mon[37097]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T08:27:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:57 smithi116 ceph-mon[37097]: Deploying daemon osd.3 on smithi044 2024-07-26T08:27:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:57 smithi171 ceph-mon[37789]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:27:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T08:27:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:57 smithi171 ceph-mon[37789]: Deploying daemon osd.3 on smithi044 2024-07-26T08:28:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:27:59 smithi044 ceph-mon[32404]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:27:59 smithi116 ceph-mon[37097]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:27:59 smithi171 ceph-mon[37789]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:01.996 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:01 smithi044 ceph-mon[32404]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:01 smithi116 ceph-mon[37097]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:01 smithi171 ceph-mon[37789]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:03.984 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:03 smithi044 ceph-mon[32404]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:03 smithi116 ceph-mon[37097]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:03 smithi171 ceph-mon[37789]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:06.024 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:05 smithi044 ceph-mon[32404]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:06.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:06.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:06.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:05 smithi171 ceph-mon[37789]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:05 smithi116 ceph-mon[37097]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:07 smithi171 ceph-mon[37789]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:08.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:07 smithi044 ceph-mon[32404]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:07 smithi116 ceph-mon[37097]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:10.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T08:28:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.024+0000 7f1f6d600640 1 -- 172.21.15.44:0/3078242217 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1f7006f5d0 con 0x7f1f580b3ca0 2024-07-26T08:28:11.026 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 3 on host 'smithi044' 2024-07-26T08:28:11.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.028+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1f580b3ca0 msgr2=0x7f1f580b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.028+0000 7f1f77a00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1f580b3ca0 0x7f1f580b61b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c006010 tx=0x7f1f5c005a20 comp rx=0 tx=0).stop 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.028+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 msgr2=0x7f1f701a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.028+0000 7f1f77a00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f701a4940 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400e9f0 tx=0x7f1f6400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 shutdown_connections 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1f580b3ca0 0x7f1f580b61b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f7010c920 0x7f1f701aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1f7010ab50 0x7f1f701a4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 --2- 172.21.15.44:0/3078242217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7010a0a0 0x7f1f701a4940 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:11.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 >> 172.21.15.44:0/3078242217 conn(0x7f1f70074110 msgr2=0x7f1f7006eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:28:11.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 shutdown_connections 2024-07-26T08:28:11.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:28:11.029+0000 7f1f77a00640 1 -- 172.21.15.44:0/3078242217 wait complete. 2024-07-26T08:28:11.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T08:28:11.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:11.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T08:28:11.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:28:11.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:11.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T08:28:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T08:28:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:28:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T08:28:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T08:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:11.922 DEBUG:teuthology.orchestra.run.smithi044:osd.3> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.3.service 2024-07-26T08:28:11.927 INFO:tasks.cephadm:Deploying osd.4 on smithi116 with /dev/vg_nvme/lv_3... 2024-07-26T08:28:11.927 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T08:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:28:12.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:12.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:28:12.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: osdmap e26: 4 total, 3 up, 4 in 2024-07-26T08:28:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] boot 2024-07-26T08:28:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:12 smithi044 ceph-mon[32404]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T08:28:12.500 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:12.500 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:28:12.500 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: osdmap e26: 4 total, 3 up, 4 in 2024-07-26T08:28:12.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] boot 2024-07-26T08:28:12.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:12 smithi116 ceph-mon[37097]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T08:28:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: from='osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: osdmap e26: 4 total, 3 up, 4 in 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: osd.3 [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] boot 2024-07-26T08:28:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:12 smithi171 ceph-mon[37789]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T08:28:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:13 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:28:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:13 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:28:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:13 smithi044 ceph-mon[32404]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T08:28:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:13 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:28:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:13 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:28:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:13 smithi116 ceph-mon[37097]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T08:28:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:13 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:28:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:13 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:28:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:13 smithi171 ceph-mon[37789]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T08:28:13.959 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T08:28:14.042 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T08:28:14.056 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-07-26T08:28:14.056 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-07-26T08:28:14.056 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113749 s, 922 MB/s 2024-07-26T08:28:14.056 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-07-26T08:28:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:14 smithi044 ceph-mon[32404]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:14 smithi044 ceph-mon[32404]: osdmap e29: 4 total, 4 up, 4 in 2024-07-26T08:28:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:14 smithi116 ceph-mon[37097]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:14 smithi116 ceph-mon[37097]: osdmap e29: 4 total, 4 up, 4 in 2024-07-26T08:28:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:14 smithi171 ceph-mon[37789]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:14 smithi171 ceph-mon[37789]: osdmap e29: 4 total, 4 up, 4 in 2024-07-26T08:28:14.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-07-26T08:28:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:28:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:16 smithi044 ceph-mon[32404]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:16.329 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:16 smithi116 ceph-mon[37097]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 -- 172.21.15.116:0/2830799015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 msgr2=0x7f52e810efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 --2- 172.21.15.116:0/2830799015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e810efd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009ca0 tx=0x7f52e002a8e0 comp rx=0 tx=0).stop 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 -- 172.21.15.116:0/2830799015 shutdown_connections 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 --2- 172.21.15.116:0/2830799015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8107e50 0x7f52e8108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 --2- 172.21.15.116:0/2830799015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e810efd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 --2- 172.21.15.116:0/2830799015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52e8102510 0x7f52e8102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 -- 172.21.15.116:0/2830799015 >> 172.21.15.116:0/2830799015 conn(0x7f52e80791a0 msgr2=0x7f52e80795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:28:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 -- 172.21.15.116:0/2830799015 shutdown_connections 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.535+0000 7f52ef828640 1 -- 172.21.15.116:0/2830799015 wait complete. 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 Processor -- start 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 -- start start 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8102510 0x7f52e810fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52e8107e50 0x7f52e810fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e8110530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8113620 con 0x7f52e8107e50 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81134a0 con 0x7f52e81087a0 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ef828640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81137a0 con 0x7f52e8102510 2024-07-26T08:28:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52eddc5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e8110530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52eddc5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e8110530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48440/0 (socket says 172.21.15.116:48440) 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52eddc5640 1 -- 172.21.15.116:0/2764341998 learned_addr learned my addr 172.21.15.116:0/2764341998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.536+0000 7f52ed5c4640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8102510 0x7f52e810fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52eddc5640 1 -- 172.21.15.116:0/2764341998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8102510 msgr2=0x7f52e810fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52eddc5640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8102510 0x7f52e810fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52eddc5640 1 -- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52e8107e50 msgr2=0x7f52e810fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52eddc5640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52e8107e50 0x7f52e810fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52eddc5640 1 -- 172.21.15.116:0/2764341998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e81a8230 con 0x7f52e81087a0 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52ed5c4640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8102510 0x7f52e810fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52eddc5640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e8110530 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52e0005cc0 tx=0x7f52e0003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:28:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0004040 con 0x7f52e81087a0 2024-07-26T08:28:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0004690 con 0x7f52e81087a0 2024-07-26T08:28:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e81a84c0 con 0x7f52e81087a0 2024-07-26T08:28:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0030e70 con 0x7f52e81087a0 2024-07-26T08:28:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.537+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e81a8ad0 con 0x7f52e81087a0 2024-07-26T08:28:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.538+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e8103790 con 0x7f52e81087a0 2024-07-26T08:28:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.540+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f52e00305e0 con 0x7f52e81087a0 2024-07-26T08:28:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.541+0000 7f52de7fc640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f52d40b3ca0 0x7f52d40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.541+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3884+0+0 (secure 0 0 0) 0x7f52e0104200 con 0x7f52e81087a0 2024-07-26T08:28:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.542+0000 7f52ed5c4640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f52d40b3ca0 0x7f52d40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.543+0000 7f52ed5c4640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f52d40b3ca0 0x7f52d40b61b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52d8006fd0 tx=0x7f52d8008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:28:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.544+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f52e00cc9a0 con 0x7f52e81087a0 2024-07-26T08:28:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:16 smithi171 ceph-mon[37789]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:16.726+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f52e8063230 con 0x7f52d40b3ca0 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:17.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:18 smithi044 ceph-mon[32404]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:18 smithi044 ceph-mon[32404]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:28:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:18 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi044 2024-07-26T08:28:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:18 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 8547M 2024-07-26T08:28:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:18 smithi116 ceph-mon[37097]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:18 smithi116 ceph-mon[37097]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:28:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:18 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi044 2024-07-26T08:28:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:18 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 8547M 2024-07-26T08:28:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:18 smithi171 ceph-mon[37789]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:18 smithi171 ceph-mon[37789]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:28:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:18 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi044 2024-07-26T08:28:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:18 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 8547M 2024-07-26T08:28:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:19 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/2447277363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]: dispatch 2024-07-26T08:28:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:19 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]: dispatch 2024-07-26T08:28:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:19 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]': finished 2024-07-26T08:28:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:19 smithi044 ceph-mon[32404]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T08:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:19 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/2447277363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]: dispatch 2024-07-26T08:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:19 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]: dispatch 2024-07-26T08:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:19 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]': finished 2024-07-26T08:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:19 smithi116 ceph-mon[37097]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T08:28:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:19 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/2447277363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]: dispatch 2024-07-26T08:28:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:19 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]: dispatch 2024-07-26T08:28:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:19 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e29d022-e84f-43f7-badb-18797f4a3f67"}]': finished 2024-07-26T08:28:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:19 smithi171 ceph-mon[37789]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T08:28:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:20 smithi044 ceph-mon[32404]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-07-26T08:28:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:20 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/1339552994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:28:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:20 smithi116 ceph-mon[37097]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-07-26T08:28:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:20 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/1339552994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:28:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:20 smithi171 ceph-mon[37789]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-07-26T08:28:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:20 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/1339552994' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:28:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:22 smithi044 ceph-mon[32404]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-07-26T08:28:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:22 smithi116 ceph-mon[37097]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-07-26T08:28:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:22 smithi171 ceph-mon[37789]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-07-26T08:28:24.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:24 smithi116 ceph-mon[37097]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-07-26T08:28:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:24 smithi171 ceph-mon[37789]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-07-26T08:28:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:24 smithi044 ceph-mon[32404]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-07-26T08:28:26.506 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:26 smithi116 ceph-mon[37097]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:28:26.506 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T08:28:26.506 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:26 smithi171 ceph-mon[37789]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:28:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T08:28:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:26 smithi044 ceph-mon[32404]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:28:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T08:28:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:27.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:27 smithi116 ceph-mon[37097]: Deploying daemon osd.4 on smithi116 2024-07-26T08:28:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:27 smithi171 ceph-mon[37789]: Deploying daemon osd.4 on smithi116 2024-07-26T08:28:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:27 smithi044 ceph-mon[32404]: Deploying daemon osd.4 on smithi116 2024-07-26T08:28:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:28 smithi171 ceph-mon[37789]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:28:28.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:28 smithi044 ceph-mon[32404]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:28:28.670 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:28 smithi116 ceph-mon[37097]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:28:30.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:30 smithi044 ceph-mon[32404]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:30.514 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:30 smithi116 ceph-mon[37097]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:30 smithi171 ceph-mon[37789]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:31.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:31.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:31.561 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:32 smithi171 ceph-mon[37789]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:32.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:32 smithi116 ceph-mon[37097]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:32 smithi044 ceph-mon[32404]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:34.454 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:34 smithi116 ceph-mon[37097]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:34.517 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 4 on host 'smithi116' 2024-07-26T08:28:34.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.515+0000 7f52de7fc640 1 -- 172.21.15.116:0/2764341998 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f52e8063230 con 0x7f52d40b3ca0 2024-07-26T08:28:34.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f52d40b3ca0 msgr2=0x7f52d40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:34.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f52d40b3ca0 0x7f52d40b61b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52d8006fd0 tx=0x7f52d8008040 comp rx=0 tx=0).stop 2024-07-26T08:28:34.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 msgr2=0x7f52e8110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:34.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e8110530 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52e0005cc0 tx=0x7f52e0003ba0 comp rx=0 tx=0).stop 2024-07-26T08:28:34.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 shutdown_connections 2024-07-26T08:28:34.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f52d40b3ca0 0x7f52d40b61b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:34.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81087a0 0x7f52e8110530 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:34.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f52e8107e50 0x7f52e810fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:34.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 --2- 172.21.15.116:0/2764341998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52e8102510 0x7f52e810fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:34.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 >> 172.21.15.116:0/2764341998 conn(0x7f52e80791a0 msgr2=0x7f52e8105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:28:34.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 shutdown_connections 2024-07-26T08:28:34.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:28:34.518+0000 7f52ef828640 1 -- 172.21.15.116:0/2764341998 wait complete. 2024-07-26T08:28:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:34 smithi171 ceph-mon[37789]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:34 smithi044 ceph-mon[32404]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:35.202 DEBUG:teuthology.orchestra.run.smithi116:osd.4> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.4.service 2024-07-26T08:28:35.205 INFO:tasks.cephadm:Deploying osd.5 on smithi171 with /dev/vg_nvme/lv_3... 2024-07-26T08:28:35.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T08:28:35.465 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: from='osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:35.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: from='osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T08:28:35.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T08:28:35.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:35.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: from='osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T08:28:35.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T08:28:35.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:35.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:36.788 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-mon[37097]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T08:28:36.788 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:36.788 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-mon[37097]: from='osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:28:36.788 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-mon[37097]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T08:28:36.788 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-mon[37097]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:28:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:36 smithi171 ceph-mon[37789]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T08:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:36 smithi171 ceph-mon[37789]: from='osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:36 smithi171 ceph-mon[37789]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T08:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:36 smithi171 ceph-mon[37789]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:36 smithi044 ceph-mon[32404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T08:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:36 smithi044 ceph-mon[32404]: from='osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:36 smithi044 ceph-mon[32404]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T08:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:36 smithi044 ceph-mon[32404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:28:37.042 INFO:journalctl@ceph.osd.4.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-4[49413]: 2024-07-26T08:28:36.966+0000 7f3a0a813640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T08:28:37.042 INFO:journalctl@ceph.osd.4.smithi116.stdout:Jul 26 08:28:36 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-4[49413]: 2024-07-26T08:28:36.972+0000 7f3a02609640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:28:37.365 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T08:28:37.450 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T08:28:37.464 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-07-26T08:28:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-07-26T08:28:37.464 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114955 s, 912 MB/s 2024-07-26T08:28:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-07-26T08:28:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:37 smithi116 ceph-mon[37097]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:28:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:37 smithi116 ceph-mon[37097]: osdmap e32: 5 total, 4 up, 5 in 2024-07-26T08:28:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:37 smithi116 ceph-mon[37097]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:37 smithi116 ceph-mon[37097]: from='osd.4 ' entity='osd.4' 2024-07-26T08:28:37.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:37 smithi171 ceph-mon[37789]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:28:37.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:37.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:37 smithi171 ceph-mon[37789]: osdmap e32: 5 total, 4 up, 5 in 2024-07-26T08:28:37.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:37.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:37 smithi171 ceph-mon[37789]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:37.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:37 smithi171 ceph-mon[37789]: from='osd.4 ' entity='osd.4' 2024-07-26T08:28:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:37 smithi044 ceph-mon[32404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:28:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:37 smithi044 ceph-mon[32404]: osdmap e32: 5 total, 4 up, 5 in 2024-07-26T08:28:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:37 smithi044 ceph-mon[32404]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-26T08:28:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:37 smithi044 ceph-mon[32404]: from='osd.4 ' entity='osd.4' 2024-07-26T08:28:38.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_3 2024-07-26T08:28:38.362 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:28:38.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:28:38.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] boot 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] boot 2024-07-26T08:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T08:28:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: osd.4 [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] boot 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:28:39.761 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi116 2024-07-26T08:28:39.761 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 8544M 2024-07-26T08:28:39.761 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:39.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:39 smithi171 ceph-mon[37789]: osdmap e34: 5 total, 5 up, 5 in 2024-07-26T08:28:39.947 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 -- 172.21.15.171:0/1578520543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8102490 msgr2=0x7f21a81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1578520543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8102490 0x7f21a81028a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2198009900 tx=0x7f219802a410 comp rx=0 tx=0).stop 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 -- 172.21.15.171:0/1578520543 shutdown_connections 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1578520543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a8108720 0x7f21a810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1578520543 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21a8107dd0 0x7f21a81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1578520543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8102490 0x7f21a81028a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.943+0000 7f21ade7d640 1 -- 172.21.15.171:0/1578520543 >> 172.21.15.171:0/1578520543 conn(0x7f21a8079160 msgr2=0x7f21a8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 -- 172.21.15.171:0/1578520543 shutdown_connections 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 -- 172.21.15.171:0/1578520543 wait complete. 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 Processor -- start 2024-07-26T08:28:39.948 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 -- start start 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a8102490 0x7f21a819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21a8107dd0 0x7f21a819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 0x7f21a819b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21a8112440 con 0x7f21a8107dd0 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21a81122c0 con 0x7f21a8102490 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.944+0000 7f21ade7d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f21a81125c0 con 0x7f21a8108720 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 0x7f21a819b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 0x7f21a819b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:47050/0 (socket says 172.21.15.171:47050) 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 -- 172.21.15.171:0/1946735355 learned_addr learned my addr 172.21.15.171:0/1946735355 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a77fe640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a8102490 0x7f21a819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 -- 172.21.15.171:0/1946735355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a8102490 msgr2=0x7f21a819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:39.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a8102490 0x7f21a819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 -- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21a8107dd0 msgr2=0x7f21a819b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21a8107dd0 0x7f21a819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 -- 172.21.15.171:0/1946735355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a81a5fa0 con 0x7f21a8108720 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a7fff640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 0x7f21a819b790 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21a000cad0 tx=0x7f21a000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a000ae20 con 0x7f21a8108720 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21a0005020 con 0x7f21a8108720 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a81a6290 con 0x7f21a8108720 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a000f6b0 con 0x7f21a8108720 2024-07-26T08:28:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.945+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a81a6800 con 0x7f21a8108720 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.946+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a81028a0 con 0x7f21a8108720 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.948+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f21a00051d0 con 0x7f21a8108720 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.948+0000 7f21a57fa640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21940b3c50 0x7f21940b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.949+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4316+0+0 (secure 0 0 0) 0x7f21a00e3220 con 0x7f21a8108720 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.949+0000 7f21a77fe640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21940b3c50 0x7f21940b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.950+0000 7f21a77fe640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21940b3c50 0x7f21940b6160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2198006010 tx=0x7f2198031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:28:39.954 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:39.952+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f21a0014030 con 0x7f21a8108720 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi116 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 8544M 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:39 smithi044 ceph-mon[32404]: osdmap e34: 5 total, 5 up, 5 in 2024-07-26T08:28:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi116 2024-07-26T08:28:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 8544M 2024-07-26T08:28:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:39 smithi116 ceph-mon[37097]: osdmap e34: 5 total, 5 up, 5 in 2024-07-26T08:28:40.137 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:40.135+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f21a8063230 con 0x7f21940b3c50 2024-07-26T08:28:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:40 smithi171 ceph-mon[37789]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:28:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:28:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:28:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:40 smithi044 ceph-mon[32404]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:28:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:28:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:28:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:40 smithi116 ceph-mon[37097]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:28:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:28:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:28:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:41 smithi044 ceph-mon[32404]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:41.903 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:41 smithi171 ceph-mon[37789]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:41 smithi116 ceph-mon[37097]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:42 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/2231681987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]: dispatch 2024-07-26T08:28:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:42 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]: dispatch 2024-07-26T08:28:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:42 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]': finished 2024-07-26T08:28:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:28:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:42 smithi044 ceph-mon[32404]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T08:28:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:42 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/2231681987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]: dispatch 2024-07-26T08:28:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:42 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]: dispatch 2024-07-26T08:28:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:42 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]': finished 2024-07-26T08:28:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:28:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:42 smithi116 ceph-mon[37097]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T08:28:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:42 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/2231681987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]: dispatch 2024-07-26T08:28:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:42 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]: dispatch 2024-07-26T08:28:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:42 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71c826e9-696b-47de-ae4c-023b02ceb6b4"}]': finished 2024-07-26T08:28:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:28:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:42 smithi171 ceph-mon[37789]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T08:28:43.984 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:43 smithi116 ceph-mon[37097]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:43.984 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:43 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/1348231846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:28:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:43 smithi044 ceph-mon[32404]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:43 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/1348231846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:28:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:43 smithi171 ceph-mon[37789]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:43 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/1348231846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:28:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:45 smithi044 ceph-mon[32404]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:45 smithi116 ceph-mon[37097]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:46.034 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:45 smithi171 ceph-mon[37789]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:47 smithi171 ceph-mon[37789]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:47 smithi044 ceph-mon[32404]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:47 smithi116 ceph-mon[37097]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:49.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:49 smithi171 ceph-mon[37789]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:49.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T08:28:49.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:49.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:49 smithi171 ceph-mon[37789]: Deploying daemon osd.5 on smithi171 2024-07-26T08:28:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:49 smithi044 ceph-mon[32404]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T08:28:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:49 smithi044 ceph-mon[32404]: Deploying daemon osd.5 on smithi171 2024-07-26T08:28:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:49 smithi116 ceph-mon[37097]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T08:28:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:49 smithi116 ceph-mon[37097]: Deploying daemon osd.5 on smithi171 2024-07-26T08:28:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:51 smithi044 ceph-mon[32404]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:51 smithi116 ceph-mon[37097]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:52.028 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:51 smithi171 ceph-mon[37789]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:54.015 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:53 smithi171 ceph-mon[37789]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:53 smithi044 ceph-mon[32404]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:53 smithi116 ceph-mon[37097]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:54.981 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:54.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:54.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:55 smithi044 ceph-mon[32404]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:55 smithi116 ceph-mon[37097]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:55 smithi171 ceph-mon[37789]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:58.210 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.207+0000 7f21a57fa640 1 -- 172.21.15.171:0/1946735355 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f21a8063230 con 0x7f21940b3c50 2024-07-26T08:28:58.210 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 5 on host 'smithi171' 2024-07-26T08:28:58.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21940b3c50 msgr2=0x7f21940b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:58.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21940b3c50 0x7f21940b6160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2198006010 tx=0x7f2198031400 comp rx=0 tx=0).stop 2024-07-26T08:28:58.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 msgr2=0x7f21a819b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:28:58.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 0x7f21a819b790 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21a000cad0 tx=0x7f21a000cf90 comp rx=0 tx=0).stop 2024-07-26T08:28:58.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 shutdown_connections 2024-07-26T08:28:58.211 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21940b3c50 0x7f21940b6160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:58.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21a8108720 0x7f21a819b790 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:58.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21a8107dd0 0x7f21a819b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:58.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 --2- 172.21.15.171:0/1946735355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a8102490 0x7f21a819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:28:58.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 >> 172.21.15.171:0/1946735355 conn(0x7f21a8079160 msgr2=0x7f21a8100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:28:58.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 shutdown_connections 2024-07-26T08:28:58.212 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:28:58.210+0000 7f21ade7d640 1 -- 172.21.15.171:0/1946735355 wait complete. 2024-07-26T08:28:58.473 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:28:58.876 DEBUG:teuthology.orchestra.run.smithi171:osd.5> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.5.service 2024-07-26T08:28:58.879 INFO:tasks.cephadm:Deploying osd.6 on smithi044 with /dev/vg_nvme/lv_2... 2024-07-26T08:28:58.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T08:28:59.335 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:28:59.408 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:59 smithi044 ceph-mon[32404]: from='osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T08:28:59.409 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:28:59 smithi044 ceph-mon[32404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T08:28:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:59 smithi171 ceph-mon[37789]: from='osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T08:28:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:28:59 smithi171 ceph-mon[37789]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T08:28:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:59 smithi116 ceph-mon[37097]: from='osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T08:28:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:28:59 smithi116 ceph-mon[37097]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T08:29:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-mon[37789]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:29:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-mon[37789]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T08:29:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-mon[37789]: from='osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:29:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-mon[37789]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T08:29:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-mon[37789]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:00 smithi044 ceph-mon[32404]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:00 smithi044 ceph-mon[32404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T08:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:00 smithi044 ceph-mon[32404]: from='osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:00 smithi044 ceph-mon[32404]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T08:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:00 smithi044 ceph-mon[32404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:29:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:00 smithi116 ceph-mon[37097]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:29:00.814 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:00 smithi116 ceph-mon[37097]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T08:29:00.814 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:00 smithi116 ceph-mon[37097]: from='osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:29:00.814 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:00.814 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:00 smithi116 ceph-mon[37097]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T08:29:00.814 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:00 smithi116 ceph-mon[37097]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:29:01.073 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-5[51079]: 2024-07-26T08:29:00.803+0000 7f3b84e92640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T08:29:01.073 INFO:journalctl@ceph.osd.5.smithi171.stdout:Jul 26 08:29:00 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-5[51079]: 2024-07-26T08:29:00.810+0000 7f3b7cc88640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:29:01.655 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:01 smithi171 ceph-mon[37789]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:29:01.655 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:01 smithi171 ceph-mon[37789]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T08:29:01.655 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:01.655 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:01.655 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:01 smithi171 ceph-mon[37789]: from='osd.5 ' entity='osd.5' 2024-07-26T08:29:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:01 smithi044 ceph-mon[32404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:29:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:01 smithi044 ceph-mon[32404]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T08:29:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:01 smithi044 ceph-mon[32404]: from='osd.5 ' entity='osd.5' 2024-07-26T08:29:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:01 smithi116 ceph-mon[37097]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:29:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:01 smithi116 ceph-mon[37097]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T08:29:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:01 smithi116 ceph-mon[37097]: from='osd.5 ' entity='osd.5' 2024-07-26T08:29:01.955 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T08:29:02.159 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T08:29:02.197 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-07-26T08:29:02.197 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-07-26T08:29:02.198 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.031914 s, 329 MB/s 2024-07-26T08:29:02.198 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-07-26T08:29:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:29:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] boot 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:02.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:29:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:29:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:29:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] boot 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:02.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:02.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: osd.5 [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] boot 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:03.119 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_2 2024-07-26T08:29:03.596 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:29:03.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:03 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi171 2024-07-26T08:29:03.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:03 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 8544M 2024-07-26T08:29:03.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:03 smithi116 ceph-mon[37097]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T08:29:03.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:03 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi171 2024-07-26T08:29:03.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:03 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 8544M 2024-07-26T08:29:03.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:03 smithi044 ceph-mon[32404]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T08:29:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:03 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi171 2024-07-26T08:29:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:03 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 8544M 2024-07-26T08:29:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:03 smithi171 ceph-mon[37789]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T08:29:04.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:04 smithi116 ceph-mon[37097]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:04.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:04 smithi116 ceph-mon[37097]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T08:29:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:04 smithi044 ceph-mon[32404]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:04 smithi044 ceph-mon[32404]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T08:29:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:04 smithi171 ceph-mon[37789]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:04 smithi171 ceph-mon[37789]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T08:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 -- 172.21.15.44:0/2911667728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b681055e0 msgr2=0x7f4b681059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 --2- 172.21.15.44:0/2911667728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b681055e0 0x7f4b681059f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009900 tx=0x7f4b5802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:29:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 -- 172.21.15.44:0/2911667728 shutdown_connections 2024-07-26T08:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 --2- 172.21.15.44:0/2911667728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b6810af20 0x7f4b6810b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 --2- 172.21.15.44:0/2911667728 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b6810b870 0x7f4b681120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 --2- 172.21.15.44:0/2911667728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b681055e0 0x7f4b681059f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 -- 172.21.15.44:0/2911667728 >> 172.21.15.44:0/2911667728 conn(0x7f4b681010b0 msgr2=0x7f4b68103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.839+0000 7f4b6e600640 1 -- 172.21.15.44:0/2911667728 shutdown_connections 2024-07-26T08:29:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.840+0000 7f4b6e600640 1 -- 172.21.15.44:0/2911667728 wait complete. 2024-07-26T08:29:05.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 Processor -- start 2024-07-26T08:29:05.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 -- start start 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b681055e0 0x7f4b6819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 0x7f4b6819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b6810b870 0x7f4b681a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4b68115520 con 0x7f4b681055e0 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681153a0 con 0x7f4b6810af20 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.841+0000 7f4b6e600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b681156a0 con 0x7f4b6810b870 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b6d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b681055e0 0x7f4b6819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 0x7f4b6819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 0x7f4b6819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47504/0 (socket says 172.21.15.44:47504) 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 -- 172.21.15.44:0/395710700 learned_addr learned my addr 172.21.15.44:0/395710700 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:29:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b6d200640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b681055e0 0x7f4b6819bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41670/0 (socket says 172.21.15.44:41670) 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b6dc00640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b6810b870 0x7f4b681a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 -- 172.21.15.44:0/395710700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b6810b870 msgr2=0x7f4b681a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b6810b870 0x7f4b681a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 -- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b681055e0 msgr2=0x7f4b6819bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b681055e0 0x7f4b6819bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 -- 172.21.15.44:0/395710700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b681a1ec0 con 0x7f4b6810af20 2024-07-26T08:29:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.842+0000 7f4b67e00640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 0x7f4b6819c4f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400d950 tx=0x7f4b5400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:29:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.843+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54017070 con 0x7f4b6810af20 2024-07-26T08:29:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.843+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b54009c30 con 0x7f4b6810af20 2024-07-26T08:29:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.843+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b540047a0 con 0x7f4b6810af20 2024-07-26T08:29:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.843+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b681a21b0 con 0x7f4b6810af20 2024-07-26T08:29:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.843+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b681a8910 con 0x7f4b6810af20 2024-07-26T08:29:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.844+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b681059f0 con 0x7f4b6810af20 2024-07-26T08:29:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.847+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f4b54004020 con 0x7f4b6810af20 2024-07-26T08:29:05.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.849+0000 7f4b65600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b4c0b3ca0 0x7f4b4c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:05.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.849+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4748+0+0 (secure 0 0 0) 0x7f4b540e2ed0 con 0x7f4b6810af20 2024-07-26T08:29:05.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.849+0000 7f4b6d200640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b4c0b3ca0 0x7f4b4c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.850+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b54020020 con 0x7f4b6810af20 2024-07-26T08:29:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:05.850+0000 7f4b6d200640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b4c0b3ca0 0x7f4b4c0b61b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4b58031440 tx=0x7f4b58009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:29:06.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:06.063+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4b68063230 con 0x7f4b4c0b3ca0 2024-07-26T08:29:06.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:06 smithi171 ceph-mon[37789]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:06.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:29:06.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:29:06.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:06 smithi044 ceph-mon[32404]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:29:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:29:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:06 smithi116 ceph-mon[37097]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 190 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:29:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:29:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:07 smithi044 ceph-mon[32404]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:29:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:07 smithi116 ceph-mon[37097]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:29:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:07 smithi171 ceph-mon[37789]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:29:08.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:08 smithi044 ceph-mon[32404]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:08 smithi116 ceph-mon[37097]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:08 smithi171 ceph-mon[37789]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:10.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:10 smithi044 ceph-mon[32404]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T08:29:10.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:10 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1545575638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "275225ee-11a7-4988-851b-7be0b8d4d155"}]: dispatch 2024-07-26T08:29:10.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:10 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1545575638' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "275225ee-11a7-4988-851b-7be0b8d4d155"}]': finished 2024-07-26T08:29:10.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:10.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:10 smithi044 ceph-mon[32404]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T08:29:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:10 smithi116 ceph-mon[37097]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T08:29:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:10 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1545575638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "275225ee-11a7-4988-851b-7be0b8d4d155"}]: dispatch 2024-07-26T08:29:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:10 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1545575638' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "275225ee-11a7-4988-851b-7be0b8d4d155"}]': finished 2024-07-26T08:29:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:10 smithi116 ceph-mon[37097]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T08:29:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:10 smithi171 ceph-mon[37789]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T08:29:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:10 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1545575638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "275225ee-11a7-4988-851b-7be0b8d4d155"}]: dispatch 2024-07-26T08:29:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:10 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1545575638' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "275225ee-11a7-4988-851b-7be0b8d4d155"}]': finished 2024-07-26T08:29:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:10 smithi171 ceph-mon[37789]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T08:29:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:11 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3440498410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:29:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:11 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3440498410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:29:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:11 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3440498410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:29:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:12 smithi044 ceph-mon[32404]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:29:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:12 smithi116 ceph-mon[37097]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:29:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:12 smithi171 ceph-mon[37789]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:29:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:14 smithi044 ceph-mon[32404]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T08:29:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:14 smithi116 ceph-mon[37097]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T08:29:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:14 smithi171 ceph-mon[37789]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T08:29:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:16 smithi044 ceph-mon[32404]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:29:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:16 smithi116 ceph-mon[37097]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:29:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:16 smithi171 ceph-mon[37789]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:29:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:18 smithi044 ceph-mon[32404]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:29:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:18 smithi116 ceph-mon[37097]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:29:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:18 smithi171 ceph-mon[37789]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:29:20.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:20 smithi044 ceph-mon[32404]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:20.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T08:29:20.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:20.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:20 smithi044 ceph-mon[32404]: Deploying daemon osd.6 on smithi044 2024-07-26T08:29:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:20 smithi116 ceph-mon[37097]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T08:29:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:20 smithi116 ceph-mon[37097]: Deploying daemon osd.6 on smithi044 2024-07-26T08:29:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:20 smithi171 ceph-mon[37789]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T08:29:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:20 smithi171 ceph-mon[37789]: Deploying daemon osd.6 on smithi044 2024-07-26T08:29:22.652 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:22 smithi044 ceph-mon[32404]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:22 smithi116 ceph-mon[37097]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:22 smithi171 ceph-mon[37789]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:24 smithi044 ceph-mon[32404]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:24 smithi116 ceph-mon[37097]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:24 smithi171 ceph-mon[37789]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:26.722 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:26 smithi044 ceph-mon[32404]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:26 smithi116 ceph-mon[37097]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:26 smithi171 ceph-mon[37789]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:28 smithi044 ceph-mon[32404]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:28 smithi116 ceph-mon[37097]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:28 smithi171 ceph-mon[37789]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:30 smithi044 ceph-mon[32404]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:30 smithi116 ceph-mon[37097]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:30 smithi171 ceph-mon[37789]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:32.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:32 smithi044 ceph-mon[32404]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:32.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:32.727 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:32 smithi116 ceph-mon[37097]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:32 smithi171 ceph-mon[37789]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:33 smithi044 ceph-mon[32404]: from='osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T08:29:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:33 smithi044 ceph-mon[32404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T08:29:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:33 smithi116 ceph-mon[37097]: from='osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T08:29:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:33 smithi116 ceph-mon[37097]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T08:29:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:33.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.776+0000 7f4b65600640 1 -- 172.21.15.44:0/395710700 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4b68063230 con 0x7f4b4c0b3ca0 2024-07-26T08:29:33.779 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 6 on host 'smithi044' 2024-07-26T08:29:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.780+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b4c0b3ca0 msgr2=0x7f4b4c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.780+0000 7f4b6e600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b4c0b3ca0 0x7f4b4c0b61b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4b58031440 tx=0x7f4b58009560 comp rx=0 tx=0).stop 2024-07-26T08:29:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 msgr2=0x7f4b6819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:33.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 0x7f4b6819c4f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400d950 tx=0x7f4b5400de10 comp rx=0 tx=0).stop 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 shutdown_connections 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b4c0b3ca0 0x7f4b4c0b61b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b6810b870 0x7f4b681a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b6810af20 0x7f4b6819c4f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 --2- 172.21.15.44:0/395710700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b681055e0 0x7f4b6819bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 >> 172.21.15.44:0/395710700 conn(0x7f4b681010b0 msgr2=0x7f4b681034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 shutdown_connections 2024-07-26T08:29:33.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:29:33.781+0000 7f4b6e600640 1 -- 172.21.15.44:0/395710700 wait complete. 2024-07-26T08:29:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:33 smithi171 ceph-mon[37789]: from='osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T08:29:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:33 smithi171 ceph-mon[37789]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T08:29:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.651 DEBUG:teuthology.orchestra.run.smithi044:osd.6> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.6.service 2024-07-26T08:29:34.656 INFO:tasks.cephadm:Deploying osd.7 on smithi116 with /dev/vg_nvme/lv_2... 2024-07-26T08:29:34.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T08:29:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T08:29:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:29:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T08:29:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:29:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:34.925 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:29:35.501 INFO:journalctl@ceph.osd.6.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-6[65694]: 2024-07-26T08:29:35.240+0000 7f8309a00640 -1 osd.6 0 waiting for initial osdmap 2024-07-26T08:29:35.501 INFO:journalctl@ceph.osd.6.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-6[65694]: 2024-07-26T08:29:35.274+0000 7f82ffa00640 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:29:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-mon[32404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:29:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-mon[32404]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T08:29:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-mon[32404]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:35 smithi044 ceph-mon[32404]: from='osd.6 ' entity='osd.6' 2024-07-26T08:29:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:35 smithi116 ceph-mon[37097]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:29:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:35 smithi116 ceph-mon[37097]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T08:29:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:35 smithi116 ceph-mon[37097]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:35 smithi116 ceph-mon[37097]: from='osd.6 ' entity='osd.6' 2024-07-26T08:29:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:35 smithi171 ceph-mon[37789]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:29:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:35 smithi171 ceph-mon[37789]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T08:29:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:35 smithi171 ceph-mon[37789]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-07-26T08:29:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:35 smithi171 ceph-mon[37789]: from='osd.6 ' entity='osd.6' 2024-07-26T08:29:36.736 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:36 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:29:36.736 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:36 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:29:36.736 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:36.736 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:36.736 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:36 smithi044 ceph-mon[32404]: osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] boot 2024-07-26T08:29:36.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:36 smithi044 ceph-mon[32404]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T08:29:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:36 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:29:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:36 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:29:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:36 smithi116 ceph-mon[37097]: osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] boot 2024-07-26T08:29:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:36 smithi116 ceph-mon[37097]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T08:29:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:36 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:29:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:36 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:29:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T08:29:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:36 smithi171 ceph-mon[37789]: osd.6 [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] boot 2024-07-26T08:29:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:36 smithi171 ceph-mon[37789]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T08:29:36.849 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T08:29:36.936 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T08:29:36.949 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-07-26T08:29:36.949 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-07-26T08:29:36.949 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113891 s, 921 MB/s 2024-07-26T08:29:36.949 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-07-26T08:29:37.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-07-26T08:29:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:37 smithi044 ceph-mon[32404]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:37 smithi116 ceph-mon[37097]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:37 smithi171 ceph-mon[37789]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:29:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:38 smithi116 ceph-mon[37097]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T08:29:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:38 smithi044 ceph-mon[32404]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T08:29:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:38 smithi171 ceph-mon[37789]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 -- 172.21.15.116:0/2226122603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20107dd0 msgr2=0x7fee201081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 --2- 172.21.15.116:0/2226122603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20107dd0 0x7fee201081e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c02a410 comp rx=0 tx=0).stop 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 -- 172.21.15.116:0/2226122603 shutdown_connections 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 --2- 172.21.15.116:0/2226122603 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee20108720 0x7fee2010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 --2- 172.21.15.116:0/2226122603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20107dd0 0x7fee201081e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 --2- 172.21.15.116:0/2226122603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20102490 0x7fee201028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 -- 172.21.15.116:0/2226122603 >> 172.21.15.116:0/2226122603 conn(0x7fee20079160 msgr2=0x7fee20079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:29:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 -- 172.21.15.116:0/2226122603 shutdown_connections 2024-07-26T08:29:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.449+0000 7fee26cb7640 1 -- 172.21.15.116:0/2226122603 wait complete. 2024-07-26T08:29:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.450+0000 7fee26cb7640 1 Processor -- start 2024-07-26T08:29:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.450+0000 7fee26cb7640 1 -- start start 2024-07-26T08:29:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.450+0000 7fee26cb7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 0x7fee20198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.450+0000 7fee26cb7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20107dd0 0x7fee201993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.450+0000 7fee26cb7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee20108720 0x7fee2019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee26cb7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee201123f0 con 0x7fee20107dd0 2024-07-26T08:29:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee26cb7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee20112270 con 0x7fee20102490 2024-07-26T08:29:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee26cb7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fee20112570 con 0x7fee20108720 2024-07-26T08:29:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 0x7fee20198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 0x7fee20198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39180/0 (socket says 172.21.15.116:39180) 2024-07-26T08:29:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 -- 172.21.15.116:0/899033466 learned_addr learned my addr 172.21.15.116:0/899033466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-07-26T08:29:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee25254640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee20108720 0x7fee2019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 -- 172.21.15.116:0/899033466 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee20108720 msgr2=0x7fee2019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee20108720 0x7fee2019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 -- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20107dd0 msgr2=0x7fee201993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20107dd0 0x7fee201993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 -- 172.21.15.116:0/899033466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2019ed90 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee24a53640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 0x7fee20198e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee1400b4a0 tx=0x7fee1400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14014070 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee14010ac0 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2019f080 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.451+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14017c30 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.452+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee201a57e0 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.453+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee20103710 con 0x7fee20102490 2024-07-26T08:29:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.454+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fee14017070 con 0x7fee20102490 2024-07-26T08:29:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.454+0000 7fee1dffb640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee100b3c20 0x7fee100b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:29:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.455+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5180+0+0 (secure 0 0 0) 0x7fee140e27f0 con 0x7fee20102490 2024-07-26T08:29:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.455+0000 7fee1ffff640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee100b3c20 0x7fee100b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:29:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.456+0000 7fee1ffff640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee100b3c20 0x7fee100b6130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee0c02a930 tx=0x7fee0c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:29:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.459+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fee140aaa80 con 0x7fee20102490 2024-07-26T08:29:39.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:39.650+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fee20063230 con 0x7fee100b3c20 2024-07-26T08:29:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:39 smithi044 ceph-mon[32404]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:39 smithi171 ceph-mon[37789]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:39 smithi116 ceph-mon[37097]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:40.758 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi044 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.759 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 5698M 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='client.24470 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:29:40.760 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:29:40.761 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi044 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 5698M 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='client.24470 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:29:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi044 2024-07-26T08:29:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 5698M 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='client.24470 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:29:41.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:41.756 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:41 smithi116 ceph-mon[37097]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:41 smithi044 ceph-mon[32404]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:41 smithi171 ceph-mon[37789]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:42.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:42 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/3227989439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]: dispatch 2024-07-26T08:29:42.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:42 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]: dispatch 2024-07-26T08:29:42.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:42 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]': finished 2024-07-26T08:29:42.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:29:42.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:42 smithi116 ceph-mon[37097]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T08:29:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:42 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/3227989439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]: dispatch 2024-07-26T08:29:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:42 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]: dispatch 2024-07-26T08:29:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:42 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]': finished 2024-07-26T08:29:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:29:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:42 smithi171 ceph-mon[37789]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T08:29:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:42 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/3227989439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]: dispatch 2024-07-26T08:29:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:42 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]: dispatch 2024-07-26T08:29:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:42 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbc14308-bca8-4549-8079-6483bda3f101"}]': finished 2024-07-26T08:29:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:29:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:42 smithi044 ceph-mon[32404]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T08:29:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:43 smithi171 ceph-mon[37789]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:43 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/1187413323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:29:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:43 smithi044 ceph-mon[32404]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:43 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/1187413323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:29:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:43 smithi116 ceph-mon[37097]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:43 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/1187413323' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:29:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:45 smithi044 ceph-mon[32404]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:45 smithi116 ceph-mon[37097]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:45 smithi171 ceph-mon[37789]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:47.792 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:47 smithi116 ceph-mon[37097]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:47 smithi044 ceph-mon[32404]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:47 smithi171 ceph-mon[37789]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:49.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:49 smithi116 ceph-mon[37097]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:49.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T08:29:49.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:49.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:49 smithi116 ceph-mon[37097]: Deploying daemon osd.7 on smithi116 2024-07-26T08:29:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:49 smithi044 ceph-mon[32404]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T08:29:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:49 smithi044 ceph-mon[32404]: Deploying daemon osd.7 on smithi116 2024-07-26T08:29:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:49 smithi171 ceph-mon[37789]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T08:29:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:49 smithi171 ceph-mon[37789]: Deploying daemon osd.7 on smithi116 2024-07-26T08:29:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:51 smithi044 ceph-mon[32404]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:52.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:51 smithi116 ceph-mon[37097]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:51 smithi171 ceph-mon[37789]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:53.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:53 smithi116 ceph-mon[37097]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:53 smithi044 ceph-mon[32404]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:53 smithi171 ceph-mon[37789]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:55 smithi044 ceph-mon[32404]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:55 smithi116 ceph-mon[37097]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:55 smithi171 ceph-mon[37789]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:58.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.201+0000 7fee1dffb640 1 -- 172.21.15.116:0/899033466 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fee20063230 con 0x7fee100b3c20 2024-07-26T08:29:58.203 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 7 on host 'smithi116' 2024-07-26T08:29:58.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee100b3c20 msgr2=0x7fee100b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:58.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee100b3c20 0x7fee100b6130 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee0c02a930 tx=0x7fee0c031400 comp rx=0 tx=0).stop 2024-07-26T08:29:58.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 msgr2=0x7fee20198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:29:58.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 0x7fee20198e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fee1400b4a0 tx=0x7fee1400b960 comp rx=0 tx=0).stop 2024-07-26T08:29:58.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 shutdown_connections 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee100b3c20 0x7fee100b6130 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee20108720 0x7fee2019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee20107dd0 0x7fee201993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 --2- 172.21.15.116:0/899033466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee20102490 0x7fee20198e20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 >> 172.21.15.116:0/899033466 conn(0x7fee20079160 msgr2=0x7fee20104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 shutdown_connections 2024-07-26T08:29:58.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:29:58.204+0000 7fee26cb7640 1 -- 172.21.15.116:0/899033466 wait complete. 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:58.477 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.478 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:58.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:29:58.933 DEBUG:teuthology.orchestra.run.smithi116:osd.7> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.7.service 2024-07-26T08:29:58.936 INFO:tasks.cephadm:Deploying osd.8 on smithi171 with /dev/vg_nvme/lv_2... 2024-07-26T08:29:58.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T08:29:59.195 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:29:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:59 smithi116 ceph-mon[37097]: from='osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T08:29:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:29:59 smithi116 ceph-mon[37097]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T08:29:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:59 smithi171 ceph-mon[37789]: from='osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T08:29:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:29:59 smithi171 ceph-mon[37789]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T08:29:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:59 smithi044 ceph-mon[32404]: from='osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T08:29:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:29:59 smithi044 ceph-mon[32404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: from='osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:30:00.690 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:00 smithi171 ceph-mon[37789]: overall HEALTH_OK 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: from='osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:00 smithi044 ceph-mon[32404]: overall HEALTH_OK 2024-07-26T08:30:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:30:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T08:30:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: from='osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:30:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T08:30:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:30:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-mon[37097]: overall HEALTH_OK 2024-07-26T08:30:00.773 INFO:journalctl@ceph.osd.7.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-7[56337]: 2024-07-26T08:30:00.395+0000 7f3c9ce45640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T08:30:00.773 INFO:journalctl@ceph.osd.7.smithi116.stdout:Jul 26 08:30:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-7[56337]: 2024-07-26T08:30:00.405+0000 7f3c98c43640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:30:01.017 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T08:30:01.150 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T08:30:01.163 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-07-26T08:30:01.163 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-07-26T08:30:01.163 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114218 s, 918 MB/s 2024-07-26T08:30:01.164 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-07-26T08:30:01.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:01 smithi171 ceph-mon[37789]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:30:01.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:01 smithi171 ceph-mon[37789]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T08:30:01.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:01.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:01.599 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:01 smithi116 ceph-mon[37097]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:30:01.599 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:01 smithi116 ceph-mon[37097]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T08:30:01.599 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:01.599 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:01 smithi044 ceph-mon[32404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:30:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:01 smithi044 ceph-mon[32404]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T08:30:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:01.862 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_2 2024-07-26T08:30:02.121 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:30:02.484 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:30:02.484 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] boot 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:02.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:02.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:30:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:30:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] boot 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:02.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:02.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: osd.7 [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] boot 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T08:30:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:03 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi116 2024-07-26T08:30:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:03 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 5696M 2024-07-26T08:30:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:03 smithi171 ceph-mon[37789]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T08:30:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:03 smithi171 ceph-mon[37789]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T08:30:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:03 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi116 2024-07-26T08:30:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:03 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 5696M 2024-07-26T08:30:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:03 smithi044 ceph-mon[32404]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T08:30:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:03 smithi044 ceph-mon[32404]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T08:30:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:03 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi116 2024-07-26T08:30:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:03 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 5696M 2024-07-26T08:30:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:03 smithi116 ceph-mon[37097]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T08:30:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:03 smithi116 ceph-mon[37097]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T08:30:03.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.791+0000 7f62b749d640 1 -- 172.21.15.171:0/1245774626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b0108f90 msgr2=0x7f62b01093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:03.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.791+0000 7f62b749d640 1 --2- 172.21.15.171:0/1245774626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b0108f90 0x7f62b01093a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009900 tx=0x7f62ac02a410 comp rx=0 tx=0).stop 2024-07-26T08:30:03.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.791+0000 7f62b749d640 1 -- 172.21.15.171:0/1245774626 shutdown_connections 2024-07-26T08:30:03.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.791+0000 7f62b749d640 1 --2- 172.21.15.171:0/1245774626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b0108f90 0x7f62b01093a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.791+0000 7f62b749d640 1 --2- 172.21.15.171:0/1245774626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62b01098e0 0x7f62b0110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.791+0000 7f62b749d640 1 --2- 172.21.15.171:0/1245774626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62b0101ab0 0x7f62b0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.792+0000 7f62b749d640 1 -- 172.21.15.171:0/1245774626 >> 172.21.15.171:0/1245774626 conn(0x7f62b00798f0 msgr2=0x7f62b0079d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.792+0000 7f62b749d640 1 -- 172.21.15.171:0/1245774626 shutdown_connections 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.792+0000 7f62b749d640 1 -- 172.21.15.171:0/1245774626 wait complete. 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.792+0000 7f62b749d640 1 Processor -- start 2024-07-26T08:30:03.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.792+0000 7f62b749d640 1 -- start start 2024-07-26T08:30:03.795 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62b0101ab0 0x7f62b0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62b0108f90 0x7f62b019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 0x7f62b019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62b0113640 con 0x7f62b0101ab0 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01134c0 con 0x7f62b0108f90 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f62b01137c0 con 0x7f62b01098e0 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 0x7f62b019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 0x7f62b019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:39880/0 (socket says 172.21.15.171:39880) 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 -- 172.21.15.171:0/4008630095 learned_addr learned my addr 172.21.15.171:0/4008630095 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 -- 172.21.15.171:0/4008630095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62b0108f90 msgr2=0x7f62b019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:03.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62b0108f90 0x7f62b019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 -- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62b0101ab0 msgr2=0x7f62b0199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62b0101ab0 0x7f62b0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 -- 172.21.15.171:0/4008630095 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b019ff60 con 0x7f62b01098e0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b5a3a640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 0x7f62b019f830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f62a800b970 tx=0x7f62a800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8019070 con 0x7f62b01098e0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b01a0250 con 0x7f62b01098e0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62a8014440 con 0x7f62b01098e0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b01a6a30 con 0x7f62b01098e0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.793+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a80075d0 con 0x7f62b01098e0 2024-07-26T08:30:03.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.794+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b0101ec0 con 0x7f62b01098e0 2024-07-26T08:30:03.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.796+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f62a801c070 con 0x7f62b01098e0 2024-07-26T08:30:03.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.796+0000 7f62a67fc640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f629c0b3c20 0x7f629c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:03.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.797+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5612+0+0 (secure 0 0 0) 0x7f62a80e2c80 con 0x7f62b01098e0 2024-07-26T08:30:03.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.797+0000 7f62b5239640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f629c0b3c20 0x7f629c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:03.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.798+0000 7f62b5239640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f629c0b3c20 0x7f629c0b6130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009710 tx=0x7f62a0006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:30:03.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.800+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f62a80aad60 con 0x7f62b01098e0 2024-07-26T08:30:03.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:03.980+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f62b0063230 con 0x7f629c0b3c20 2024-07-26T08:30:04.677 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:04 smithi171 ceph-mon[37789]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:04.677 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:04 smithi171 ceph-mon[37789]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-07-26T08:30:04.677 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:30:04.677 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:30:04.677 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:04.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:04 smithi116 ceph-mon[37097]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:04.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:04 smithi116 ceph-mon[37097]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-07-26T08:30:04.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:30:04.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:30:04.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:04 smithi044 ceph-mon[32404]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:04 smithi044 ceph-mon[32404]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-07-26T08:30:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:30:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:30:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:05 smithi044 ceph-mon[32404]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:05 smithi171 ceph-mon[37789]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:30:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:05 smithi116 ceph-mon[37097]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:30:06.736 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:06 smithi171 ceph-mon[37789]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:06 smithi044 ceph-mon[32404]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:06 smithi116 ceph-mon[37097]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:07 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/1528207631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]: dispatch 2024-07-26T08:30:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:07 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]: dispatch 2024-07-26T08:30:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:07 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]': finished 2024-07-26T08:30:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:07 smithi044 ceph-mon[32404]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T08:30:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:07 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/3298090922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:30:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:07 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/1528207631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]: dispatch 2024-07-26T08:30:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:07 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]: dispatch 2024-07-26T08:30:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:07 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]': finished 2024-07-26T08:30:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:07 smithi116 ceph-mon[37097]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T08:30:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:07 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/3298090922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:30:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:07 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/1528207631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]: dispatch 2024-07-26T08:30:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:07 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]: dispatch 2024-07-26T08:30:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:07 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20dacf62-1c49-4dd9-8d1f-467506b1d1bd"}]': finished 2024-07-26T08:30:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:07 smithi171 ceph-mon[37789]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T08:30:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:07 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/3298090922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:30:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:08 smithi044 ceph-mon[32404]: pgmap v187: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:08 smithi116 ceph-mon[37097]: pgmap v187: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:08 smithi171 ceph-mon[37789]: pgmap v187: 1 pgs: 1 unknown; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:09.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:09 smithi044 ceph-mon[32404]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s, 0 objects/s recovering 2024-07-26T08:30:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:09 smithi044 ceph-mon[32404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-07-26T08:30:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:09 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:30:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:09 smithi116 ceph-mon[37097]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s, 0 objects/s recovering 2024-07-26T08:30:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:09 smithi116 ceph-mon[37097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-07-26T08:30:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:09 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:30:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:09 smithi171 ceph-mon[37789]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s, 0 objects/s recovering 2024-07-26T08:30:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:09 smithi171 ceph-mon[37789]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-07-26T08:30:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:09 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:30:11.984 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:11 smithi171 ceph-mon[37789]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-07-26T08:30:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:11 smithi044 ceph-mon[32404]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-07-26T08:30:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:11 smithi116 ceph-mon[37097]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-07-26T08:30:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:13 smithi171 ceph-mon[37789]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-07-26T08:30:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T08:30:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:13 smithi044 ceph-mon[32404]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-07-26T08:30:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T08:30:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:13 smithi116 ceph-mon[37097]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-07-26T08:30:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T08:30:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:15.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:14 smithi171 ceph-mon[37789]: Deploying daemon osd.8 on smithi171 2024-07-26T08:30:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:14 smithi044 ceph-mon[32404]: Deploying daemon osd.8 on smithi171 2024-07-26T08:30:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:14 smithi116 ceph-mon[37097]: Deploying daemon osd.8 on smithi171 2024-07-26T08:30:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:15 smithi171 ceph-mon[37789]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-07-26T08:30:16.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:15 smithi044 ceph-mon[32404]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-07-26T08:30:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:15 smithi116 ceph-mon[37097]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-07-26T08:30:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:17 smithi044 ceph-mon[32404]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-07-26T08:30:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:17 smithi116 ceph-mon[37097]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-07-26T08:30:18.033 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:17 smithi171 ceph-mon[37789]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-07-26T08:30:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:19 smithi044 ceph-mon[32404]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T08:30:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:19 smithi116 ceph-mon[37097]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T08:30:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:19 smithi171 ceph-mon[37789]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T08:30:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:21 smithi116 ceph-mon[37097]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:21 smithi171 ceph-mon[37789]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:21 smithi044 ceph-mon[32404]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:22.547 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.544+0000 7f62a67fc640 1 -- 172.21.15.171:0/4008630095 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f62b0063230 con 0x7f629c0b3c20 2024-07-26T08:30:22.547 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 8 on host 'smithi171' 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f629c0b3c20 msgr2=0x7f629c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f629c0b3c20 0x7f629c0b6130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009710 tx=0x7f62a0006c90 comp rx=0 tx=0).stop 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 msgr2=0x7f62b019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 0x7f62b019f830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f62a800b970 tx=0x7f62a800be30 comp rx=0 tx=0).stop 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 shutdown_connections 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f629c0b3c20 0x7f629c0b6130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62b01098e0 0x7f62b019f830 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62b0108f90 0x7f62b019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 --2- 172.21.15.171:0/4008630095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62b0101ab0 0x7f62b0199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 >> 172.21.15.171:0/4008630095 conn(0x7f62b00798f0 msgr2=0x7f62b0106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:30:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 shutdown_connections 2024-07-26T08:30:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:30:22.547+0000 7f62b749d640 1 -- 172.21.15.171:0/4008630095 wait complete. 2024-07-26T08:30:23.149 DEBUG:teuthology.orchestra.run.smithi171:osd.8> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.8.service 2024-07-26T08:30:23.151 INFO:tasks.cephadm:Deploying osd.9 on smithi044 with /dev/vg_nvme/lv_1... 2024-07-26T08:30:23.151 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T08:30:23.635 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: from='osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: from='osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:23.785 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: from='osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T08:30:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T08:30:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-mon[37789]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T08:30:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-mon[37789]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T08:30:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-mon[37789]: from='osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:30:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-mon[37789]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:30:24.837 INFO:journalctl@ceph.osd.8.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-8[57889]: 2024-07-26T08:30:24.557+0000 7f69d42c6640 -1 osd.8 0 waiting for initial osdmap 2024-07-26T08:30:24.837 INFO:journalctl@ceph.osd.8.smithi171.stdout:Jul 26 08:30:24 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-8[57889]: 2024-07-26T08:30:24.572+0000 7f69d00c4640 -1 osd.8 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:30:24.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:24 smithi044 ceph-mon[32404]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T08:30:24.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:24.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:24 smithi044 ceph-mon[32404]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T08:30:24.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:24 smithi044 ceph-mon[32404]: from='osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:30:24.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:24 smithi044 ceph-mon[32404]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:30:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:24 smithi116 ceph-mon[37097]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T08:30:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:24 smithi116 ceph-mon[37097]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T08:30:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:24 smithi116 ceph-mon[37097]: from='osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:30:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:24 smithi116 ceph-mon[37097]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] boot 2024-07-26T08:30:25.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:25 smithi044 ceph-mon[32404]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] boot 2024-07-26T08:30:25.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:25 smithi171 ceph-mon[37789]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T08:30:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:30:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T08:30:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-07-26T08:30:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T08:30:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: osd.8 [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] boot 2024-07-26T08:30:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:25 smithi116 ceph-mon[37097]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T08:30:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T08:30:26.760 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi171 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 5696M 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:26.800 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-07-26T08:30:26.800 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-07-26T08:30:26.801 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0332935 s, 315 MB/s 2024-07-26T08:30:26.801 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-07-26T08:30:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:30:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:30:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi171 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 5696M 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:27.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:27.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:27.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:27.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:27.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi171 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 5696M 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:27.836 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_1 2024-07-26T08:30:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:27 smithi044 ceph-mon[32404]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:27 smithi044 ceph-mon[32404]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T08:30:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:27 smithi116 ceph-mon[37097]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:27 smithi116 ceph-mon[37097]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T08:30:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:27 smithi171 ceph-mon[37789]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:27 smithi171 ceph-mon[37789]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T08:30:28.289 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:30:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:29 smithi044 ceph-mon[32404]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-07-26T08:30:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:29 smithi116 ceph-mon[37097]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-07-26T08:30:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:29 smithi171 ceph-mon[37789]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-07-26T08:30:30.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 -- 172.21.15.44:0/109389195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc801057e0 msgr2=0x7fdc80105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 --2- 172.21.15.44:0/109389195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc801057e0 0x7fdc80105bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009900 tx=0x7fdc7002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 -- 172.21.15.44:0/109389195 shutdown_connections 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 --2- 172.21.15.44:0/109389195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8010ba70 0x7fdc801122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 --2- 172.21.15.44:0/109389195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc8010b120 0x7fdc8010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 --2- 172.21.15.44:0/109389195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc801057e0 0x7fdc80105bf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 -- 172.21.15.44:0/109389195 >> 172.21.15.44:0/109389195 conn(0x7fdc801011f0 msgr2=0x7fdc80103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 -- 172.21.15.44:0/109389195 shutdown_connections 2024-07-26T08:30:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.895+0000 7fdc85a00640 1 -- 172.21.15.44:0/109389195 wait complete. 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.896+0000 7fdc85a00640 1 Processor -- start 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.896+0000 7fdc85a00640 1 -- start start 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc85a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 0x7fdc8019c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc85a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc8010b120 0x7fdc8019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc85a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc8010ba70 0x7fdc801a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc85a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc80115740 con 0x7fdc8010b120 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc85a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801155c0 con 0x7fdc801057e0 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc85a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdc801158c0 con 0x7fdc8010ba70 2024-07-26T08:30:30.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc7ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 0x7fdc8019c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc7e000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc8010b120 0x7fdc8019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc7ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 0x7fdc8019c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:42138/0 (socket says 172.21.15.44:42138) 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc7ea00640 1 -- 172.21.15.44:0/856842520 learned_addr learned my addr 172.21.15.44:0/856842520 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.897+0000 7fdc7f400640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc8010ba70 0x7fdc801a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc7ea00640 1 -- 172.21.15.44:0/856842520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc8010ba70 msgr2=0x7fdc801a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc7ea00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc8010ba70 0x7fdc801a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc7ea00640 1 -- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc8010b120 msgr2=0x7fdc8019c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc7ea00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc8010b120 0x7fdc8019c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:30.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc7ea00640 1 -- 172.21.15.44:0/856842520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc801a20a0 con 0x7fdc801057e0 2024-07-26T08:30:30.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc7ea00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 0x7fdc8019c130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc70005940 tx=0x7fdc700040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:30:30.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7003f030 con 0x7fdc801057e0 2024-07-26T08:30:30.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc700044c0 con 0x7fdc801057e0 2024-07-26T08:30:30.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70035070 con 0x7fdc801057e0 2024-07-26T08:30:30.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.898+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc801a2330 con 0x7fdc801057e0 2024-07-26T08:30:30.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.899+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc801a8af0 con 0x7fdc801057e0 2024-07-26T08:30:30.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.900+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc80105bf0 con 0x7fdc801057e0 2024-07-26T08:30:30.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.902+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fdc70042030 con 0x7fdc801057e0 2024-07-26T08:30:30.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.904+0000 7fdc77400640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdc640b3ca0 0x7fdc640b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:30:30.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.904+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6044+0+0 (secure 0 0 0) 0x7fdc70103940 con 0x7fdc801057e0 2024-07-26T08:30:30.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.904+0000 7fdc7e000640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdc640b3ca0 0x7fdc640b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:30:30.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.905+0000 7fdc7e000640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdc640b3ca0 0x7fdc640b61b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdc80104ae0 tx=0x7fdc6c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:30:30.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:30.905+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdc700cb180 con 0x7fdc801057e0 2024-07-26T08:30:31.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:31.121+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdc80063230 con 0x7fdc640b3ca0 2024-07-26T08:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:31 smithi044 ceph-mon[32404]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T08:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:31 smithi044 ceph-mon[32404]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:31 smithi116 ceph-mon[37097]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T08:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:31 smithi116 ceph-mon[37097]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:31 smithi171 ceph-mon[37789]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T08:30:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:31 smithi171 ceph-mon[37789]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:30:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:30:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:30:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:33 smithi044 ceph-mon[32404]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:30:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:33 smithi116 ceph-mon[37097]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:30:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:33 smithi171 ceph-mon[37789]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:30:35.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:35 smithi044 ceph-mon[32404]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-07-26T08:30:35.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:35 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3003370540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]: dispatch 2024-07-26T08:30:35.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:35 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]: dispatch 2024-07-26T08:30:35.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:35 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]': finished 2024-07-26T08:30:35.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:30:35.915 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:35 smithi044 ceph-mon[32404]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T08:30:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:35 smithi116 ceph-mon[37097]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-07-26T08:30:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:35 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3003370540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]: dispatch 2024-07-26T08:30:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:35 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]: dispatch 2024-07-26T08:30:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:35 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]': finished 2024-07-26T08:30:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:30:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:35 smithi116 ceph-mon[37097]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T08:30:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:35 smithi171 ceph-mon[37789]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-07-26T08:30:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:35 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3003370540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]: dispatch 2024-07-26T08:30:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:35 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]: dispatch 2024-07-26T08:30:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:35 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa16cab-5114-438f-bb1e-e636366e290d"}]': finished 2024-07-26T08:30:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:30:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:35 smithi171 ceph-mon[37789]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T08:30:36.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:36 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1980007795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:30:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:36 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1980007795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:30:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:36 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1980007795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:30:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:37 smithi044 ceph-mon[32404]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:30:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:37 smithi116 ceph-mon[37097]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:30:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:37 smithi171 ceph-mon[37789]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:30:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:39 smithi044 ceph-mon[32404]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:39 smithi116 ceph-mon[37097]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:39 smithi171 ceph-mon[37789]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:41 smithi044 ceph-mon[32404]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:41 smithi116 ceph-mon[37097]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:41 smithi171 ceph-mon[37789]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:43 smithi044 ceph-mon[32404]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:43 smithi116 ceph-mon[37097]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:43 smithi171 ceph-mon[37789]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T08:30:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:44 smithi116 ceph-mon[37097]: Deploying daemon osd.9 on smithi044 2024-07-26T08:30:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T08:30:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:44 smithi171 ceph-mon[37789]: Deploying daemon osd.9 on smithi044 2024-07-26T08:30:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T08:30:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:44 smithi044 ceph-mon[32404]: Deploying daemon osd.9 on smithi044 2024-07-26T08:30:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:45 smithi044 ceph-mon[32404]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:45 smithi116 ceph-mon[37097]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:45 smithi171 ceph-mon[37789]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:47 smithi044 ceph-mon[32404]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:47 smithi171 ceph-mon[37789]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:47 smithi116 ceph-mon[37097]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:49.916 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:49 smithi044 ceph-mon[32404]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:49 smithi171 ceph-mon[37789]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:49 smithi116 ceph-mon[37097]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:51 smithi044 ceph-mon[32404]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:51 smithi171 ceph-mon[37789]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:51 smithi116 ceph-mon[37097]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:53 smithi171 ceph-mon[37789]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:54.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:53 smithi044 ceph-mon[32404]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:54.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:54.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:54.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:53 smithi116 ceph-mon[37097]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:55 smithi044 ceph-mon[32404]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:55 smithi171 ceph-mon[37789]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:55 smithi116 ceph-mon[37097]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T08:30:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:30:58.611 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:58.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T08:30:58.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:58.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:58.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:59.178 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 9 on host 'smithi044' 2024-07-26T08:30:59.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.176+0000 7fdc77400640 1 -- 172.21.15.44:0/856842520 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdc80063230 con 0x7fdc640b3ca0 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdc640b3ca0 msgr2=0x7fdc640b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdc640b3ca0 0x7fdc640b61b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdc80104ae0 tx=0x7fdc6c005f90 comp rx=0 tx=0).stop 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 msgr2=0x7fdc8019c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 0x7fdc8019c130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc70005940 tx=0x7fdc700040c0 comp rx=0 tx=0).stop 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 shutdown_connections 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdc640b3ca0 0x7fdc640b61b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc8010ba70 0x7fdc801a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc8010b120 0x7fdc8019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:59.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.180+0000 7fdc85a00640 1 --2- 172.21.15.44:0/856842520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc801057e0 0x7fdc8019c130 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:30:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.181+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 >> 172.21.15.44:0/856842520 conn(0x7fdc801011f0 msgr2=0x7fdc80103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:30:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.181+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 shutdown_connections 2024-07-26T08:30:59.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:30:59.181+0000 7fdc85a00640 1 -- 172.21.15.44:0/856842520 wait complete. 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:30:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T08:30:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:30:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T08:30:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:30:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:30:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T08:30:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:30:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T08:30:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-07-26T08:30:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:30:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:30:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:00.133 DEBUG:teuthology.orchestra.run.smithi044:osd.9> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.9.service 2024-07-26T08:31:00.142 INFO:tasks.cephadm:Deploying osd.10 on smithi116 with /dev/vg_nvme/lv_1... 2024-07-26T08:31:00.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T08:31:00.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:00 smithi044 ceph-mon[32404]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:31:00.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:00 smithi044 ceph-mon[32404]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:31:00.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:00 smithi044 ceph-mon[32404]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T08:31:00.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:00.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:31:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:00 smithi116 ceph-mon[37097]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:31:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:00 smithi116 ceph-mon[37097]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:31:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:00 smithi116 ceph-mon[37097]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T08:31:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:00 smithi171 ceph-mon[37789]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:31:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:00 smithi171 ceph-mon[37789]: from='osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-07-26T08:31:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:00 smithi171 ceph-mon[37789]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T08:31:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:01 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:31:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:01 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:31:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:01 smithi044 ceph-mon[32404]: osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] boot 2024-07-26T08:31:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:01 smithi044 ceph-mon[32404]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T08:31:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:01 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:31:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:01 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:31:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:01 smithi171 ceph-mon[37789]: osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] boot 2024-07-26T08:31:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:01 smithi171 ceph-mon[37789]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T08:31:01.604 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:01 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:31:01.604 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:01 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:31:01.604 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T08:31:01.604 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:01 smithi116 ceph-mon[37097]: osd.9 [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] boot 2024-07-26T08:31:01.605 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:01 smithi116 ceph-mon[37097]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T08:31:02.296 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T08:31:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T08:31:02.405 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-07-26T08:31:02.405 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-07-26T08:31:02.405 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115883 s, 905 MB/s 2024-07-26T08:31:02.406 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-07-26T08:31:02.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:02 smithi116 ceph-mon[37097]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:31:02.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:02 smithi116 ceph-mon[37097]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T08:31:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:02 smithi171 ceph-mon[37789]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:31:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:02 smithi171 ceph-mon[37789]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T08:31:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:02 smithi044 ceph-mon[32404]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-07-26T08:31:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:02 smithi044 ceph-mon[32404]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T08:31:03.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-07-26T08:31:03.385 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T08:31:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:04 smithi044 ceph-mon[32404]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:04 smithi171 ceph-mon[37789]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:04.645 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:04 smithi116 ceph-mon[37097]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:05.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 -- 172.21.15.116:0/3673795801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c108720 msgr2=0x7fc16c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:05.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 --2- 172.21.15.116:0/3673795801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c108720 0x7fc16c10ef50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc164009ca0 tx=0x7fc16402a8c0 comp rx=0 tx=0).stop 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 -- 172.21.15.116:0/3673795801 shutdown_connections 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 --2- 172.21.15.116:0/3673795801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c108720 0x7fc16c10ef50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 --2- 172.21.15.116:0/3673795801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc16c107dd0 0x7fc16c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 --2- 172.21.15.116:0/3673795801 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc16c102490 0x7fc16c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 -- 172.21.15.116:0/3673795801 >> 172.21.15.116:0/3673795801 conn(0x7fc16c079160 msgr2=0x7fc16c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 -- 172.21.15.116:0/3673795801 shutdown_connections 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.068+0000 7fc1719a0640 1 -- 172.21.15.116:0/3673795801 wait complete. 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 Processor -- start 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 -- start start 2024-07-26T08:31:05.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc16c102490 0x7fc16c199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 0x7fc16c19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc16c108720 0x7fc16c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c112410 con 0x7fc16c102490 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c112290 con 0x7fc16c107dd0 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc1719a0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c112590 con 0x7fc16c108720 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.069+0000 7fc163fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 0x7fc16c19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 0x7fc16c19a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59702/0 (socket says 172.21.15.116:59702) 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 -- 172.21.15.116:0/1822188404 learned_addr learned my addr 172.21.15.116:0/1822188404 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc16b7fe640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc16c108720 0x7fc16c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 -- 172.21.15.116:0/1822188404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc16c108720 msgr2=0x7fc16c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc16c108720 0x7fc16c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 -- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc16c102490 msgr2=0x7fc16c199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc16c102490 0x7fc16c199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 -- 172.21.15.116:0/1822188404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c19fe20 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc163fff640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 0x7fc16c19a450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc15400b450 tx=0x7fc15400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154015020 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16c1a0110 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc154011d40 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15400f340 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.070+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16c1a6870 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.071+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16c1028a0 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.073+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fc154018020 con 0x7fc16c107dd0 2024-07-26T08:31:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.073+0000 7fc17099e640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc1580b3c20 0x7fc1580b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.073+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6476+0+0 (secure 0 0 0) 0x7fc1540e3650 con 0x7fc16c107dd0 2024-07-26T08:31:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.074+0000 7fc16affd640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc1580b3c20 0x7fc1580b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.075+0000 7fc16affd640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc1580b3c20 0x7fc1580b6130 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0058c0 tx=0x7fc15c005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.077+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc15401f020 con 0x7fc16c107dd0 2024-07-26T08:31:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:05.260+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc16c063230 con 0x7fc1580b3c20 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi044 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 4273M 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='client.24536 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:31:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:31:06.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:31:06.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi044 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 4273M 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='client.24536 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:31:06.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi044 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 4273M 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:06.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='client.24536 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:31:06.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:31:06.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:06.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:31:06.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:08.369 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:08 smithi116 ceph-mon[37097]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:08.369 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:08 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/2516667737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]: dispatch 2024-07-26T08:31:08.369 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:08 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]: dispatch 2024-07-26T08:31:08.369 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:08 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]': finished 2024-07-26T08:31:08.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:08.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:08 smithi116 ceph-mon[37097]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T08:31:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:08 smithi044 ceph-mon[32404]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:08 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/2516667737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]: dispatch 2024-07-26T08:31:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:08 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]: dispatch 2024-07-26T08:31:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:08 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]': finished 2024-07-26T08:31:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:08 smithi044 ceph-mon[32404]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T08:31:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:08 smithi171 ceph-mon[37789]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:08 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/2516667737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]: dispatch 2024-07-26T08:31:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:08 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]: dispatch 2024-07-26T08:31:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:08 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cec6245-933e-4d2f-8d4f-5107276fe094"}]': finished 2024-07-26T08:31:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:08 smithi171 ceph-mon[37789]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T08:31:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:09 smithi044 ceph-mon[32404]: from='client.? 172.21.15.116:0/94562593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:31:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:09 smithi116 ceph-mon[37097]: from='client.? 172.21.15.116:0/94562593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:31:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:09 smithi171 ceph-mon[37789]: from='client.? 172.21.15.116:0/94562593' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:31:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:10 smithi044 ceph-mon[32404]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:10 smithi116 ceph-mon[37097]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:10 smithi171 ceph-mon[37789]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:12 smithi044 ceph-mon[32404]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:12.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:12 smithi116 ceph-mon[37097]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:12 smithi171 ceph-mon[37789]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:14 smithi044 ceph-mon[32404]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:14 smithi116 ceph-mon[37097]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:14 smithi171 ceph-mon[37789]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:15.505 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T08:31:15.505 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T08:31:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T08:31:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:16.368 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:16 smithi116 ceph-mon[37097]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:16.368 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:16 smithi116 ceph-mon[37097]: Deploying daemon osd.10 on smithi116 2024-07-26T08:31:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:16 smithi044 ceph-mon[32404]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:16 smithi044 ceph-mon[32404]: Deploying daemon osd.10 on smithi116 2024-07-26T08:31:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:16 smithi171 ceph-mon[37789]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:16 smithi171 ceph-mon[37789]: Deploying daemon osd.10 on smithi116 2024-07-26T08:31:18.509 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:18 smithi116 ceph-mon[37097]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:18 smithi044 ceph-mon[32404]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:18 smithi171 ceph-mon[37789]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:20 smithi044 ceph-mon[32404]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:20 smithi116 ceph-mon[37097]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:20 smithi171 ceph-mon[37789]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:22 smithi116 ceph-mon[37097]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:22 smithi044 ceph-mon[32404]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:22 smithi171 ceph-mon[37789]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:23.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.812+0000 7fc17099e640 1 -- 172.21.15.116:0/1822188404 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc16c063230 con 0x7fc1580b3c20 2024-07-26T08:31:23.814 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 10 on host 'smithi116' 2024-07-26T08:31:23.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc1580b3c20 msgr2=0x7fc1580b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc1580b3c20 0x7fc1580b6130 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0058c0 tx=0x7fc15c005850 comp rx=0 tx=0).stop 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 msgr2=0x7fc16c19a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 0x7fc16c19a450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc15400b450 tx=0x7fc15400b910 comp rx=0 tx=0).stop 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 shutdown_connections 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc1580b3c20 0x7fc1580b6130 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc16c108720 0x7fc16c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc16c107dd0 0x7fc16c19a450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 --2- 172.21.15.116:0/1822188404 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc16c102490 0x7fc16c199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 >> 172.21.15.116:0/1822188404 conn(0x7fc16c079160 msgr2=0x7fc16c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 shutdown_connections 2024-07-26T08:31:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-07-26T08:31:23.814+0000 7fc1719a0640 1 -- 172.21.15.116:0/1822188404 wait complete. 2024-07-26T08:31:24.477 DEBUG:teuthology.orchestra.run.smithi116:osd.10> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.10.service 2024-07-26T08:31:24.479 INFO:tasks.cephadm:Deploying osd.11 on smithi171 with /dev/vg_nvme/lv_1... 2024-07-26T08:31:24.479 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T08:31:24.744 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:31:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:24 smithi044 ceph-mon[32404]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:24 smithi044 ceph-mon[32404]: from='osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T08:31:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:24 smithi044 ceph-mon[32404]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T08:31:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:24 smithi116 ceph-mon[37097]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:24 smithi116 ceph-mon[37097]: from='osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T08:31:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:24 smithi116 ceph-mon[37097]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T08:31:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:24 smithi171 ceph-mon[37789]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:24 smithi171 ceph-mon[37789]: from='osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T08:31:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:24 smithi171 ceph-mon[37789]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: from='osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T08:31:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-mon[37097]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: from='osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:31:25.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T08:31:25.877 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:25 smithi171 ceph-mon[37789]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: from='osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T08:31:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:25 smithi044 ceph-mon[32404]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-07-26T08:31:26.081 INFO:journalctl@ceph.osd.10.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-10[62904]: 2024-07-26T08:31:25.817+0000 7f3db9070640 -1 osd.10 0 waiting for initial osdmap 2024-07-26T08:31:26.081 INFO:journalctl@ceph.osd.10.smithi116.stdout:Jul 26 08:31:25 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-10[62904]: 2024-07-26T08:31:25.834+0000 7f3db4e6e640 -1 osd.10 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:31:26.528 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T08:31:26.632 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T08:31:26.647 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-07-26T08:31:26.647 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-07-26T08:31:26.647 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133194 s, 787 MB/s 2024-07-26T08:31:26.647 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-07-26T08:31:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:26 smithi171 ceph-mon[37789]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:31:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:26 smithi171 ceph-mon[37789]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T08:31:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:27.106 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:26 smithi116 ceph-mon[37097]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:31:27.107 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:27.107 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:26 smithi116 ceph-mon[37097]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T08:31:27.107 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:26 smithi044 ceph-mon[32404]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-07-26T08:31:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:26 smithi044 ceph-mon[32404]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T08:31:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:27.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_1 2024-07-26T08:31:27.539 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] boot 2024-07-26T08:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:27 smithi171 ceph-mon[37789]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T08:31:28.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:31:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:31:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] boot 2024-07-26T08:31:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:27 smithi044 ceph-mon[32404]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T08:31:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:31:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:31:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-07-26T08:31:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T08:31:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: osd.10 [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] boot 2024-07-26T08:31:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:27 smithi116 ceph-mon[37097]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi116 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 4272M 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:28.931 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 -- 172.21.15.171:0/742271970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa541024a0 msgr2=0x7ffa541028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:29.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 --2- 172.21.15.171:0/742271970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa541024a0 0x7ffa541028b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffa44009900 tx=0x7ffa4402a410 comp rx=0 tx=0).stop 2024-07-26T08:31:29.164 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 -- 172.21.15.171:0/742271970 shutdown_connections 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 --2- 172.21.15.171:0/742271970 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa54108730 0x7ffa5410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 --2- 172.21.15.171:0/742271970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa54107de0 0x7ffa541081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 --2- 172.21.15.171:0/742271970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa541024a0 0x7ffa541028b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 -- 172.21.15.171:0/742271970 >> 172.21.15.171:0/742271970 conn(0x7ffa54079140 msgr2=0x7ffa54079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 -- 172.21.15.171:0/742271970 shutdown_connections 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.154+0000 7ffa58a15640 1 -- 172.21.15.171:0/742271970 wait complete. 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 Processor -- start 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 -- start start 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa541024a0 0x7ffa54198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa54107de0 0x7ffa54199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:29.165 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 0x7ffa5419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54112470 con 0x7ffa54107de0 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffa541122f0 con 0x7ffa541024a0 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.155+0000 7ffa58a15640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa541125f0 con 0x7ffa54108730 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 0x7ffa5419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 0x7ffa5419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:41778/0 (socket says 172.21.15.171:41778) 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 -- 172.21.15.171:0/1008710718 learned_addr learned my addr 172.21.15.171:0/1008710718 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa5259c640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa541024a0 0x7ffa54198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa49d9b640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa54107de0 0x7ffa54199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 -- 172.21.15.171:0/1008710718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa541024a0 msgr2=0x7ffa54198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa541024a0 0x7ffa54198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:29.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 -- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa54107de0 msgr2=0x7ffa54199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa54107de0 0x7ffa54199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 -- 172.21.15.171:0/1008710718 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5419edf0 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa52d9d640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 0x7ffa5419e6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00b970 tx=0x7ffa4c00be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c019070 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa4c014440 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5419f0e0 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0075d0 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.156+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa541a5840 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.157+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa54103720 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.159+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7ffa4c01c070 con 0x7ffa54108730 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.159+0000 7ffa4bfff640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffa400b3c20 0x7ffa400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:29.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.160+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6909+0+0 (secure 0 0 0) 0x7ffa4c0e31d0 con 0x7ffa54108730 2024-07-26T08:31:29.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.160+0000 7ffa5259c640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffa400b3c20 0x7ffa400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:29.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.161+0000 7ffa5259c640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffa400b3c20 0x7ffa400b6130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffa44031440 tx=0x7ffa44005d50 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:29.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.163+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ffa4c0aada0 con 0x7ffa54108730 2024-07-26T08:31:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T08:31:29.312 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi116 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.313 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 4272M 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:29.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:29.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:29.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:29.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi116 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.316 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 4272M 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:29.317 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:29.347 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:29.345+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ffa54063230 con 0x7ffa400b3c20 2024-07-26T08:31:30.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:29 smithi171 ceph-mon[37789]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:30.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:29 smithi171 ceph-mon[37789]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T08:31:30.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:29 smithi171 ceph-mon[37789]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:31:30.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:31:30.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:31:30.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:29 smithi044 ceph-mon[32404]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:29 smithi044 ceph-mon[32404]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T08:31:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:29 smithi044 ceph-mon[32404]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:31:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:31:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:31:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:29 smithi116 ceph-mon[37097]: pgmap v243: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:29 smithi116 ceph-mon[37097]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T08:31:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:29 smithi116 ceph-mon[37097]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:31:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T08:31:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T08:31:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:31 smithi044 ceph-mon[32404]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:31 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/3816554120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]: dispatch 2024-07-26T08:31:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:31 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]: dispatch 2024-07-26T08:31:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:31 smithi044 ceph-mon[32404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]': finished 2024-07-26T08:31:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:31 smithi044 ceph-mon[32404]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T08:31:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:31 smithi116 ceph-mon[37097]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:31 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/3816554120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]: dispatch 2024-07-26T08:31:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:31 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]: dispatch 2024-07-26T08:31:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:31 smithi116 ceph-mon[37097]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]': finished 2024-07-26T08:31:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:31 smithi116 ceph-mon[37097]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T08:31:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:31 smithi171 ceph-mon[37789]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:31 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/3816554120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]: dispatch 2024-07-26T08:31:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:31 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]: dispatch 2024-07-26T08:31:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:31 smithi171 ceph-mon[37789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f406895e-aab0-4a79-bd14-20a6f0626b29"}]': finished 2024-07-26T08:31:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:31 smithi171 ceph-mon[37789]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T08:31:33.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:32 smithi116 ceph-mon[37097]: from='client.? 172.21.15.171:0/897729190' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:31:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:32 smithi044 ceph-mon[32404]: from='client.? 172.21.15.171:0/897729190' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:31:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:32 smithi171 ceph-mon[37789]: from='client.? 172.21.15.171:0/897729190' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T08:31:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:33 smithi044 ceph-mon[32404]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-26T08:31:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:33 smithi116 ceph-mon[37097]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-26T08:31:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:33 smithi171 ceph-mon[37789]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-26T08:31:36.031 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:35 smithi171 ceph-mon[37789]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-07-26T08:31:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:35 smithi044 ceph-mon[32404]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-07-26T08:31:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:35 smithi116 ceph-mon[37097]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-07-26T08:31:38.043 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:37 smithi171 ceph-mon[37789]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:31:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:37 smithi044 ceph-mon[32404]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:31:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:37 smithi116 ceph-mon[37097]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T08:31:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T08:31:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T08:31:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T08:31:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:40.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:39 smithi171 ceph-mon[37789]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:31:40.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:39 smithi171 ceph-mon[37789]: Deploying daemon osd.11 on smithi171 2024-07-26T08:31:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:39 smithi044 ceph-mon[32404]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:31:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:39 smithi044 ceph-mon[32404]: Deploying daemon osd.11 on smithi171 2024-07-26T08:31:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:39 smithi116 ceph-mon[37097]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:31:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:39 smithi116 ceph-mon[37097]: Deploying daemon osd.11 on smithi171 2024-07-26T08:31:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:41 smithi171 ceph-mon[37789]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:31:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:41 smithi044 ceph-mon[32404]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:31:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:41 smithi116 ceph-mon[37097]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T08:31:44.199 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:43 smithi171 ceph-mon[37789]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-07-26T08:31:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:43 smithi044 ceph-mon[32404]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-07-26T08:31:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:43 smithi116 ceph-mon[37097]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-07-26T08:31:44.985 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:44.986 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:44.986 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:46 smithi171 ceph-mon[37789]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:46 smithi044 ceph-mon[32404]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:46 smithi116 ceph-mon[37097]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:47.281 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 11 on host 'smithi171' 2024-07-26T08:31:47.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.279+0000 7ffa4bfff640 1 -- 172.21.15.171:0/1008710718 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ffa54063230 con 0x7ffa400b3c20 2024-07-26T08:31:47.285 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffa400b3c20 msgr2=0x7ffa400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffa400b3c20 0x7ffa400b6130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffa44031440 tx=0x7ffa44005d50 comp rx=0 tx=0).stop 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 msgr2=0x7ffa5419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 0x7ffa5419e6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00b970 tx=0x7ffa4c00be30 comp rx=0 tx=0).stop 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 shutdown_connections 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffa400b3c20 0x7ffa400b6130 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa54108730 0x7ffa5419e6c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa54107de0 0x7ffa54199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 --2- 172.21.15.171:0/1008710718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa541024a0 0x7ffa54198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 >> 172.21.15.171:0/1008710718 conn(0x7ffa54079140 msgr2=0x7ffa54105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 shutdown_connections 2024-07-26T08:31:47.286 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:31:47.282+0000 7ffa58a15640 1 -- 172.21.15.171:0/1008710718 wait complete. 2024-07-26T08:31:48.328 DEBUG:teuthology.orchestra.run.smithi171:osd.11> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@osd.11.service 2024-07-26T08:31:48.331 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-26T08:31:48.331 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd stat -f json 2024-07-26T08:31:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:48 smithi044 ceph-mon[32404]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:48 smithi044 ceph-mon[32404]: from='osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T08:31:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:48 smithi044 ceph-mon[32404]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T08:31:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:48 smithi116 ceph-mon[37097]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:48 smithi116 ceph-mon[37097]: from='osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T08:31:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:48 smithi116 ceph-mon[37097]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T08:31:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:48 smithi171 ceph-mon[37789]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:48 smithi171 ceph-mon[37789]: from='osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T08:31:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:48 smithi171 ceph-mon[37789]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T08:31:48.819 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:31:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T08:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: from='osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T08:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: from='osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: from='osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:50.087 INFO:journalctl@ceph.osd.11.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-11[64611]: 2024-07-26T08:31:49.729+0000 7f761c58a640 -1 osd.11 0 waiting for initial osdmap 2024-07-26T08:31:50.087 INFO:journalctl@ceph.osd.11.smithi171.stdout:Jul 26 08:31:49 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-osd-11[64611]: 2024-07-26T08:31:49.744+0000 7f7614380640 -1 osd.11 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T08:31:50.439 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:50.439 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:31:50.439 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.439 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T08:31:50.439 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: from='osd.11 ' entity='osd.11' 2024-07-26T08:31:50.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: from='osd.11 ' entity='osd.11' 2024-07-26T08:31:50.748 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-07-26T08:31:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T08:31:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: from='osd.11 ' entity='osd.11' 2024-07-26T08:31:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 -- 172.21.15.44:0/4118395374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf501055e0 msgr2=0x7fbf501059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 --2- 172.21.15.44:0/4118395374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf501055e0 0x7fbf501059f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009900 tx=0x7fbf4002a400 comp rx=0 tx=0).stop 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 -- 172.21.15.44:0/4118395374 shutdown_connections 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 --2- 172.21.15.44:0/4118395374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf5010af20 0x7fbf5010b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 --2- 172.21.15.44:0/4118395374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5010b870 0x7fbf501120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 --2- 172.21.15.44:0/4118395374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf501055e0 0x7fbf501059f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 -- 172.21.15.44:0/4118395374 >> 172.21.15.44:0/4118395374 conn(0x7fbf501010b0 msgr2=0x7fbf50103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:51.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.288+0000 7fbf4f200640 1 -- 172.21.15.44:0/4118395374 shutdown_connections 2024-07-26T08:31:51.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.289+0000 7fbf4f200640 1 -- 172.21.15.44:0/4118395374 wait complete. 2024-07-26T08:31:51.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.290+0000 7fbf4f200640 1 Processor -- start 2024-07-26T08:31:51.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.290+0000 7fbf4f200640 1 -- start start 2024-07-26T08:31:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 0x7fbf5019bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:51.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5010af20 0x7fbf5019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4f200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf5010b870 0x7fbf501a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbf50115520 con 0x7fbf5010af20 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501153a0 con 0x7fbf501055e0 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4f200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501156a0 con 0x7fbf5010b870 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 0x7fbf5019bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4d400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5010af20 0x7fbf5019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 0x7fbf5019bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:56940/0 (socket says 172.21.15.44:56940) 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4de00640 1 -- 172.21.15.44:0/2116407182 learned_addr learned my addr 172.21.15.44:0/2116407182 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.291+0000 7fbf4e800640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf5010b870 0x7fbf501a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4de00640 1 -- 172.21.15.44:0/2116407182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf5010b870 msgr2=0x7fbf501a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4de00640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf5010b870 0x7fbf501a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4de00640 1 -- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5010af20 msgr2=0x7fbf5019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4de00640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5010af20 0x7fbf5019c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4de00640 1 -- 172.21.15.44:0/2116407182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf501a1f00 con 0x7fbf501055e0 2024-07-26T08:31:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4d400640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5010af20 0x7fbf5019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:31:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf4de00640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 0x7fbf5019bf90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005940 tx=0x7fbf400040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.292+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4003f030 con 0x7fbf501055e0 2024-07-26T08:31:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.293+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf400044c0 con 0x7fbf501055e0 2024-07-26T08:31:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.293+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf501a2190 con 0x7fbf501055e0 2024-07-26T08:31:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.293+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf40035070 con 0x7fbf501055e0 2024-07-26T08:31:51.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.293+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf501a8950 con 0x7fbf501055e0 2024-07-26T08:31:51.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.294+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf501059f0 con 0x7fbf501055e0 2024-07-26T08:31:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.297+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fbf40042030 con 0x7fbf501055e0 2024-07-26T08:31:51.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.298+0000 7fbf46a00640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbf340b3ca0 0x7fbf340b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:51.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.298+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fbf40103be0 con 0x7fbf501055e0 2024-07-26T08:31:51.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.298+0000 7fbf4d400640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbf340b3ca0 0x7fbf340b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:51.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.299+0000 7fbf4d400640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbf340b3ca0 0x7fbf340b61b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf5010ab90 tx=0x7fbf3c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:51.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.303+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbf400cb560 con 0x7fbf501055e0 2024-07-26T08:31:51.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.511+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fbf5002e0c0 con 0x7fbf501055e0 2024-07-26T08:31:51.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.512+0000 7fbf46a00640 1 -- 172.21.15.44:0/2116407182 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7fbf400cae60 con 0x7fbf501055e0 2024-07-26T08:31:51.516 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:31:51.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbf340b3ca0 msgr2=0x7fbf340b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:51.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbf340b3ca0 0x7fbf340b61b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf5010ab90 tx=0x7fbf3c005f90 comp rx=0 tx=0).stop 2024-07-26T08:31:51.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 msgr2=0x7fbf5019bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 0x7fbf5019bf90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005940 tx=0x7fbf400040c0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 shutdown_connections 2024-07-26T08:31:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbf340b3ca0 0x7fbf340b61b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf5010b870 0x7fbf501a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf5010af20 0x7fbf5019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 --2- 172.21.15.44:0/2116407182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf501055e0 0x7fbf5019bf90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:51.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.518+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 >> 172.21.15.44:0/2116407182 conn(0x7fbf501010b0 msgr2=0x7fbf501034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:51.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.519+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 shutdown_connections 2024-07-26T08:31:51.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:51.519+0000 7fbf4f200640 1 -- 172.21.15.44:0/2116407182 wait complete. 2024-07-26T08:31:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: purged_snaps scrub starts 2024-07-26T08:31:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: purged_snaps scrub ok 2024-07-26T08:31:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] boot 2024-07-26T08:31:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T08:31:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:51 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2116407182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T08:31:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: purged_snaps scrub starts 2024-07-26T08:31:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: purged_snaps scrub ok 2024-07-26T08:31:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] boot 2024-07-26T08:31:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T08:31:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:51 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2116407182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: purged_snaps scrub starts 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: purged_snaps scrub ok 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: osd.11 [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] boot 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T08:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:51 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2116407182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T08:31:52.448 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1721982710,"num_in_osds":12,"osd_in_since":1721982691,"num_remapped_pgs":0} 2024-07-26T08:31:52.449 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:31:52.945 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: Detected new or changed devices on smithi171 2024-07-26T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 4272M 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:53.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:53.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:53.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: Detected new or changed devices on smithi171 2024-07-26T08:31:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T08:31:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 4272M 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: Detected new or changed devices on smithi171 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 4272M 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:31:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:54 smithi044 ceph-mon[32404]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:54 smithi171 ceph-mon[37789]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:54 smithi116 ceph-mon[37097]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:55.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 -- 172.21.15.44:0/3369401188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 msgr2=0x7f190c1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:55.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 --2- 172.21.15.44:0/3369401188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c1037f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009ca0 tx=0x7f18fc02a8c0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 -- 172.21.15.44:0/3369401188 shutdown_connections 2024-07-26T08:31:55.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 --2- 172.21.15.44:0/3369401188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f190c109670 0x7f190c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 --2- 172.21.15.44:0/3369401188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 0x7f190c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 --2- 172.21.15.44:0/3369401188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c1037f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 -- 172.21.15.44:0/3369401188 >> 172.21.15.44:0/3369401188 conn(0x7f190c0fee30 msgr2=0x7f190c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:55.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 -- 172.21.15.44:0/3369401188 shutdown_connections 2024-07-26T08:31:55.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.160+0000 7f190be00640 1 -- 172.21.15.44:0/3369401188 wait complete. 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 Processor -- start 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 -- start start 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 0x7f190c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f190c109670 0x7f190c1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f190c113390 con 0x7f190c1033e0 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f190c113210 con 0x7f190c108d20 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.162+0000 7f190be00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f190c113510 con 0x7f190c109670 2024-07-26T08:31:55.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:55.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 0x7f190c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:55.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c19bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53806/0 (socket says 172.21.15.44:53806) 2024-07-26T08:31:55.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190aa00640 1 -- 172.21.15.44:0/1994986317 learned_addr learned my addr 172.21.15.44:0/1994986317 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:31:55.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190b400640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f190c109670 0x7f190c1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190a000640 1 -- 172.21.15.44:0/1994986317 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f190c109670 msgr2=0x7f190c1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190a000640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f190c109670 0x7f190c1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190a000640 1 -- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 msgr2=0x7f190c19bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190a000640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c19bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.163+0000 7f190a000640 1 -- 172.21.15.44:0/1994986317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c1a1eb0 con 0x7f190c108d20 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.164+0000 7f190a000640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 0x7f190c19c4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f190c06b360 tx=0x7f18f400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.164+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f400cd00 con 0x7f190c108d20 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.164+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18f40035f0 con 0x7f190c108d20 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.164+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190c1a21a0 con 0x7f190c108d20 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.164+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4010690 con 0x7f190c108d20 2024-07-26T08:31:55.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.164+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f190c1a8900 con 0x7f190c108d20 2024-07-26T08:31:55.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.165+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f190c1037f0 con 0x7f190c108d20 2024-07-26T08:31:55.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.167+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f18f40037a0 con 0x7f190c108d20 2024-07-26T08:31:55.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.169+0000 7f1910a00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f18f00b3ca0 0x7f18f00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:55.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.169+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f18f40e3af0 con 0x7f190c108d20 2024-07-26T08:31:55.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.169+0000 7f190aa00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f18f00b3ca0 0x7f18f00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:55.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.170+0000 7f190aa00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f18f00b3ca0 0x7f18f00b61b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009a60 tx=0x7f18fc005d80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:55.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.173+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f18f40ab470 con 0x7f190c108d20 2024-07-26T08:31:55.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.379+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f190c1a8c40 con 0x7f190c108d20 2024-07-26T08:31:55.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.381+0000 7f1910a00640 1 -- 172.21.15.44:0/1994986317 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18769 (secure 0 0 0) 0x7f18f40aad70 con 0x7f190c108d20 2024-07-26T08:31:55.384 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:31:55.384 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":73,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:31:51.741976+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:31:55.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f18f00b3ca0 msgr2=0x7f18f00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:55.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f18f00b3ca0 0x7f18f00b61b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009a60 tx=0x7f18fc005d80 comp rx=0 tx=0).stop 2024-07-26T08:31:55.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 msgr2=0x7f190c19c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:55.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 0x7f190c19c4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f190c06b360 tx=0x7f18f400ee50 comp rx=0 tx=0).stop 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 shutdown_connections 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f18f00b3ca0 0x7f18f00b61b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f190c109670 0x7f190c1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f190c108d20 0x7f190c19c4e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 --2- 172.21.15.44:0/1994986317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f190c1033e0 0x7f190c19bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 >> 172.21.15.44:0/1994986317 conn(0x7f190c0fee30 msgr2=0x7f190c10e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 shutdown_connections 2024-07-26T08:31:55.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:55.384+0000 7f190be00640 1 -- 172.21.15.44:0/1994986317 wait complete. 2024-07-26T08:31:56.269 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T08:27:40.734083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T08:31:56.270 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-26T08:31:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:56 smithi171 ceph-mon[37789]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:56 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1994986317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:31:56.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:56 smithi044 ceph-mon[32404]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:56.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:56 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1994986317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:31:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:56 smithi116 ceph-mon[37097]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:56 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1994986317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:31:56.761 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:31:58.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:31:58 smithi044 ceph-mon[32404]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:31:58 smithi171 ceph-mon[37789]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:31:58 smithi116 ceph-mon[37097]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 -- 172.21.15.44:0/2840346274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410f330 msgr2=0x7f81c410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 --2- 172.21.15.44:0/2840346274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410f330 0x7f81c410f740 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009900 tx=0x7f81b002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 -- 172.21.15.44:0/2840346274 shutdown_connections 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 --2- 172.21.15.44:0/2840346274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81c410fc80 0x7f81c41164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 --2- 172.21.15.44:0/2840346274 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410f330 0x7f81c410f740 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 --2- 172.21.15.44:0/2840346274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c41099f0 0x7f81c4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.979+0000 7f81cb000640 1 -- 172.21.15.44:0/2840346274 >> 172.21.15.44:0/2840346274 conn(0x7f81c4078ff0 msgr2=0x7f81c4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.980+0000 7f81cb000640 1 -- 172.21.15.44:0/2840346274 shutdown_connections 2024-07-26T08:31:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.980+0000 7f81cb000640 1 -- 172.21.15.44:0/2840346274 wait complete. 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.981+0000 7f81cb000640 1 Processor -- start 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.981+0000 7f81cb000640 1 -- start start 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81cb000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81c41099f0 0x7f81c41a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81cb000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 0x7f81c41a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81cb000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410fc80 0x7f81c41a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81cb000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f81c4119940 con 0x7f81c410fc80 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81cb000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81c41197c0 con 0x7f81c410f330 2024-07-26T08:31:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81cb000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f81c4119ac0 con 0x7f81c41099f0 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81c9200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 0x7f81c41a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81c9200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 0x7f81c41a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:38574/0 (socket says 172.21.15.44:38574) 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81c9200640 1 -- 172.21.15.44:0/2527915307 learned_addr learned my addr 172.21.15.44:0/2527915307 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81ca600640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410fc80 0x7f81c41a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.982+0000 7f81c9c00640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81c41099f0 0x7f81c41a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c9200640 1 -- 172.21.15.44:0/2527915307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81c41099f0 msgr2=0x7f81c41a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c9200640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81c41099f0 0x7f81c41a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c9200640 1 -- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410fc80 msgr2=0x7f81c41a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c9200640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410fc80 0x7f81c41a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c9200640 1 -- 172.21.15.44:0/2527915307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c41a62a0 con 0x7f81c410f330 2024-07-26T08:31:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c9200640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 0x7f81c41a08d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81b002a910 tx=0x7f81b0031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.983+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b003f030 con 0x7f81c410f330 2024-07-26T08:31:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.984+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c41a6530 con 0x7f81c410f330 2024-07-26T08:31:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.984+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81b0004490 con 0x7f81c410f330 2024-07-26T08:31:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.984+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0035070 con 0x7f81c410f330 2024-07-26T08:31:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.984+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c41accf0 con 0x7f81c410f330 2024-07-26T08:31:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.985+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c4109e00 con 0x7f81c410f330 2024-07-26T08:31:58.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.988+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f81b0042030 con 0x7f81c410f330 2024-07-26T08:31:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.989+0000 7f81c2000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81ac0b3c20 0x7f81ac0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:31:58.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.989+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f81b0103ff0 con 0x7f81c410f330 2024-07-26T08:31:58.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.989+0000 7f81c9c00640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81ac0b3c20 0x7f81ac0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:31:58.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.990+0000 7f81c9c00640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81ac0b3c20 0x7f81ac0b6130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009bd0 tx=0x7f81b8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:31:58.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:58.991+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f81b00cb360 con 0x7f81c410f330 2024-07-26T08:31:59.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.194+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f81c4111a70 con 0x7f81c410f330 2024-07-26T08:31:59.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.195+0000 7f81c2000640 1 -- 172.21.15.44:0/2527915307 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f81b00cf8b0 con 0x7f81c410f330 2024-07-26T08:31:59.198 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-07-26T08:31:59.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.198+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81ac0b3c20 msgr2=0x7f81ac0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:59.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.198+0000 7f81cb000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81ac0b3c20 0x7f81ac0b6130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009bd0 tx=0x7f81b8005f90 comp rx=0 tx=0).stop 2024-07-26T08:31:59.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.198+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 msgr2=0x7f81c41a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:31:59.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.198+0000 7f81cb000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 0x7f81c41a08d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81b002a910 tx=0x7f81b0031a20 comp rx=0 tx=0).stop 2024-07-26T08:31:59.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 shutdown_connections 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81ac0b3c20 0x7f81ac0b6130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81c410fc80 0x7f81c41a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81c410f330 0x7f81c41a08d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 --2- 172.21.15.44:0/2527915307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81c41099f0 0x7f81c41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 >> 172.21.15.44:0/2527915307 conn(0x7f81c4078ff0 msgr2=0x7f81c4107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 shutdown_connections 2024-07-26T08:31:59.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:31:59.199+0000 7f81cb000640 1 -- 172.21.15.44:0/2527915307 wait complete. 2024-07-26T08:32:00.027 INFO:tasks.cephadm:Adding mds.a on smithi044 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.d on smithi044 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.g on smithi044 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.j on smithi044 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.b on smithi116 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.e on smithi116 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.h on smithi116 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.k on smithi116 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.c on smithi171 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.f on smithi171 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.i on smithi171 2024-07-26T08:32:00.028 INFO:tasks.cephadm:Adding mds.l on smithi171 2024-07-26T08:32:00.029 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph orch apply mds all '12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l' 2024-07-26T08:32:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:00 smithi044 ceph-mon[32404]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:00 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2527915307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T08:32:00.293 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T08:32:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:00 smithi171 ceph-mon[37789]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:00 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2527915307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T08:32:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:00 smithi116 ceph-mon[37097]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:00 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2527915307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T08:32:01.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 -- 172.21.15.171:0/4032599369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 msgr2=0x7f19d01081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:01.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 --2- 172.21.15.171:0/4032599369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d01081e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009900 tx=0x7f19b802a410 comp rx=0 tx=0).stop 2024-07-26T08:32:01.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 -- 172.21.15.171:0/4032599369 shutdown_connections 2024-07-26T08:32:01.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 --2- 172.21.15.171:0/4032599369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d01081e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:01.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 --2- 172.21.15.171:0/4032599369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19d0108720 0x7f19d010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:01.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 --2- 172.21.15.171:0/4032599369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d0102490 0x7f19d01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:01.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 -- 172.21.15.171:0/4032599369 >> 172.21.15.171:0/4032599369 conn(0x7f19d0079160 msgr2=0x7f19d0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:01.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 -- 172.21.15.171:0/4032599369 shutdown_connections 2024-07-26T08:32:01.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.958+0000 7f19d5143640 1 -- 172.21.15.171:0/4032599369 wait complete. 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.959+0000 7f19d5143640 1 Processor -- start 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 -- start start 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d0102490 0x7f19d010f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d010fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19d0108720 0x7f19d0110420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f19d01134f0 con 0x7f19d0102490 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19d0113370 con 0x7f19d0108720 2024-07-26T08:32:01.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19d5143640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f19d0113670 con 0x7f19d0107dd0 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d010fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d010fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:44794/0 (socket says 172.21.15.171:44794) 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 -- 172.21.15.171:0/4054257988 learned_addr learned my addr 172.21.15.171:0/4054257988 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19cf59e640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19d0108720 0x7f19d0110420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 -- 172.21.15.171:0/4054257988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19d0108720 msgr2=0x7f19d0110420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19d0108720 0x7f19d0110420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 -- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d0102490 msgr2=0x7f19d010f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d0102490 0x7f19d010f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.960+0000 7f19ce59c640 1 -- 172.21.15.171:0/4054257988 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d01a81a0 con 0x7f19d0107dd0 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.961+0000 7f19ce59c640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d010fee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f19b8032fd0 tx=0x7f19b8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.961+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b803f030 con 0x7f19d0107dd0 2024-07-26T08:32:01.965 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.961+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b8004490 con 0x7f19d0107dd0 2024-07-26T08:32:01.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.961+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d01a8430 con 0x7f19d0107dd0 2024-07-26T08:32:01.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.961+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8035070 con 0x7f19d0107dd0 2024-07-26T08:32:01.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.961+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d01a8990 con 0x7f19d0107dd0 2024-07-26T08:32:01.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.962+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d01028a0 con 0x7f19d0107dd0 2024-07-26T08:32:01.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.963+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f19b8042030 con 0x7f19d0107dd0 2024-07-26T08:32:01.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.964+0000 7f19c7fff640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19bc0b3c20 0x7f19bc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:01.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.964+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f19b81041a0 con 0x7f19d0107dd0 2024-07-26T08:32:01.967 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.965+0000 7f19ced9d640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19bc0b3c20 0x7f19bc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:01.968 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.966+0000 7f19ced9d640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19bc0b3c20 0x7f19bc0b6130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19d01113f0 tx=0x7f19c0006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:01.969 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:01.967+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f19b80cbbc0 con 0x7f19d0107dd0 2024-07-26T08:32:02.154 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.152+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}) -- 0x7f19d0063330 con 0x7f19bc0b3c20 2024-07-26T08:32:02.173 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.171+0000 7f19c7fff640 1 -- 172.21.15.171:0/4054257988 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f19d0063330 con 0x7f19bc0b3c20 2024-07-26T08:32:02.174 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mds.all update... 2024-07-26T08:32:02.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.173+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19bc0b3c20 msgr2=0x7f19bc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:02.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.173+0000 7f19d5143640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19bc0b3c20 0x7f19bc0b6130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f19d01113f0 tx=0x7f19c0006c90 comp rx=0 tx=0).stop 2024-07-26T08:32:02.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 msgr2=0x7f19d010fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d010fee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f19b8032fd0 tx=0x7f19b8004270 comp rx=0 tx=0).stop 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 shutdown_connections 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19bc0b3c20 0x7f19bc0b6130 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19d0107dd0 0x7f19d010fee0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19d0108720 0x7f19d0110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 --2- 172.21.15.171:0/4054257988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d0102490 0x7f19d010f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 >> 172.21.15.171:0/4054257988 conn(0x7f19d0079160 msgr2=0x7f19d0104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 shutdown_connections 2024-07-26T08:32:02.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-07-26T08:32:02.174+0000 7f19d5143640 1 -- 172.21.15.171:0/4054257988 wait complete. 2024-07-26T08:32:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:02 smithi171 ceph-mon[37789]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:02 smithi044 ceph-mon[32404]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:02 smithi116 ceph-mon[37097]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:02.842 DEBUG:teuthology.orchestra.run.smithi044:mds.a> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service 2024-07-26T08:32:02.848 DEBUG:teuthology.orchestra.run.smithi044:mds.d> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.d.service 2024-07-26T08:32:02.897 DEBUG:teuthology.orchestra.run.smithi044:mds.g> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.g.service 2024-07-26T08:32:02.943 DEBUG:teuthology.orchestra.run.smithi044:mds.j> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service 2024-07-26T08:32:02.988 DEBUG:teuthology.orchestra.run.smithi116:mds.b> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service 2024-07-26T08:32:02.990 DEBUG:teuthology.orchestra.run.smithi116:mds.e> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.e.service 2024-07-26T08:32:03.033 DEBUG:teuthology.orchestra.run.smithi116:mds.h> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.h.service 2024-07-26T08:32:03.077 DEBUG:teuthology.orchestra.run.smithi116:mds.k> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.k.service 2024-07-26T08:32:03.121 DEBUG:teuthology.orchestra.run.smithi171:mds.c> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.c.service 2024-07-26T08:32:03.123 DEBUG:teuthology.orchestra.run.smithi171:mds.f> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service 2024-07-26T08:32:03.166 DEBUG:teuthology.orchestra.run.smithi171:mds.i> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.i.service 2024-07-26T08:32:03.209 DEBUG:teuthology.orchestra.run.smithi171:mds.l> sudo journalctl -f -n 0 -u ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.l.service 2024-07-26T08:32:03.252 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-26T08:32:03.252 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:03.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:03.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:03.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:03.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:03.441 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:03.739 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:04 smithi044 ceph-mon[32404]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:04 smithi044 ceph-mon[32404]: Deploying daemon mds.l on smithi171 2024-07-26T08:32:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:04 smithi116 ceph-mon[37097]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:04 smithi116 ceph-mon[37097]: Deploying daemon mds.l on smithi171 2024-07-26T08:32:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:04 smithi171 ceph-mon[37789]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:04 smithi171 ceph-mon[37789]: Deploying daemon mds.l on smithi171 2024-07-26T08:32:05.314 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 systemd[1]: Starting Ceph mds.l for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:05.587 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 podman[68948]: 2024-07-26 08:32:05.216239975 +0000 UTC m=+0.016810068 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:05.587 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 podman[68948]: 2024-07-26 08:32:05.363693779 +0000 UTC m=+0.164263852 container create 2b05a4793a1ba26e93295f815008a39d51121a8a02cb34fb83b7620aec9e0112 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-l, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux ) 2024-07-26T08:32:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 -- 172.21.15.44:0/1020618834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 msgr2=0x7f09bc1164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 --2- 172.21.15.44:0/1020618834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1164b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009be0 tx=0x7f09b402a6c0 comp rx=0 tx=0).stop 2024-07-26T08:32:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 -- 172.21.15.44:0/1020618834 shutdown_connections 2024-07-26T08:32:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 --2- 172.21.15.44:0/1020618834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1164b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 --2- 172.21.15.44:0/1020618834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09bc10f330 0x7f09bc10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 --2- 172.21.15.44:0/1020618834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09bc1099f0 0x7f09bc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 -- 172.21.15.44:0/1020618834 >> 172.21.15.44:0/1020618834 conn(0x7f09bc078ff0 msgr2=0x7f09bc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 -- 172.21.15.44:0/1020618834 shutdown_connections 2024-07-26T08:32:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.978+0000 7f09c2600640 1 -- 172.21.15.44:0/1020618834 wait complete. 2024-07-26T08:32:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.979+0000 7f09c2600640 1 Processor -- start 2024-07-26T08:32:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 -- start start 2024-07-26T08:32:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09bc1099f0 0x7f09bc1a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09bc10f330 0x7f09bc1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09bc119940 con 0x7f09bc1099f0 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f09bc1197c0 con 0x7f09bc10fc80 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c2600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f09bc119ac0 con 0x7f09bc10f330 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c1c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c1c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1a5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:42068/0 (socket says 172.21.15.44:42068) 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09bbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09bc10f330 0x7f09bc1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.980+0000 7f09c1c00640 1 -- 172.21.15.44:0/171841523 learned_addr learned my addr 172.21.15.44:0/171841523 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1200640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09bc1099f0 0x7f09bc1a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1c00640 1 -- 172.21.15.44:0/171841523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09bc10f330 msgr2=0x7f09bc1a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1c00640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09bc10f330 0x7f09bc1a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1c00640 1 -- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09bc1099f0 msgr2=0x7f09bc1a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1c00640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09bc1099f0 0x7f09bc1a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1c00640 1 -- 172.21.15.44:0/171841523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09bc1a6300 con 0x7f09bc10fc80 2024-07-26T08:32:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.981+0000 7f09c1c00640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1a5bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009b30 tx=0x7f09b4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.982+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4040020 con 0x7f09bc10fc80 2024-07-26T08:32:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.982+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09b4030970 con 0x7f09bc10fc80 2024-07-26T08:32:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.982+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09bc1a6590 con 0x7f09bc10fc80 2024-07-26T08:32:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.982+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4036040 con 0x7f09bc10fc80 2024-07-26T08:32:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.982+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09bc1acd50 con 0x7f09bc10fc80 2024-07-26T08:32:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.983+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09bc109e00 con 0x7f09bc10fc80 2024-07-26T08:32:05.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.986+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f09b4043020 con 0x7f09bc10fc80 2024-07-26T08:32:05.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.987+0000 7f09b9600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09a40b3ca0 0x7f09a40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:05.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.987+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f09b4104820 con 0x7f09bc10fc80 2024-07-26T08:32:05.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.988+0000 7f09c1200640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09a40b3ca0 0x7f09a40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:05.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.988+0000 7f09c1200640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09a40b3ca0 0x7f09a40b61b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009bd0 tx=0x7f09b0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:05.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:05.989+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f09b404a020 con 0x7f09bc10fc80 2024-07-26T08:32:06.087 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 podman[68948]: 2024-07-26 08:32:05.603208852 +0000 UTC m=+0.403778926 container init 2b05a4793a1ba26e93295f815008a39d51121a8a02cb34fb83b7620aec9e0112 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-07-26T08:32:06.087 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 podman[68948]: 2024-07-26 08:32:05.605914763 +0000 UTC m=+0.406484840 container start 2b05a4793a1ba26e93295f815008a39d51121a8a02cb34fb83b7620aec9e0112 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True) 2024-07-26T08:32:06.087 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 ceph-mds[68967]: -- 172.21.15.171:0/1890006079 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ead7bf860 con 0x556eb1961600 2024-07-26T08:32:06.087 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-l[68963]: starting mds.l at 2024-07-26T08:32:06.087 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 bash[68948]: 2b05a4793a1ba26e93295f815008a39d51121a8a02cb34fb83b7620aec9e0112 2024-07-26T08:32:06.087 INFO:journalctl@ceph.mds.l.smithi171.stdout:Jul 26 08:32:05 smithi171 systemd[1]: Started Ceph mds.l for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.232+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09bc111a70 con 0x7f09bc10fc80 2024-07-26T08:32:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.233+0000 7f09b9600640 1 -- 172.21.15.44:0/171841523 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f09b40cc1a0 con 0x7f09bc10fc80 2024-07-26T08:32:06.235 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:06.235 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":2,"btime":"2024-07-26T08:32:05:644346+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-26T08:32:06.235 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 2 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09a40b3ca0 msgr2=0x7f09a40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09a40b3ca0 0x7f09a40b61b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009bd0 tx=0x7f09b0005f90 comp rx=0 tx=0).stop 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 msgr2=0x7f09bc1a5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1a5bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009b30 tx=0x7f09b4003c10 comp rx=0 tx=0).stop 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 shutdown_connections 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09a40b3ca0 0x7f09a40b61b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09bc10f330 0x7f09bc1a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:06.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09bc10fc80 0x7f09bc1a5bd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:06.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 --2- 172.21.15.44:0/171841523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09bc1099f0 0x7f09bc1a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:06.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 >> 172.21.15.44:0/171841523 conn(0x7f09bc078ff0 msgr2=0x7f09bc079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:06.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.237+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 shutdown_connections 2024-07-26T08:32:06.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:06.238+0000 7f09c2600640 1 -- 172.21.15.44:0/171841523 wait complete. 2024-07-26T08:32:06.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6832/4260884949,v1:172.21.15.171:6833/4260884949] up:boot 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: fsmap 1 up:standby 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:06.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6832/4260884949,v1:172.21.15.171:6833/4260884949] up:boot 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: fsmap 1 up:standby 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6832/4260884949,v1:172.21.15.171:6833/4260884949] up:boot 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: fsmap 1 up:standby 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:07 smithi044 ceph-mon[32404]: Deploying daemon mds.k on smithi116 2024-07-26T08:32:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:07 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/171841523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:07.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:07 smithi116 ceph-mon[37097]: Deploying daemon mds.k on smithi116 2024-07-26T08:32:07.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:07 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/171841523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:07 smithi171 ceph-mon[37789]: Deploying daemon mds.k on smithi116 2024-07-26T08:32:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:07 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/171841523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:08.199 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 podman[67290]: 2024-07-26 08:32:08.058584719 +0000 UTC m=+0.135276549 container create 484d029805f4056a0c882de3add682bdc3f3d75c038e11a5479b6038966111a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:32:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:08 smithi044 ceph-mon[32404]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:08 smithi116 ceph-mon[37097]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:08.522 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 podman[67290]: 2024-07-26 08:32:08.319943078 +0000 UTC m=+0.396634908 container init 484d029805f4056a0c882de3add682bdc3f3d75c038e11a5479b6038966111a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T08:32:08.522 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 podman[67290]: 2024-07-26 08:32:08.322846488 +0000 UTC m=+0.399538328 container start 484d029805f4056a0c882de3add682bdc3f3d75c038e11a5479b6038966111a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-k, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:32:08.523 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 ceph-mds[67309]: -- 172.21.15.116:0/3756485182 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55672ddc7860 con 0x556731f69180 2024-07-26T08:32:08.523 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-k[67305]: starting mds.k at 2024-07-26T08:32:08.523 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 bash[67290]: 484d029805f4056a0c882de3add682bdc3f3d75c038e11a5479b6038966111a3 2024-07-26T08:32:08.523 INFO:journalctl@ceph.mds.k.smithi116.stdout:Jul 26 08:32:08 smithi116 systemd[1]: Started Ceph mds.k for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:08 smithi171 ceph-mon[37789]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:09.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:09.729 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.729 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.729 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6832/2097534847,v1:172.21.15.116:6833/2097534847] up:boot 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: fsmap 2 up:standby 2024-07-26T08:32:09.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T08:32:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6832/2097534847,v1:172.21.15.116:6833/2097534847] up:boot 2024-07-26T08:32:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: fsmap 2 up:standby 2024-07-26T08:32:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6832/2097534847,v1:172.21.15.116:6833/2097534847] up:boot 2024-07-26T08:32:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: fsmap 2 up:standby 2024-07-26T08:32:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T08:32:10.555 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:10.728 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:10 smithi044 ceph-mon[32404]: Deploying daemon mds.j on smithi044 2024-07-26T08:32:10.728 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:10 smithi044 ceph-mon[32404]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:10.728 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:10 smithi116 ceph-mon[37097]: Deploying daemon mds.j on smithi044 2024-07-26T08:32:10.728 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:10 smithi116 ceph-mon[37097]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:10 smithi171 ceph-mon[37789]: Deploying daemon mds.j on smithi044 2024-07-26T08:32:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:10 smithi171 ceph-mon[37789]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:11.517 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:11 smithi044 systemd[1]: Starting Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:12.419 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 podman[78047]: 2024-07-26 08:32:12.041743896 +0000 UTC m=+0.036116892 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:12.419 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 podman[78047]: 2024-07-26 08:32:12.152177367 +0000 UTC m=+0.146550396 container create bd4dfc52cffd01e3bc1a5b41aebeb643578aed1231d0beeec659a99478c2146d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T08:32:12.736 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:12 smithi044 ceph-mon[32404]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:12.736 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 podman[78047]: 2024-07-26 08:32:12.457147674 +0000 UTC m=+0.451520707 container init bd4dfc52cffd01e3bc1a5b41aebeb643578aed1231d0beeec659a99478c2146d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:32:12.737 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 podman[78047]: 2024-07-26 08:32:12.467901945 +0000 UTC m=+0.462274947 container start bd4dfc52cffd01e3bc1a5b41aebeb643578aed1231d0beeec659a99478c2146d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T08:32:12.737 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 ceph-mds[78084]: -- 172.21.15.44:0/2801867982 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5630b6237860 con 0x5630ba3d9a80 2024-07-26T08:32:12.737 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: starting mds.j at 2024-07-26T08:32:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:12 smithi116 ceph-mon[37097]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:12 smithi171 ceph-mon[37789]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:13.004 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 bash[78047]: bd4dfc52cffd01e3bc1a5b41aebeb643578aed1231d0beeec659a99478c2146d 2024-07-26T08:32:13.005 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:32:12 smithi044 systemd[1]: Started Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:13.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 -- 172.21.15.44:0/1827884520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104d90 msgr2=0x7fe02410d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:13.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 --2- 172.21.15.44:0/1827884520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104d90 0x7fe02410d2e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe010009860 tx=0x7fe01002a340 comp rx=0 tx=0).stop 2024-07-26T08:32:13.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 -- 172.21.15.44:0/1827884520 shutdown_connections 2024-07-26T08:32:13.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 --2- 172.21.15.44:0/1827884520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 0x7fe02410fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 --2- 172.21.15.44:0/1827884520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104d90 0x7fe02410d2e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 --2- 172.21.15.44:0/1827884520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe024104440 0x7fe024104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 -- 172.21.15.44:0/1827884520 >> 172.21.15.44:0/1827884520 conn(0x7fe024079150 msgr2=0x7fe024079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:13.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.112+0000 7fe029e00640 1 -- 172.21.15.44:0/1827884520 shutdown_connections 2024-07-26T08:32:13.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.113+0000 7fe029e00640 1 -- 172.21.15.44:0/1827884520 wait complete. 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 Processor -- start 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 -- start start 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104440 0x7fe02419c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe024104d90 0x7fe02419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 0x7fe0241a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe024113480 con 0x7fe024104d90 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe024113300 con 0x7fe024104440 2024-07-26T08:32:13.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.114+0000 7fe029e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe024113600 con 0x7fe02410d870 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe023400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 0x7fe0241a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104440 0x7fe02419c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe023400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 0x7fe0241a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:45228/0 (socket says 172.21.15.44:45228) 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe023400640 1 -- 172.21.15.44:0/2440146435 learned_addr learned my addr 172.21.15.44:0/2440146435 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022000640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe024104d90 0x7fe02419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022a00640 1 -- 172.21.15.44:0/2440146435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 msgr2=0x7fe0241a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022a00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 0x7fe0241a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022a00640 1 -- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe024104d90 msgr2=0x7fe02419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022a00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe024104d90 0x7fe02419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.115+0000 7fe022a00640 1 -- 172.21.15.44:0/2440146435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0241a2060 con 0x7fe024104440 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.116+0000 7fe022a00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104440 0x7fe02419c0f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe01400e790 tx=0x7fe01400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.116+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01400eea0 con 0x7fe024104440 2024-07-26T08:32:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.116+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe01400ccc0 con 0x7fe024104440 2024-07-26T08:32:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.116+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe014018680 con 0x7fe024104440 2024-07-26T08:32:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.116+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0241a2350 con 0x7fe024104440 2024-07-26T08:32:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.116+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241a8ab0 con 0x7fe024104440 2024-07-26T08:32:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.117+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe024104850 con 0x7fe024104440 2024-07-26T08:32:13.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.120+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fe014005030 con 0x7fe024104440 2024-07-26T08:32:13.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.121+0000 7fe028a00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe0080b3ca0 0x7fe0080b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:13.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.121+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7fe0140e3a30 con 0x7fe024104440 2024-07-26T08:32:13.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.121+0000 7fe022000640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe0080b3ca0 0x7fe0080b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:13.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.122+0000 7fe022000640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe0080b3ca0 0x7fe0080b61b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe010035040 tx=0x7fe010030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:13.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.123+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe0140aad60 con 0x7fe024104440 2024-07-26T08:32:13.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.372+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0241a8df0 con 0x7fe024104440 2024-07-26T08:32:13.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.373+0000 7fe028a00640 1 -- 172.21.15.44:0/2440146435 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2266 (secure 0 0 0) 0x7fe0140cd020 con 0x7fe024104440 2024-07-26T08:32:13.375 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:13.375 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":3,"btime":"2024-07-26T08:32:09:197895+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-07-26T08:32:13.376 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 3 2024-07-26T08:32:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe0080b3ca0 msgr2=0x7fe0080b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe0080b3ca0 0x7fe0080b61b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe010035040 tx=0x7fe010030340 comp rx=0 tx=0).stop 2024-07-26T08:32:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104440 msgr2=0x7fe02419c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:13.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104440 0x7fe02419c0f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe01400e790 tx=0x7fe01400ec50 comp rx=0 tx=0).stop 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 shutdown_connections 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe0080b3ca0 0x7fe0080b61b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe02410d870 0x7fe0241a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe024104d90 0x7fe02419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 --2- 172.21.15.44:0/2440146435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe024104440 0x7fe02419c0f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 >> 172.21.15.44:0/2440146435 conn(0x7fe024079150 msgr2=0x7fe024100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 shutdown_connections 2024-07-26T08:32:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:13.377+0000 7fe029e00640 1 -- 172.21.15.44:0/2440146435 wait complete. 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: Deploying daemon mds.d on smithi044 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2440146435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:14.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:boot 2024-07-26T08:32:14.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: fsmap 3 up:standby 2024-07-26T08:32:14.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:32:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: Deploying daemon mds.d on smithi044 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2440146435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:boot 2024-07-26T08:32:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: fsmap 3 up:standby 2024-07-26T08:32:14.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: Deploying daemon mds.d on smithi044 2024-07-26T08:32:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2440146435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:boot 2024-07-26T08:32:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: fsmap 3 up:standby 2024-07-26T08:32:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:32:16.180 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:16.259 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:15 smithi044 ceph-mon[32404]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:15 smithi116 ceph-mon[37097]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:15 smithi171 ceph-mon[37789]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:16.985 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:16 smithi044 podman[78433]: 2024-07-26 08:32:16.711118131 +0000 UTC m=+0.159713553 container create 75404d7d97753f2b1845fce0362d2f281e191bb4219c8bb50641a6bfe55fdd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:32:16.985 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:16 smithi044 podman[78433]: 2024-07-26 08:32:16.981916284 +0000 UTC m=+0.430511736 container init 75404d7d97753f2b1845fce0362d2f281e191bb4219c8bb50641a6bfe55fdd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T08:32:17.122 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:17.266 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:16 smithi044 podman[78433]: 2024-07-26 08:32:16.992174556 +0000 UTC m=+0.440769956 container start 75404d7d97753f2b1845fce0362d2f281e191bb4219c8bb50641a6bfe55fdd83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:32:17.266 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:17 smithi044 bash[78433]: 75404d7d97753f2b1845fce0362d2f281e191bb4219c8bb50641a6bfe55fdd83 2024-07-26T08:32:17.266 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:17 smithi044 ceph-mds[78480]: -- 172.21.15.44:0/3839702273 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b2ee0d860 con 0x564b32faf180 2024-07-26T08:32:17.266 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:17 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-d[78464]: starting mds.d at 2024-07-26T08:32:17.266 INFO:journalctl@ceph.mds.d.smithi044.stdout:Jul 26 08:32:17 smithi044 systemd[1]: Started Ceph mds.d for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: Deploying daemon mds.e on smithi116 2024-07-26T08:32:18.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:32:18.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6836/3938332919,v1:172.21.15.44:6837/3938332919] up:boot 2024-07-26T08:32:18.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:18 smithi116 ceph-mon[37097]: fsmap 4 up:standby 2024-07-26T08:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: Deploying daemon mds.e on smithi116 2024-07-26T08:32:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:32:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6836/3938332919,v1:172.21.15.44:6837/3938332919] up:boot 2024-07-26T08:32:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:18 smithi044 ceph-mon[32404]: fsmap 4 up:standby 2024-07-26T08:32:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:18.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:18.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:18.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:18.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:18.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: Deploying daemon mds.e on smithi116 2024-07-26T08:32:18.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T08:32:18.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6836/3938332919,v1:172.21.15.44:6837/3938332919] up:boot 2024-07-26T08:32:18.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:18 smithi171 ceph-mon[37789]: fsmap 4 up:standby 2024-07-26T08:32:19.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 -- 172.21.15.44:0/797945641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4108d00 msgr2=0x7fe9d4109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:19.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 --2- 172.21.15.44:0/797945641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4108d00 0x7fe9d4109110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009900 tx=0x7fe9bc02a410 comp rx=0 tx=0).stop 2024-07-26T08:32:19.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 -- 172.21.15.44:0/797945641 shutdown_connections 2024-07-26T08:32:19.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 --2- 172.21.15.44:0/797945641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d4109650 0x7fe9d410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 --2- 172.21.15.44:0/797945641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4108d00 0x7fe9d4109110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 --2- 172.21.15.44:0/797945641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9d41033c0 0x7fe9d41037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.443+0000 7fe9da400640 1 -- 172.21.15.44:0/797945641 >> 172.21.15.44:0/797945641 conn(0x7fe9d40fee10 msgr2=0x7fe9d4101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:19.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.444+0000 7fe9da400640 1 -- 172.21.15.44:0/797945641 shutdown_connections 2024-07-26T08:32:19.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.444+0000 7fe9da400640 1 -- 172.21.15.44:0/797945641 wait complete. 2024-07-26T08:32:19.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.445+0000 7fe9da400640 1 Processor -- start 2024-07-26T08:32:19.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.445+0000 7fe9da400640 1 -- start start 2024-07-26T08:32:19.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9da400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 0x7fe9d4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:19.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9da400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9d4108d00 0x7fe9d4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:19.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9da400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4109650 0x7fe9d4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:19.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9da400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d4113310 con 0x7fe9d41033c0 2024-07-26T08:32:19.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9da400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d4113190 con 0x7fe9d4109650 2024-07-26T08:32:19.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9da400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d4113490 con 0x7fe9d4108d00 2024-07-26T08:32:19.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9d9000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 0x7fe9d4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:19.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9d9000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 0x7fe9d4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46808/0 (socket says 172.21.15.44:46808) 2024-07-26T08:32:19.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9d9000640 1 -- 172.21.15.44:0/4069720642 learned_addr learned my addr 172.21.15.44:0/4069720642 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:19.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9cbe00640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9d4108d00 0x7fe9d4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:19.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.446+0000 7fe9d9a00640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4109650 0x7fe9d4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:19.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.447+0000 7fe9d9000640 1 -- 172.21.15.44:0/4069720642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9d4108d00 msgr2=0x7fe9d4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:19.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.447+0000 7fe9d9000640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9d4108d00 0x7fe9d4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.447+0000 7fe9d9000640 1 -- 172.21.15.44:0/4069720642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4109650 msgr2=0x7fe9d4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:19.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.447+0000 7fe9d9000640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4109650 0x7fe9d4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.447+0000 7fe9d9000640 1 -- 172.21.15.44:0/4069720642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d41a7fc0 con 0x7fe9d41033c0 2024-07-26T08:32:19.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.447+0000 7fe9d9000640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 0x7fe9d4075140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400e9f0 tx=0x7fe9c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:19.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.448+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c400cd80 con 0x7fe9d41033c0 2024-07-26T08:32:19.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.448+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d41a82b0 con 0x7fe9d41033c0 2024-07-26T08:32:19.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.448+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9c40108e0 con 0x7fe9d41033c0 2024-07-26T08:32:19.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.448+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d41a87c0 con 0x7fe9d41033c0 2024-07-26T08:32:19.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.448+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4018920 con 0x7fe9d41033c0 2024-07-26T08:32:19.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.449+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d41037d0 con 0x7fe9d41033c0 2024-07-26T08:32:19.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.452+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fe9c4010430 con 0x7fe9d41033c0 2024-07-26T08:32:19.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.453+0000 7fe9d2000640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9b80b3c20 0x7fe9b80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:19.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.453+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7fe9c40e3e30 con 0x7fe9d41033c0 2024-07-26T08:32:19.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.454+0000 7fe9cbe00640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9b80b3c20 0x7fe9b80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:19.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.454+0000 7fe9cbe00640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9b80b3c20 0x7fe9b80b6130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc02aa00 tx=0x7fe9bc005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:19.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.455+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe9c40ab160 con 0x7fe9d41033c0 2024-07-26T08:32:19.507 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 systemd[1]: Starting Ceph mds.e for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:19.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.695+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9d402e0c0 con 0x7fe9d41033c0 2024-07-26T08:32:19.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.697+0000 7fe9d2000640 1 -- 172.21.15.44:0/4069720642 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3916 (secure 0 0 0) 0x7fe9c40af6b0 con 0x7fe9d41033c0 2024-07-26T08:32:19.698 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:19.699 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":5,"btime":"2024-07-26T08:32:17:929065+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-07-26T08:32:19.699 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 5 2024-07-26T08:32:19.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.699+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9b80b3c20 msgr2=0x7fe9b80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.699+0000 7fe9da400640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9b80b3c20 0x7fe9b80b6130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc02aa00 tx=0x7fe9bc005a20 comp rx=0 tx=0).stop 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 msgr2=0x7fe9d4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 0x7fe9d4075140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400e9f0 tx=0x7fe9c400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 shutdown_connections 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9b80b3c20 0x7fe9b80b6130 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9d4108d00 0x7fe9d4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9d4109650 0x7fe9d4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 --2- 172.21.15.44:0/4069720642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9d41033c0 0x7fe9d4075140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 >> 172.21.15.44:0/4069720642 conn(0x7fe9d40fee10 msgr2=0x7fe9d4101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:19.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 shutdown_connections 2024-07-26T08:32:19.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:19.700+0000 7fe9da400640 1 -- 172.21.15.44:0/4069720642 wait complete. 2024-07-26T08:32:19.767 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 podman[67598]: 2024-07-26 08:32:19.409108855 +0000 UTC m=+0.017396439 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:19.767 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 podman[67598]: 2024-07-26 08:32:19.522165032 +0000 UTC m=+0.130452601 container create 58f14acae1acd98e6e8b6221d45248c3a2d47dc02f547861ca4bec626ce6097d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-e, org.label-schema.build-date=20240724, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-07-26T08:32:19.767 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 podman[67598]: 2024-07-26 08:32:19.766498126 +0000 UTC m=+0.374785693 container init 58f14acae1acd98e6e8b6221d45248c3a2d47dc02f547861ca4bec626ce6097d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-26T08:32:20.022 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 podman[67598]: 2024-07-26 08:32:19.769105385 +0000 UTC m=+0.377392977 container start 58f14acae1acd98e6e8b6221d45248c3a2d47dc02f547861ca4bec626ce6097d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-e, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:32:20.022 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-e[67613]: starting mds.e at 2024-07-26T08:32:20.022 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 bash[67598]: 58f14acae1acd98e6e8b6221d45248c3a2d47dc02f547861ca4bec626ce6097d 2024-07-26T08:32:20.023 INFO:journalctl@ceph.mds.e.smithi116.stdout:Jul 26 08:32:19 smithi116 systemd[1]: Started Ceph mds.e for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4069720642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:20.451 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4069720642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:20.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:20.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4069720642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:20.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:20.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:20.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:20.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:20.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:21 smithi044 ceph-mon[32404]: Deploying daemon mds.f on smithi171 2024-07-26T08:32:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:32:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:21 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6834/1052813979,v1:172.21.15.116:6835/1052813979] up:boot 2024-07-26T08:32:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:21 smithi044 ceph-mon[32404]: fsmap 5 up:standby 2024-07-26T08:32:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:21 smithi116 ceph-mon[37097]: Deploying daemon mds.f on smithi171 2024-07-26T08:32:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:32:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:21 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6834/1052813979,v1:172.21.15.116:6835/1052813979] up:boot 2024-07-26T08:32:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:21 smithi116 ceph-mon[37097]: fsmap 5 up:standby 2024-07-26T08:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:21 smithi171 ceph-mon[37789]: Deploying daemon mds.f on smithi171 2024-07-26T08:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T08:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:21 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6834/1052813979,v1:172.21.15.116:6835/1052813979] up:boot 2024-07-26T08:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:21 smithi171 ceph-mon[37789]: fsmap 5 up:standby 2024-07-26T08:32:22.238 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 systemd[1]: Starting Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:22 smithi044 ceph-mon[32404]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:22 smithi116 ceph-mon[37097]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:22.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:22.554 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:22 smithi171 ceph-mon[37789]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:22.555 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 podman[69250]: 2024-07-26 08:32:22.218269682 +0000 UTC m=+0.017466524 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:22.555 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 podman[69250]: 2024-07-26 08:32:22.316834473 +0000 UTC m=+0.116031308 container create 22bb25534d60ea3db68a759b7cc8f9f0fc45366c95a7993c488501ac7d89bbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-07-26T08:32:22.555 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 podman[69250]: 2024-07-26 08:32:22.554422839 +0000 UTC m=+0.353619675 container init 22bb25534d60ea3db68a759b7cc8f9f0fc45366c95a7993c488501ac7d89bbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T08:32:22.836 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 podman[69250]: 2024-07-26 08:32:22.557737058 +0000 UTC m=+0.356933897 container start 22bb25534d60ea3db68a759b7cc8f9f0fc45366c95a7993c488501ac7d89bbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-07-26T08:32:22.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 ceph-mds[69269]: -- 172.21.15.171:0/1252368996 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557425495860 con 0x557429637a80 2024-07-26T08:32:22.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: starting mds.f at 2024-07-26T08:32:22.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 bash[69250]: 22bb25534d60ea3db68a759b7cc8f9f0fc45366c95a7993c488501ac7d89bbf9 2024-07-26T08:32:22.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:32:22 smithi171 systemd[1]: Started Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:23.010 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: Deploying daemon mds.c on smithi171 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:boot 2024-07-26T08:32:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:23 smithi044 ceph-mon[32404]: fsmap 6 up:standby 2024-07-26T08:32:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: Deploying daemon mds.c on smithi171 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:boot 2024-07-26T08:32:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:23 smithi116 ceph-mon[37097]: fsmap 6 up:standby 2024-07-26T08:32:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: Deploying daemon mds.c on smithi171 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:32:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:boot 2024-07-26T08:32:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:23 smithi171 ceph-mon[37789]: fsmap 6 up:standby 2024-07-26T08:32:24.837 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:24 smithi171 systemd[1]: Starting Ceph mds.c for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:25.257 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:24 smithi171 podman[69553]: 2024-07-26 08:32:24.869219181 +0000 UTC m=+0.017274271 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:25.257 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:24 smithi171 podman[69553]: 2024-07-26 08:32:24.995214472 +0000 UTC m=+0.143269561 container create 1ffdcba204787488fcc04f1b16c897f20bc861eb24f4d90efc2c7c823dc40d2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-26T08:32:25.257 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:25 smithi171 podman[69553]: 2024-07-26 08:32:25.256989625 +0000 UTC m=+0.405044720 container init 1ffdcba204787488fcc04f1b16c897f20bc861eb24f4d90efc2c7c823dc40d2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T08:32:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 -- 172.21.15.44:0/1824999206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 msgr2=0x7fcad810b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 --2- 172.21.15.44:0/1824999206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 0x7fcad810b340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009860 tx=0x7fcac402a340 comp rx=0 tx=0).stop 2024-07-26T08:32:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 -- 172.21.15.44:0/1824999206 shutdown_connections 2024-07-26T08:32:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 --2- 172.21.15.44:0/1824999206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 0x7fcad810b340 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 --2- 172.21.15.44:0/1824999206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 --2- 172.21.15.44:0/1824999206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 0x7fcad8105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 -- 172.21.15.44:0/1824999206 >> 172.21.15.44:0/1824999206 conn(0x7fcad81010a0 msgr2=0x7fcad81034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 -- 172.21.15.44:0/1824999206 shutdown_connections 2024-07-26T08:32:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.332+0000 7fcadca00640 1 -- 172.21.15.44:0/1824999206 wait complete. 2024-07-26T08:32:25.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.334+0000 7fcadca00640 1 Processor -- start 2024-07-26T08:32:25.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.334+0000 7fcadca00640 1 -- start start 2024-07-26T08:32:25.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.334+0000 7fcadca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 0x7fcad819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:25.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcadca00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 0x7fcad819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcadca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcadca00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8115510 con 0x7fcad81055f0 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcadca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8115390 con 0x7fcad810b880 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcadca00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcad8115690 con 0x7fcad810af30 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad6800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 0x7fcad819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34856/0 (socket says 172.21.15.44:34856) 2024-07-26T08:32:25.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad6800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 0x7fcad819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40742/0 (socket says 172.21.15.44:40742) 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad7200640 1 -- 172.21.15.44:0/3171479672 learned_addr learned my addr 172.21.15.44:0/3171479672 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad5e00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 0x7fcad819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad7200640 1 -- 172.21.15.44:0/3171479672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 msgr2=0x7fcad819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad7200640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 0x7fcad819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.335+0000 7fcad7200640 1 -- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 msgr2=0x7fcad819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcad7200640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 0x7fcad819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcad7200640 1 -- 172.21.15.44:0/3171479672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad81a1eb0 con 0x7fcad810b880 2024-07-26T08:32:25.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcad7200640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81a1780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009fd0 tx=0x7fcad000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:25.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad0004b20 con 0x7fcad810b880 2024-07-26T08:32:25.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcad0003e20 con 0x7fcad810b880 2024-07-26T08:32:25.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad81a21a0 con 0x7fcad810b880 2024-07-26T08:32:25.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad0004820 con 0x7fcad810b880 2024-07-26T08:32:25.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.336+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad81a8900 con 0x7fcad810b880 2024-07-26T08:32:25.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.337+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad8105a00 con 0x7fcad810b880 2024-07-26T08:32:25.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.340+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fcad001d020 con 0x7fcad810b880 2024-07-26T08:32:25.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.341+0000 7fcacf400640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcabc0b3c20 0x7fcabc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.342+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7fcad00e39f0 con 0x7fcad810b880 2024-07-26T08:32:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.342+0000 7fcad6800640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcabc0b3c20 0x7fcabc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:25.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.343+0000 7fcad6800640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcabc0b3c20 0x7fcabc0b6130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcac800bbe0 tx=0x7fcac800b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:25.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.343+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcad00ab370 con 0x7fcad810b880 2024-07-26T08:32:25.587 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:25 smithi171 podman[69553]: 2024-07-26 08:32:25.259720683 +0000 UTC m=+0.407775778 container start 1ffdcba204787488fcc04f1b16c897f20bc861eb24f4d90efc2c7c823dc40d2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-c, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T08:32:25.587 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:25 smithi171 ceph-mds[69572]: -- 172.21.15.171:0/2778903247 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f45093860 con 0x559f49235a80 2024-07-26T08:32:25.587 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:25 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-c[69568]: starting mds.c at 2024-07-26T08:32:25.587 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:25 smithi171 bash[69553]: 1ffdcba204787488fcc04f1b16c897f20bc861eb24f4d90efc2c7c823dc40d2b 2024-07-26T08:32:25.587 INFO:journalctl@ceph.mds.c.smithi171.stdout:Jul 26 08:32:25 smithi171 systemd[1]: Started Ceph mds.c for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:25.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.589+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcad810d630 con 0x7fcad810b880 2024-07-26T08:32:25.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.590+0000 7fcacf400640 1 -- 172.21.15.44:0/3171479672 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5569 (secure 0 0 0) 0x7fcad00aac70 con 0x7fcad810b880 2024-07-26T08:32:25.592 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:25.592 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"btime":"2024-07-26T08:32:23:236883+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24650,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24655,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-07-26T08:32:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcabc0b3c20 msgr2=0x7fcabc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcabc0b3c20 0x7fcabc0b6130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcac800bbe0 tx=0x7fcac800b390 comp rx=0 tx=0).stop 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 msgr2=0x7fcad81a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81a1780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009fd0 tx=0x7fcad000c800 comp rx=0 tx=0).stop 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 shutdown_connections 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcabc0b3c20 0x7fcabc0b6130 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcad810af30 0x7fcad819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcad810b880 0x7fcad81a1780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 --2- 172.21.15.44:0/3171479672 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcad81055f0 0x7fcad819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 >> 172.21.15.44:0/3171479672 conn(0x7fcad81010a0 msgr2=0x7fcad810a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.593+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 shutdown_connections 2024-07-26T08:32:25.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:25.594+0000 7fcadca00640 1 -- 172.21.15.44:0/3171479672 wait complete. 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:26.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: Deploying daemon mds.h on smithi116 2024-07-26T08:32:26.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3171479672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:26.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T08:32:26.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] up:boot 2024-07-26T08:32:26.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:26 smithi044 ceph-mon[32404]: fsmap 7 up:standby 2024-07-26T08:32:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: Deploying daemon mds.h on smithi116 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3171479672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] up:boot 2024-07-26T08:32:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:26 smithi116 ceph-mon[37097]: fsmap 7 up:standby 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: Deploying daemon mds.h on smithi116 2024-07-26T08:32:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3171479672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T08:32:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] up:boot 2024-07-26T08:32:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:26 smithi171 ceph-mon[37789]: fsmap 7 up:standby 2024-07-26T08:32:27.522 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 systemd[1]: Starting Ceph mds.h for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:27.899 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 podman[67902]: 2024-07-26 08:32:27.496540068 +0000 UTC m=+0.017392053 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:27.900 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 podman[67902]: 2024-07-26 08:32:27.62958826 +0000 UTC m=+0.150440246 container create 39f632c200849c07f25d48435cb972aa8ba413e79682543d470746ba99d91981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-h, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T08:32:28.272 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 podman[67902]: 2024-07-26 08:32:27.899311476 +0000 UTC m=+0.420163475 container init 39f632c200849c07f25d48435cb972aa8ba413e79682543d470746ba99d91981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-07-26T08:32:28.272 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 podman[67902]: 2024-07-26 08:32:27.901877512 +0000 UTC m=+0.422729498 container start 39f632c200849c07f25d48435cb972aa8ba413e79682543d470746ba99d91981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-26T08:32:28.273 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 ceph-mds[67921]: -- 172.21.15.116:0/3919771700 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648b0bb9860 con 0x5648b4d5ba80 2024-07-26T08:32:28.273 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:27 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-h[67917]: starting mds.h at 2024-07-26T08:32:28.273 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:28 smithi116 bash[67902]: 39f632c200849c07f25d48435cb972aa8ba413e79682543d470746ba99d91981 2024-07-26T08:32:28.273 INFO:journalctl@ceph.mds.h.smithi116.stdout:Jul 26 08:32:28 smithi116 systemd[1]: Started Ceph mds.h for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:28.311 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:28.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:28.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:28.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:28.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:28.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:28.803 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:29 smithi171 ceph-mon[37789]: Deploying daemon mds.i on smithi171 2024-07-26T08:32:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T08:32:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:29 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] up:boot 2024-07-26T08:32:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:29 smithi171 ceph-mon[37789]: fsmap 8 up:standby 2024-07-26T08:32:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:29 smithi044 ceph-mon[32404]: Deploying daemon mds.i on smithi171 2024-07-26T08:32:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T08:32:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:29 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] up:boot 2024-07-26T08:32:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:29 smithi044 ceph-mon[32404]: fsmap 8 up:standby 2024-07-26T08:32:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:29 smithi116 ceph-mon[37097]: Deploying daemon mds.i on smithi171 2024-07-26T08:32:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T08:32:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:29 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] up:boot 2024-07-26T08:32:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:29 smithi116 ceph-mon[37097]: fsmap 8 up:standby 2024-07-26T08:32:30.337 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 systemd[1]: Starting Ceph mds.i for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:30.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:30 smithi044 ceph-mon[32404]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:30.653 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:30 smithi171 ceph-mon[37789]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:30.653 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 podman[69859]: 2024-07-26 08:32:30.260953459 +0000 UTC m=+0.017350245 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:30.653 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 podman[69859]: 2024-07-26 08:32:30.398775335 +0000 UTC m=+0.155172112 container create a8ae39647f3a7fcbe76da665172ee556acfdd81e8fe3551e9f6676f84391c8e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-07-26T08:32:30.653 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 podman[69859]: 2024-07-26 08:32:30.652803429 +0000 UTC m=+0.409200207 container init a8ae39647f3a7fcbe76da665172ee556acfdd81e8fe3551e9f6676f84391c8e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724) 2024-07-26T08:32:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:30 smithi116 ceph-mon[37097]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:31.087 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 podman[69859]: 2024-07-26 08:32:30.65545552 +0000 UTC m=+0.411852293 container start a8ae39647f3a7fcbe76da665172ee556acfdd81e8fe3551e9f6676f84391c8e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:32:31.087 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 ceph-mds[69878]: -- 172.21.15.171:0/3793125223 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc94bc9860 con 0x55cc98d6ba80 2024-07-26T08:32:31.087 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-i[69874]: starting mds.i at 2024-07-26T08:32:31.087 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 bash[69859]: a8ae39647f3a7fcbe76da665172ee556acfdd81e8fe3551e9f6676f84391c8e8 2024-07-26T08:32:31.087 INFO:journalctl@ceph.mds.i.smithi171.stdout:Jul 26 08:32:30 smithi171 systemd[1]: Started Ceph mds.i for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1051386238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e4101e70 msgr2=0x7fb7e4102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1051386238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e4101e70 0x7fb7e4102280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009900 tx=0x7fb7d802a480 comp rx=0 tx=0).stop 2024-07-26T08:32:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1051386238 shutdown_connections 2024-07-26T08:32:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1051386238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e410fa20 0x7fb7e4111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1051386238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e41027c0 0x7fb7e410f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1051386238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e4101e70 0x7fb7e4102280 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.176+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1051386238 >> 172.21.15.44:0/1051386238 conn(0x7fb7e40fdcf0 msgr2=0x7fb7e4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.177+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1051386238 shutdown_connections 2024-07-26T08:32:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.177+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1051386238 wait complete. 2024-07-26T08:32:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.178+0000 7fb7eaa00640 1 Processor -- start 2024-07-26T08:32:31.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.178+0000 7fb7eaa00640 1 -- start start 2024-07-26T08:32:31.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7eaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 0x7fb7e419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:31.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7eaa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e41027c0 0x7fb7e419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:31.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7eaa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e410fa20 0x7fb7e41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:31.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7eaa00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e4115470 con 0x7fb7e4101e70 2024-07-26T08:32:31.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7eaa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41152f0 con 0x7fb7e410fa20 2024-07-26T08:32:31.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7eaa00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41155f0 con 0x7fb7e41027c0 2024-07-26T08:32:31.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7e9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 0x7fb7e419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7e8c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e41027c0 0x7fb7e419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7e9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 0x7fb7e419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40766/0 (socket says 172.21.15.44:40766) 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7e9600640 1 -- 172.21.15.44:0/1677902111 learned_addr learned my addr 172.21.15.44:0/1677902111 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7ea000640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e410fa20 0x7fb7e41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7e8c00640 1 -- 172.21.15.44:0/1677902111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e410fa20 msgr2=0x7fb7e41a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.179+0000 7fb7e8c00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e410fa20 0x7fb7e41a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e8c00640 1 -- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 msgr2=0x7fb7e419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e8c00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 0x7fb7e419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e8c00640 1 -- 172.21.15.44:0/1677902111 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e41a1e10 con 0x7fb7e41027c0 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e9600640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 0x7fb7e419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:32:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e8c00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e41027c0 0x7fb7e419c440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400e9f0 tx=0x7fb7d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d400cd80 con 0x7fb7e41027c0 2024-07-26T08:32:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d40035f0 con 0x7fb7e41027c0 2024-07-26T08:32:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e41a2100 con 0x7fb7e41027c0 2024-07-26T08:32:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e41a8860 con 0x7fb7e41027c0 2024-07-26T08:32:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.180+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4010670 con 0x7fb7e41027c0 2024-07-26T08:32:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.182+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e4102280 con 0x7fb7e41027c0 2024-07-26T08:32:31.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.184+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fb7d4004020 con 0x7fb7e41027c0 2024-07-26T08:32:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.185+0000 7fb7e2000640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7cc0b3ca0 0x7fb7cc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:31.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.185+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7fb7d40e3b20 con 0x7fb7e41027c0 2024-07-26T08:32:31.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.186+0000 7fb7e9600640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7cc0b3ca0 0x7fb7cc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:31.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.186+0000 7fb7e9600640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7cc0b3ca0 0x7fb7cc0b61b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8031440 tx=0x7fb7d80058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:31.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.189+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7d40ab4a0 con 0x7fb7e41027c0 2024-07-26T08:32:31.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.451+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7e410abb0 con 0x7fb7e41027c0 2024-07-26T08:32:31.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.457+0000 7fb7e2000640 1 -- 172.21.15.44:0/1677902111 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8055 (secure 0 0 0) 0x7fb7d40aada0 con 0x7fb7e41027c0 2024-07-26T08:32:31.458 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 10 2024-07-26T08:32:31.459 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:31.459 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":10,"btime":"2024-07-26T08:32:31:446026+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24650,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24655,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24668,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24679,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-26T08:32:31.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7cc0b3ca0 msgr2=0x7fb7cc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:31.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7cc0b3ca0 0x7fb7cc0b61b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8031440 tx=0x7fb7d80058d0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e41027c0 msgr2=0x7fb7e419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:31.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e41027c0 0x7fb7e419c440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400e9f0 tx=0x7fb7d400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 shutdown_connections 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7cc0b3ca0 0x7fb7cc0b61b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7e41027c0 0x7fb7e419c440 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7e410fa20 0x7fb7e41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 --2- 172.21.15.44:0/1677902111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7e4101e70 0x7fb7e419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 >> 172.21.15.44:0/1677902111 conn(0x7fb7e40fdcf0 msgr2=0x7fb7e4111bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 shutdown_connections 2024-07-26T08:32:31.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:31.460+0000 7fb7eaa00640 1 -- 172.21.15.44:0/1677902111 wait complete. 2024-07-26T08:32:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: Deploying daemon mds.g on smithi044 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1677902111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] up:boot 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T08:32:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:31 smithi044 ceph-mon[32404]: fsmap 9 up:standby 2024-07-26T08:32:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: Deploying daemon mds.g on smithi044 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1677902111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] up:boot 2024-07-26T08:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T08:32:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:31 smithi171 ceph-mon[37789]: fsmap 9 up:standby 2024-07-26T08:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: Deploying daemon mds.g on smithi044 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1677902111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] up:boot 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T08:32:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:31 smithi116 ceph-mon[37097]: fsmap 9 up:standby 2024-07-26T08:32:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:33 smithi044 ceph-mon[32404]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:34.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:33 smithi116 ceph-mon[37097]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:33 smithi171 ceph-mon[37789]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:34.722 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:34 smithi044 systemd[1]: Starting Ceph mds.g for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:35.016 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:34 smithi044 podman[79181]: 2024-07-26 08:32:34.625526925 +0000 UTC m=+0.034228405 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:35.017 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:34 smithi044 podman[79181]: 2024-07-26 08:32:34.769065746 +0000 UTC m=+0.177767259 container create d9bdde87056b6d28f567d00bd3062f3c880bdfdadf51cbc4d4af8282355e94d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:32:35.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:35.507 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:35.515 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:35 smithi044 podman[79181]: 2024-07-26 08:32:35.039718022 +0000 UTC m=+0.448419538 container init d9bdde87056b6d28f567d00bd3062f3c880bdfdadf51cbc4d4af8282355e94d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD) 2024-07-26T08:32:35.515 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:35 smithi044 podman[79181]: 2024-07-26 08:32:35.053760809 +0000 UTC m=+0.462462351 container start d9bdde87056b6d28f567d00bd3062f3c880bdfdadf51cbc4d4af8282355e94d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724) 2024-07-26T08:32:35.516 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:35 smithi044 bash[79181]: d9bdde87056b6d28f567d00bd3062f3c880bdfdadf51cbc4d4af8282355e94d5 2024-07-26T08:32:35.516 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:35 smithi044 systemd[1]: Started Ceph mds.g for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:35.516 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:35 smithi044 ceph-mds[79205]: -- 172.21.15.44:0/1692154968 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4c0679860 con 0x55c4c481b180 2024-07-26T08:32:35.516 INFO:journalctl@ceph.mds.g.smithi044.stdout:Jul 26 08:32:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-g[79196]: starting mds.g at 2024-07-26T08:32:36.504 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: Deploying daemon mds.a on smithi044 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:boot 2024-07-26T08:32:36.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:36 smithi044 ceph-mon[32404]: fsmap 10 up:standby 2024-07-26T08:32:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: Deploying daemon mds.a on smithi044 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:boot 2024-07-26T08:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:36 smithi116 ceph-mon[37097]: fsmap 10 up:standby 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: Deploying daemon mds.a on smithi044 2024-07-26T08:32:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T08:32:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:boot 2024-07-26T08:32:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:36 smithi171 ceph-mon[37789]: fsmap 10 up:standby 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.180+0000 7f345e400640 1 -- 172.21.15.44:0/2893595823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 msgr2=0x7f3458112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.180+0000 7f345e400640 1 --2- 172.21.15.44:0/2893595823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f3458112090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3450009ca0 tx=0x7f345002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 -- 172.21.15.44:0/2893595823 shutdown_connections 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 --2- 172.21.15.44:0/2893595823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f3458112090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 --2- 172.21.15.44:0/2893595823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f345810af10 0x7f345810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 --2- 172.21.15.44:0/2893595823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34581055d0 0x7f34581059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 -- 172.21.15.44:0/2893595823 >> 172.21.15.44:0/2893595823 conn(0x7f3458101040 msgr2=0x7f3458103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:38.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 -- 172.21.15.44:0/2893595823 shutdown_connections 2024-07-26T08:32:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.181+0000 7f345e400640 1 -- 172.21.15.44:0/2893595823 wait complete. 2024-07-26T08:32:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.182+0000 7f345e400640 1 Processor -- start 2024-07-26T08:32:38.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 -- start start 2024-07-26T08:32:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34581055d0 0x7f345819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 0x7f345819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f34581a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3458115510 con 0x7f345810b860 2024-07-26T08:32:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3458115390 con 0x7f34581055d0 2024-07-26T08:32:38.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.183+0000 7f345e400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3458115690 con 0x7f345810af10 2024-07-26T08:32:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 0x7f345819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:38.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 0x7f345819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:59670/0 (socket says 172.21.15.44:59670) 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 -- 172.21.15.44:0/3177051331 learned_addr learned my addr 172.21.15.44:0/3177051331 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f345da00640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f34581a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 -- 172.21.15.44:0/3177051331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34581055d0 msgr2=0x7f345819bf20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34581055d0 0x7f345819bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 -- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 msgr2=0x7f34581a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f34581a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f3457e00640 1 -- 172.21.15.44:0/3177051331 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34581a1e90 con 0x7f345810af10 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.184+0000 7f345da00640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f34581a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.185+0000 7f3457e00640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 0x7f345819c4c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f344400e9f0 tx=0x7f344400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.185+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344400ccc0 con 0x7f345810af10 2024-07-26T08:32:38.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.185+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34440108e0 con 0x7f345810af10 2024-07-26T08:32:38.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.185+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34581a2180 con 0x7f345810af10 2024-07-26T08:32:38.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.185+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444018860 con 0x7f345810af10 2024-07-26T08:32:38.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.185+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34581a88e0 con 0x7f345810af10 2024-07-26T08:32:38.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.187+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34581059e0 con 0x7f345810af10 2024-07-26T08:32:38.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.189+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f3444010430 con 0x7f345810af10 2024-07-26T08:32:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.191+0000 7f3455600640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f34400b3ca0 0x7f34400b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.191+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f34440e3bf0 con 0x7f345810af10 2024-07-26T08:32:38.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.191+0000 7f345d000640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f34400b3ca0 0x7f34400b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:38.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.192+0000 7f345d000640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f34400b3ca0 0x7f34400b61b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f344c009a60 tx=0x7f344c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:38.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.195+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f34440ab570 con 0x7f345810af10 2024-07-26T08:32:38.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.438+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f345810d6b0 con 0x7f345810af10 2024-07-26T08:32:38.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.439+0000 7f3455600640 1 -- 172.21.15.44:0/3177051331 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8881 (secure 0 0 0) 0x7f34440aae70 con 0x7f345810af10 2024-07-26T08:32:38.441 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:38.441 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":11,"btime":"2024-07-26T08:32:35:818309+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24650,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24655,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24668,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24679,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24686,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-07-26T08:32:38.442 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 11 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f34400b3ca0 msgr2=0x7f34400b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f34400b3ca0 0x7f34400b61b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f344c009a60 tx=0x7f344c005f90 comp rx=0 tx=0).stop 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 msgr2=0x7f345819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 0x7f345819c4c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f344400e9f0 tx=0x7f344400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 shutdown_connections 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f34400b3ca0 0x7f34400b61b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f345810af10 0x7f345819c4c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f345810b860 0x7f34581a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 --2- 172.21.15.44:0/3177051331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34581055d0 0x7f345819bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:38.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.442+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 >> 172.21.15.44:0/3177051331 conn(0x7f3458101040 msgr2=0x7f3458109550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:38.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.443+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 shutdown_connections 2024-07-26T08:32:38.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:38.443+0000 7f345e400640 1 -- 172.21.15.44:0/3177051331 wait complete. 2024-07-26T08:32:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:38 smithi044 ceph-mon[32404]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:38 smithi116 ceph-mon[37097]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:38 smithi171 ceph-mon[37789]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:39 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3177051331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:39 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3177051331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:39 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3177051331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:40.175 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:39 smithi044 systemd[1]: Starting Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:40 smithi044 ceph-mon[32404]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:40.516 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 podman[79637]: 2024-07-26 08:32:40.07691245 +0000 UTC m=+0.033134660 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:40.516 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 podman[79637]: 2024-07-26 08:32:40.260170601 +0000 UTC m=+0.216392854 container create e06cea071add4b91342a9670f51f33717549b0f6b33062a248629211642e9dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:32:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:40 smithi116 ceph-mon[37097]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:40 smithi171 ceph-mon[37789]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:41.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 podman[79637]: 2024-07-26 08:32:40.531299007 +0000 UTC m=+0.487521249 container init e06cea071add4b91342a9670f51f33717549b0f6b33062a248629211642e9dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T08:32:41.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 podman[79637]: 2024-07-26 08:32:40.542065131 +0000 UTC m=+0.498287349 container start e06cea071add4b91342a9670f51f33717549b0f6b33062a248629211642e9dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T08:32:41.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 bash[79637]: e06cea071add4b91342a9670f51f33717549b0f6b33062a248629211642e9dc6 2024-07-26T08:32:41.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 systemd[1]: Started Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:41.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 ceph-mds[79656]: -- 172.21.15.44:0/1915314559 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b366ffd860 con 0x55b36b19f600 2024-07-26T08:32:41.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:32:40 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: starting mds.a at 2024-07-26T08:32:41.461 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:32:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: Deploying daemon mds.b on smithi116 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:boot 2024-07-26T08:32:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:41 smithi116 ceph-mon[37097]: fsmap 11 up:standby 2024-07-26T08:32:41.981 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: Deploying daemon mds.b on smithi116 2024-07-26T08:32:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:32:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:boot 2024-07-26T08:32:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:41 smithi044 ceph-mon[32404]: fsmap 11 up:standby 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: Deploying daemon mds.b on smithi116 2024-07-26T08:32:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:32:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:boot 2024-07-26T08:32:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:41 smithi171 ceph-mon[37789]: fsmap 11 up:standby 2024-07-26T08:32:42.939 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:42 smithi116 systemd[1]: Starting Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:32:43.198 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:42 smithi116 podman[68215]: 2024-07-26 08:32:42.841407772 +0000 UTC m=+0.017801251 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:32:43.198 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:42 smithi116 podman[68215]: 2024-07-26 08:32:42.95359583 +0000 UTC m=+0.129989300 container create bad20a32fc44c967ac13339e65d4d1c81ca77772ec2afa5550a4802bb4949513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-07-26T08:32:43.199 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:43 smithi116 podman[68215]: 2024-07-26 08:32:43.198057751 +0000 UTC m=+0.374451241 container init bad20a32fc44c967ac13339e65d4d1c81ca77772ec2afa5550a4802bb4949513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T08:32:43.522 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:43 smithi116 podman[68215]: 2024-07-26 08:32:43.200763396 +0000 UTC m=+0.377156876 container start bad20a32fc44c967ac13339e65d4d1c81ca77772ec2afa5550a4802bb4949513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T08:32:43.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:43 smithi116 ceph-mds[68235]: -- 172.21.15.116:0/1587129730 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b08eb1d860 con 0x55b092cbfa80 2024-07-26T08:32:43.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:43 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: starting mds.b at 2024-07-26T08:32:43.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:43 smithi116 bash[68215]: bad20a32fc44c967ac13339e65d4d1c81ca77772ec2afa5550a4802bb4949513 2024-07-26T08:32:43.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:32:43 smithi116 systemd[1]: Started Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:32:44.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 -- 172.21.15.44:0/1232558298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc108f30 msgr2=0x7f3dfc109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:44.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 --2- 172.21.15.44:0/1232558298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc108f30 0x7f3dfc109340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009860 tx=0x7f3de402a340 comp rx=0 tx=0).stop 2024-07-26T08:32:44.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 -- 172.21.15.44:0/1232558298 shutdown_connections 2024-07-26T08:32:44.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 --2- 172.21.15.44:0/1232558298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc108f30 0x7f3dfc109340 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:44.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 --2- 172.21.15.44:0/1232558298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc109880 0x7f3dfc1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:44.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 --2- 172.21.15.44:0/1232558298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:44.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 -- 172.21.15.44:0/1232558298 >> 172.21.15.44:0/1232558298 conn(0x7f3dfc0799c0 msgr2=0x7f3dfc079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:44.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 -- 172.21.15.44:0/1232558298 shutdown_connections 2024-07-26T08:32:44.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.285+0000 7f3e01400640 1 -- 172.21.15.44:0/1232558298 wait complete. 2024-07-26T08:32:44.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.286+0000 7f3e01400640 1 Processor -- start 2024-07-26T08:32:44.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 -- start start 2024-07-26T08:32:44.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc19c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:44.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc108f30 0x7f3dfc19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:44.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc109880 0x7f3dfc1a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:44.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc1134f0 con 0x7f3dfc108f30 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc113370 con 0x7f3dfc1019f0 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3e01400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc113670 con 0x7f3dfc109880 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3dfa800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc19c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3dfa800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc19c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:38104/0 (socket says 172.21.15.44:38104) 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3dfa800640 1 -- 172.21.15.44:0/801529241 learned_addr learned my addr 172.21.15.44:0/801529241 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.287+0000 7f3df9e00640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc108f30 0x7f3dfc19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfb200640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc109880 0x7f3dfc1a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfa800640 1 -- 172.21.15.44:0/801529241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc109880 msgr2=0x7f3dfc1a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfa800640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc109880 0x7f3dfc1a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfa800640 1 -- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc108f30 msgr2=0x7f3dfc19c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfa800640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc108f30 0x7f3dfc19c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfa800640 1 -- 172.21.15.44:0/801529241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc1a2090 con 0x7f3dfc1019f0 2024-07-26T08:32:44.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfb200640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc109880 0x7f3dfc1a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] up:boot 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: fsmap 12 up:standby 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:45 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/801529241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:46 smithi171 ceph-mon[37789]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] up:boot 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: fsmap 12 up:standby 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:45 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/801529241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:46 smithi116 ceph-mon[37097]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:49.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:49.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.288+0000 7f3dfa800640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc19c120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00e9c0 tx=0x7f3dec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:49.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.289+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00ccc0 con 0x7f3dfc1019f0 2024-07-26T08:32:49.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.289+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dec0054d0 con 0x7f3dfc1019f0 2024-07-26T08:32:49.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.289+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dfc1a2380 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.289+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec010430 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.289+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dfc1a8ae0 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.291+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dfc101e00 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.293+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f3dec005020 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.294+0000 7f3df3400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3de00b3ca0 0x7f3de00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.294+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f3dec0e3d30 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.294+0000 7f3df9e00640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3de00b3ca0 0x7f3de00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.295+0000 7f3df9e00640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3de00b3ca0 0x7f3de00b61b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3de4035040 tx=0x7f3de4030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.298+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3dec0ab6b0 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.557+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dfc1a8d40 con 0x7f3dfc1019f0 2024-07-26T08:32:49.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.558+0000 7f3df3400640 1 -- 172.21.15.44:0/801529241 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10536 (secure 0 0 0) 0x7f3dec0aafb0 con 0x7f3dfc1019f0 2024-07-26T08:32:49.348 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:49.348 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"btime":"2024-07-26T08:32:43:738948+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24650,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24655,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24668,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24679,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24686,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24701,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3de00b3ca0 msgr2=0x7f3de00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3de00b3ca0 0x7f3de00b61b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3de4035040 tx=0x7f3de4030340 comp rx=0 tx=0).stop 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 msgr2=0x7f3dfc19c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc19c120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00e9c0 tx=0x7f3dec00ee80 comp rx=0 tx=0).stop 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 shutdown_connections 2024-07-26T08:32:49.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3de00b3ca0 0x7f3de00b61b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:49.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3dfc109880 0x7f3dfc1a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:49.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3dfc108f30 0x7f3dfc19c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:49.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 --2- 172.21.15.44:0/801529241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dfc1019f0 0x7f3dfc19c120 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:49.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 >> 172.21.15.44:0/801529241 conn(0x7f3dfc0799c0 msgr2=0x7f3dfc106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:49.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 shutdown_connections 2024-07-26T08:32:49.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:44.561+0000 7f3e01400640 1 -- 172.21.15.44:0/801529241 wait complete. 2024-07-26T08:32:49.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] up:boot 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: fsmap 12 up:standby 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:45 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/801529241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.353 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:46 smithi044 ceph-mon[32404]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:49.355 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:49.819 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:50.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:49 smithi044 ceph-mon[32404]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:50.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:49 smithi116 ceph-mon[37097]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:49 smithi171 ceph-mon[37789]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.066+0000 7faeeee00640 1 -- 172.21.15.44:0/1792687984 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 msgr2=0x7faee810f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.066+0000 7faeeee00640 1 --2- 172.21.15.44:0/1792687984 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee810f680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faed4009860 tx=0x7faed402a340 comp rx=0 tx=0).stop 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 -- 172.21.15.44:0/1792687984 shutdown_connections 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 --2- 172.21.15.44:0/1792687984 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee810f680 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 --2- 172.21.15.44:0/1792687984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee810fbc0 0x7faee8111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 --2- 172.21.15.44:0/1792687984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee8075160 0x7faee8075570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 -- 172.21.15.44:0/1792687984 >> 172.21.15.44:0/1792687984 conn(0x7faee80ff030 msgr2=0x7faee8101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 -- 172.21.15.44:0/1792687984 shutdown_connections 2024-07-26T08:32:52.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.067+0000 7faeeee00640 1 -- 172.21.15.44:0/1792687984 wait complete. 2024-07-26T08:32:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.069+0000 7faeeee00640 1 Processor -- start 2024-07-26T08:32:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.069+0000 7faeeee00640 1 -- start start 2024-07-26T08:32:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeeee00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee8075160 0x7faee819c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeeee00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeeee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee810fbc0 0x7faee81a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeeee00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faee81156d0 con 0x7faee8075160 2024-07-26T08:32:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeeee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faee8115550 con 0x7faee810fbc0 2024-07-26T08:32:52.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeeee00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faee8115850 con 0x7faee8075ab0 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faee7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee8075160 0x7faee819c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faee7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee8075160 0x7faee819c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49136/0 (socket says 172.21.15.44:49136) 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faee7e00640 1 -- 172.21.15.44:0/931634311 learned_addr learned my addr 172.21.15.44:0/931634311 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faee7400640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.070+0000 7faeece00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee810fbc0 0x7faee81a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee7400640 1 -- 172.21.15.44:0/931634311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee810fbc0 msgr2=0x7faee81a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee7400640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee810fbc0 0x7faee81a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee7400640 1 -- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee8075160 msgr2=0x7faee819c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee7400640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee8075160 0x7faee819c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee7400640 1 -- 172.21.15.44:0/931634311 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee81a20b0 con 0x7faee8075ab0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faeece00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee810fbc0 0x7faee81a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee7400640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee819c6e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faed4009780 tx=0x7faed4032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4037040 con 0x7faee8075ab0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faed4030c10 con 0x7faee8075ab0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee81a2340 con 0x7faee8075ab0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.071+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed40035f0 con 0x7faee8075ab0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.072+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee81a8b00 con 0x7faee8075ab0 2024-07-26T08:32:52.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.073+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee8075570 con 0x7faee8075ab0 2024-07-26T08:32:52.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.076+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7faed4043020 con 0x7faee8075ab0 2024-07-26T08:32:52.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.077+0000 7faee4c00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7faecc0b3ca0 0x7faecc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:52.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.077+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7faed4102e10 con 0x7faee8075ab0 2024-07-26T08:32:52.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.077+0000 7faee7e00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7faecc0b3ca0 0x7faecc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.078+0000 7faee7e00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7faecc0b3ca0 0x7faecc0b61b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faed8009fd0 tx=0x7faed8009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:52.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.078+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faed404a060 con 0x7faee8075ab0 2024-07-26T08:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:52 smithi044 ceph-mon[32404]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.282+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7faee81a8cd0 con 0x7faee8075ab0 2024-07-26T08:32:52.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.283+0000 7faee4c00640 1 -- 172.21.15.44:0/931634311 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) ==== 84+0+65 (secure 0 0 0) 0x7faed40ca890 con 0x7faee8075ab0 2024-07-26T08:32:52.285 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:32:52.285 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-07-26T08:32:52.286 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-07-26T08:32:52.286 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-07-26T08:32:52.286 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-07-26T08:32:52.286 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-07-26T08:32:52.286 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-07-26T08:32:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.285+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7faecc0b3ca0 msgr2=0x7faecc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7faecc0b3ca0 0x7faecc0b61b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faed8009fd0 tx=0x7faed8009500 comp rx=0 tx=0).stop 2024-07-26T08:32:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 msgr2=0x7faee819c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee819c6e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faed4009780 tx=0x7faed4032d50 comp rx=0 tx=0).stop 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 shutdown_connections 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7faecc0b3ca0 0x7faecc0b61b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faee8075ab0 0x7faee819c6e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee810fbc0 0x7faee81a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 --2- 172.21.15.44:0/931634311 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faee8075160 0x7faee819c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 >> 172.21.15.44:0/931634311 conn(0x7faee80ff030 msgr2=0x7faee8108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 shutdown_connections 2024-07-26T08:32:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:52.286+0000 7faeeee00640 1 -- 172.21.15.44:0/931634311 wait complete. 2024-07-26T08:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:52 smithi171 ceph-mon[37789]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:52 smithi116 ceph-mon[37097]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:32:53.249 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T08:32:53.249 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T08:32:53.323 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:53 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/931634311' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T08:32:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:53 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/931634311' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T08:32:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:53 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/931634311' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T08:32:53.718 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:54.233 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:54 smithi044 ceph-mon[32404]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:54 smithi171 ceph-mon[37789]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:54 smithi116 ceph-mon[37097]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 -- 172.21.15.44:0/1698578996 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077370 msgr2=0x7f0ad8077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 --2- 172.21.15.44:0/1698578996 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077370 0x7f0ad8077780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009860 tx=0x7f0ac802a340 comp rx=0 tx=0).stop 2024-07-26T08:32:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 -- 172.21.15.44:0/1698578996 shutdown_connections 2024-07-26T08:32:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 --2- 172.21.15.44:0/1698578996 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad81141a0 0x7f0ad81165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 --2- 172.21.15.44:0/1698578996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8077cc0 0x7f0ad8113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 --2- 172.21.15.44:0/1698578996 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077370 0x7f0ad8077780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:55.999+0000 7f0ade600640 1 -- 172.21.15.44:0/1698578996 >> 172.21.15.44:0/1698578996 conn(0x7f0ad81011f0 msgr2=0x7f0ad8103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.000+0000 7f0ade600640 1 -- 172.21.15.44:0/1698578996 shutdown_connections 2024-07-26T08:32:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.000+0000 7f0ade600640 1 -- 172.21.15.44:0/1698578996 wait complete. 2024-07-26T08:32:56.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.001+0000 7f0ade600640 1 Processor -- start 2024-07-26T08:32:56.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.001+0000 7f0ade600640 1 -- start start 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ade600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad8077370 0x7f0ad81a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ade600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077cc0 0x7f0ad81a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ade600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad81141a0 0x7f0ad81a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ade600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad8119b00 con 0x7f0ad8077370 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ade600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad8119980 con 0x7f0ad81141a0 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ade600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad8119c80 con 0x7f0ad8077cc0 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ad7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad8077370 0x7f0ad81a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0acea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077cc0 0x7f0ad81a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ad7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad8077370 0x7f0ad81a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49158/0 (socket says 172.21.15.44:49158) 2024-07-26T08:32:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.002+0000 7f0ad7400640 1 -- 172.21.15.44:0/552029571 learned_addr learned my addr 172.21.15.44:0/552029571 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:32:56.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0ad7e00640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad81141a0 0x7f0ad81a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:56.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0acea00640 1 -- 172.21.15.44:0/552029571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad81141a0 msgr2=0x7f0ad81a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:56.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0acea00640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad81141a0 0x7f0ad81a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0acea00640 1 -- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad8077370 msgr2=0x7f0ad81a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0acea00640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad8077370 0x7f0ad81a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0acea00640 1 -- 172.21.15.44:0/552029571 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad81a6460 con 0x7f0ad8077cc0 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0ad7e00640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad81141a0 0x7f0ad81a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.003+0000 7f0acea00640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077cc0 0x7f0ad81a0a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ad806c3a0 tx=0x7f0ac000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.004+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac000cc40 con 0x7f0ad8077cc0 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.004+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac000cde0 con 0x7f0ad8077cc0 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.004+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0018780 con 0x7f0ad8077cc0 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.004+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad81a6750 con 0x7f0ad8077cc0 2024-07-26T08:32:56.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.004+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad81aceb0 con 0x7f0ad8077cc0 2024-07-26T08:32:56.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.005+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad8078880 con 0x7f0ad8077cc0 2024-07-26T08:32:56.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.008+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f0ac0010a90 con 0x7f0ad8077cc0 2024-07-26T08:32:56.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.009+0000 7f0ad4c00640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0abc0b3c20 0x7f0abc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:32:56.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.010+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7f0ac00e3fe0 con 0x7f0ad8077cc0 2024-07-26T08:32:56.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.010+0000 7f0ad7400640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0abc0b3c20 0x7f0abc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:32:56.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.011+0000 7f0ad7400640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0abc0b3c20 0x7f0abc0b6130 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80095f0 tx=0x7f0ac8030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:32:56.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.011+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0ac00ab370 con 0x7f0ad8077cc0 2024-07-26T08:32:56.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:56.213+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0ad8077780 con 0x7f0ad8077cc0 2024-07-26T08:32:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:56 smithi171 ceph-mon[37789]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:56 smithi044 ceph-mon[32404]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:56 smithi116 ceph-mon[37097]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:57.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.067+0000 7f0ad4c00640 1 -- 172.21.15.44:0/552029571 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) ==== 138+0+0 (secure 0 0 0) 0x7f0ad8077780 con 0x7f0ad8077cc0 2024-07-26T08:32:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-07-26T08:32:57.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.074+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0abc0b3c20 msgr2=0x7f0abc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:57.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.074+0000 7f0ade600640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0abc0b3c20 0x7f0abc0b6130 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80095f0 tx=0x7f0ac8030340 comp rx=0 tx=0).stop 2024-07-26T08:32:57.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.074+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077cc0 msgr2=0x7f0ad81a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:32:57.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.074+0000 7f0ade600640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077cc0 0x7f0ad81a0a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ad806c3a0 tx=0x7f0ac000ee50 comp rx=0 tx=0).stop 2024-07-26T08:32:57.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.075+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 shutdown_connections 2024-07-26T08:32:57.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.075+0000 7f0ade600640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0abc0b3c20 0x7f0abc0b6130 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:57.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.075+0000 7f0ade600640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad8077cc0 0x7f0ad81a0a90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:57.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.075+0000 7f0ade600640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad81141a0 0x7f0ad81a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:57.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.075+0000 7f0ade600640 1 --2- 172.21.15.44:0/552029571 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0ad8077370 0x7f0ad81a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:32:57.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.075+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 >> 172.21.15.44:0/552029571 conn(0x7f0ad81011f0 msgr2=0x7f0ad8103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:32:57.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.076+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 shutdown_connections 2024-07-26T08:32:57.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:32:57.076+0000 7f0ade600640 1 -- 172.21.15.44:0/552029571 wait complete. 2024-07-26T08:32:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:57 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/552029571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T08:32:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:57 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T08:32:57.510 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:57 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/552029571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T08:32:57.511 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:57 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T08:32:57.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:57 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/552029571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T08:32:57.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:57 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T08:32:58.054 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-07-26T08:32:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:58 smithi044 ceph-mon[32404]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:58 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T08:32:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:58 smithi044 ceph-mon[32404]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T08:32:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:58 smithi171 ceph-mon[37789]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:58 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T08:32:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:58 smithi171 ceph-mon[37789]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T08:32:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:58 smithi116 ceph-mon[37097]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:32:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:58 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T08:32:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:58 smithi116 ceph-mon[37097]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T08:32:58.533 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:32:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:32:59 smithi171 ceph-mon[37789]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T08:32:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:32:59 smithi044 ceph-mon[32404]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T08:32:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:32:59 smithi116 ceph-mon[37097]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T08:33:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:00 smithi044 ceph-mon[32404]: pgmap v296: 33 pgs: 9 creating+peering, 4 active+clean, 20 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:00 smithi044 ceph-mon[32404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:00 smithi044 ceph-mon[32404]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T08:33:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:00 smithi116 ceph-mon[37097]: pgmap v296: 33 pgs: 9 creating+peering, 4 active+clean, 20 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:00 smithi116 ceph-mon[37097]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:00 smithi116 ceph-mon[37097]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T08:33:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:00 smithi171 ceph-mon[37789]: pgmap v296: 33 pgs: 9 creating+peering, 4 active+clean, 20 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:00 smithi171 ceph-mon[37789]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:00 smithi171 ceph-mon[37789]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T08:33:00.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 -- 172.21.15.44:0/988452951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 msgr2=0x7fd304113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:00.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 --2- 172.21.15.44:0/988452951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 0x7fd304113a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009900 tx=0x7fd2f002a410 comp rx=0 tx=0).stop 2024-07-26T08:33:00.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 -- 172.21.15.44:0/988452951 shutdown_connections 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 --2- 172.21.15.44:0/988452951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 0x7fd304113a70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 --2- 172.21.15.44:0/988452951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd304113fb0 0x7fd3041163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 --2- 172.21.15.44:0/988452951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd304077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 -- 172.21.15.44:0/988452951 >> 172.21.15.44:0/988452951 conn(0x7fd304101060 msgr2=0x7fd3041034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 -- 172.21.15.44:0/988452951 shutdown_connections 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.882+0000 7fd309e00640 1 -- 172.21.15.44:0/988452951 wait complete. 2024-07-26T08:33:00.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.884+0000 7fd309e00640 1 Processor -- start 2024-07-26T08:33:00.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.884+0000 7fd309e00640 1 -- start start 2024-07-26T08:33:00.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd3041a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:00.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 0x7fd3041a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd304113fb0 0x7fd3041a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd304119910 con 0x7fd304077370 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd304119790 con 0x7fd304113fb0 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd304119a90 con 0x7fd304077cc0 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd308a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd3041a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd308a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd3041a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49186/0 (socket says 172.21.15.44:49186) 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd308a00640 1 -- 172.21.15.44:0/2904128220 learned_addr learned my addr 172.21.15.44:0/2904128220 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd309400640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd304113fb0 0x7fd3041a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:00.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.885+0000 7fd303e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 0x7fd3041a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:00.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.886+0000 7fd308a00640 1 -- 172.21.15.44:0/2904128220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 msgr2=0x7fd3041a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:00.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.886+0000 7fd308a00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 0x7fd3041a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:00.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.886+0000 7fd308a00640 1 -- 172.21.15.44:0/2904128220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd304113fb0 msgr2=0x7fd3041a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:00.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.886+0000 7fd308a00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd304113fb0 0x7fd3041a5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.886+0000 7fd308a00640 1 -- 172.21.15.44:0/2904128220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3041a6310 con 0x7fd304077370 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.886+0000 7fd308a00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd3041a03a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800e9f0 tx=0x7fd2f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.887+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f800cd80 con 0x7fd304077370 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.887+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3041a6600 con 0x7fd304077370 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.887+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f80108e0 con 0x7fd304077370 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.887+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8018980 con 0x7fd304077370 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.887+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3041acd60 con 0x7fd304077370 2024-07-26T08:33:00.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.888+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd304077780 con 0x7fd304077370 2024-07-26T08:33:00.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.891+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fd2f8010430 con 0x7fd304077370 2024-07-26T08:33:00.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.893+0000 7fd301600640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd2ec0b3c20 0x7fd2ec0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:00.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.893+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7712+0+0 (secure 0 0 0) 0x7fd2f80e4250 con 0x7fd304077370 2024-07-26T08:33:00.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.893+0000 7fd303e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd2ec0b3c20 0x7fd2ec0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:00.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.894+0000 7fd303e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd2ec0b3c20 0x7fd2ec0b6130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009230 tx=0x7fd2f0031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:00.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:00.895+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd2f80ab430 con 0x7fd304077370 2024-07-26T08:33:01.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:01.101+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fd30402e0c0 con 0x7fd304077370 2024-07-26T08:33:02.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.107+0000 7fd301600640 1 -- 172.21.15.44:0/2904128220 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v77) ==== 171+0+0 (secure 0 0 0) 0x7fd2f80af980 con 0x7fd304077370 2024-07-26T08:33:02.110 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-07-26T08:33:02.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.112+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd2ec0b3c20 msgr2=0x7fd2ec0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:02.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.112+0000 7fd309e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd2ec0b3c20 0x7fd2ec0b6130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009230 tx=0x7fd2f0031400 comp rx=0 tx=0).stop 2024-07-26T08:33:02.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.112+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 msgr2=0x7fd3041a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:02.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.112+0000 7fd309e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd3041a03a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800e9f0 tx=0x7fd2f800eeb0 comp rx=0 tx=0).stop 2024-07-26T08:33:02.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.113+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 shutdown_connections 2024-07-26T08:33:02.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.113+0000 7fd309e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd2ec0b3c20 0x7fd2ec0b6130 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:02.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.113+0000 7fd309e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd304077cc0 0x7fd3041a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:02.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.113+0000 7fd309e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd304113fb0 0x7fd3041a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:02.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.113+0000 7fd309e00640 1 --2- 172.21.15.44:0/2904128220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd304077370 0x7fd3041a03a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.113+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 >> 172.21.15.44:0/2904128220 conn(0x7fd304101060 msgr2=0x7fd304116110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.115+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 shutdown_connections 2024-07-26T08:33:02.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:02.115+0000 7fd309e00640 1 -- 172.21.15.44:0/2904128220 wait complete. 2024-07-26T08:33:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:02 smithi044 ceph-mon[32404]: pgmap v298: 33 pgs: 9 creating+peering, 13 active+clean, 11 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:02 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2904128220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T08:33:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:02 smithi116 ceph-mon[37097]: pgmap v298: 33 pgs: 9 creating+peering, 13 active+clean, 11 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:02 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2904128220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T08:33:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:02 smithi171 ceph-mon[37789]: pgmap v298: 33 pgs: 9 creating+peering, 13 active+clean, 11 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:02 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2904128220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T08:33:02.965 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T08:33:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:03 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2904128220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T08:33:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:03 smithi044 ceph-mon[32404]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T08:33:03.461 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:03 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2904128220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T08:33:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:03 smithi116 ceph-mon[37097]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T08:33:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:03 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2904128220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T08:33:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:03 smithi171 ceph-mon[37789]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T08:33:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:04 smithi044 ceph-mon[32404]: pgmap v300: 97 pgs: 42 unknown, 25 creating+peering, 30 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:04 smithi044 ceph-mon[32404]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T08:33:04.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:04 smithi116 ceph-mon[37097]: pgmap v300: 97 pgs: 42 unknown, 25 creating+peering, 30 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:04.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:04 smithi116 ceph-mon[37097]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T08:33:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:04 smithi171 ceph-mon[37789]: pgmap v300: 97 pgs: 42 unknown, 25 creating+peering, 30 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:04 smithi171 ceph-mon[37789]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T08:33:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:05 smithi044 ceph-mon[32404]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T08:33:05.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:05 smithi044 ceph-mon[32404]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:05.439 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:05 smithi116 ceph-mon[37097]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T08:33:05.440 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:05 smithi116 ceph-mon[37097]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:05 smithi171 ceph-mon[37789]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T08:33:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:05 smithi171 ceph-mon[37789]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 -- 172.21.15.44:0/3141436181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010af10 msgr2=0x7f72c010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 --2- 172.21.15.44:0/3141436181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010af10 0x7f72c010b320 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009900 tx=0x7f72a802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 -- 172.21.15.44:0/3141436181 shutdown_connections 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 --2- 172.21.15.44:0/3141436181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72c010b860 0x7f72c0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 --2- 172.21.15.44:0/3141436181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010af10 0x7f72c010b320 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 --2- 172.21.15.44:0/3141436181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72c01055d0 0x7f72c01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 -- 172.21.15.44:0/3141436181 >> 172.21.15.44:0/3141436181 conn(0x7f72c0101040 msgr2=0x7f72c0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:05.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 -- 172.21.15.44:0/3141436181 shutdown_connections 2024-07-26T08:33:05.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.851+0000 7f72c5600640 1 -- 172.21.15.44:0/3141436181 wait complete. 2024-07-26T08:33:05.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.852+0000 7f72c5600640 1 Processor -- start 2024-07-26T08:33:05.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 -- start start 2024-07-26T08:33:05.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72c01055d0 0x7f72c019bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:05.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72c010af10 0x7f72c019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:05.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 0x7f72c01a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:05.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72c0115520 con 0x7f72c01055d0 2024-07-26T08:33:05.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01153a0 con 0x7f72c010b860 2024-07-26T08:33:05.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c5600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01156a0 con 0x7f72c010af10 2024-07-26T08:33:05.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 0x7f72c01a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:05.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72c010af10 0x7f72c019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:05.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 0x7f72c01a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:40562/0 (socket says 172.21.15.44:40562) 2024-07-26T08:33:05.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72bfe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72c01055d0 0x7f72c019bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:05.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.853+0000 7f72c4c00640 1 -- 172.21.15.44:0/1801355136 learned_addr learned my addr 172.21.15.44:0/1801355136 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:05.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72c4c00640 1 -- 172.21.15.44:0/1801355136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72c010af10 msgr2=0x7f72c019c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72c4c00640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72c010af10 0x7f72c019c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72c4c00640 1 -- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72c01055d0 msgr2=0x7f72c019bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72c4c00640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72c01055d0 0x7f72c019bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72c4c00640 1 -- 172.21.15.44:0/1801355136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c01a1ea0 con 0x7f72c010b860 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72c4c00640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 0x7f72c01a1770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f72b800eb10 tx=0x7f72b800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.854+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8004b20 con 0x7f72c010b860 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.855+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b8004cc0 con 0x7f72c010b860 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.855+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8010670 con 0x7f72c010b860 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.855+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c01a2190 con 0x7f72c010b860 2024-07-26T08:33:05.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.855+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c01a88f0 con 0x7f72c010b860 2024-07-26T08:33:05.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.856+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c01059e0 con 0x7f72c010b860 2024-07-26T08:33:05.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.858+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f72b8005020 con 0x7f72c010b860 2024-07-26T08:33:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.860+0000 7f72bd600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f72a40b3c20 0x7f72a40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.860+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8094+0+0 (secure 0 0 0) 0x7f72b80e4270 con 0x7f72c010b860 2024-07-26T08:33:05.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.860+0000 7f72bfe00640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f72a40b3c20 0x7f72a40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.861+0000 7f72bfe00640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f72a40b3c20 0x7f72a40b6130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009bd0 tx=0x7f72b0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:05.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:05.861+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f72b80ab900 con 0x7f72c010b860 2024-07-26T08:33:06.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.108+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f72c019d060 con 0x7f72c010b860 2024-07-26T08:33:06.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.168+0000 7f72bd600640 1 -- 172.21.15.44:0/1801355136 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f72c019d060 con 0x7f72c010b860 2024-07-26T08:33:06.170 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T08:33:06.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.174+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f72a40b3c20 msgr2=0x7f72a40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:06.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.174+0000 7f72c5600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f72a40b3c20 0x7f72a40b6130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009bd0 tx=0x7f72b0005f90 comp rx=0 tx=0).stop 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.174+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 msgr2=0x7f72c01a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.174+0000 7f72c5600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 0x7f72c01a1770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f72b800eb10 tx=0x7f72b800c2d0 comp rx=0 tx=0).stop 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.175+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 shutdown_connections 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.175+0000 7f72c5600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f72a40b3c20 0x7f72a40b6130 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.175+0000 7f72c5600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72c010af10 0x7f72c019c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.175+0000 7f72c5600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72c010b860 0x7f72c01a1770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.175+0000 7f72c5600640 1 --2- 172.21.15.44:0/1801355136 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72c01055d0 0x7f72c019bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:06.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.175+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 >> 172.21.15.44:0/1801355136 conn(0x7f72c0101040 msgr2=0x7f72c0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:06.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.176+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 shutdown_connections 2024-07-26T08:33:06.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:06.176+0000 7f72c5600640 1 -- 172.21.15.44:0/1801355136 wait complete. 2024-07-26T08:33:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:06 smithi171 ceph-mon[37789]: pgmap v303: 97 pgs: 28 unknown, 34 creating+peering, 35 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:06 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1801355136' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T08:33:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:06 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T08:33:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:06 smithi171 ceph-mon[37789]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:33:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:06 smithi044 ceph-mon[32404]: pgmap v303: 97 pgs: 28 unknown, 34 creating+peering, 35 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:06 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1801355136' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T08:33:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:06 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T08:33:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:06 smithi044 ceph-mon[32404]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:33:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:06 smithi116 ceph-mon[37097]: pgmap v303: 97 pgs: 28 unknown, 34 creating+peering, 35 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:06 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1801355136' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T08:33:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:06 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T08:33:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:06 smithi116 ceph-mon[37097]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:33:07.154 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-07-26T08:33:07.154 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-07-26T08:33:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:07 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T08:33:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:07 smithi044 ceph-mon[32404]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T08:33:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:07 smithi044 ceph-mon[32404]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T08:33:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:07 smithi044 ceph-mon[32404]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:33:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:07 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T08:33:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:07 smithi116 ceph-mon[37097]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T08:33:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:07 smithi116 ceph-mon[37097]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T08:33:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:07 smithi116 ceph-mon[37097]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:33:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:07 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T08:33:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:07 smithi171 ceph-mon[37789]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T08:33:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:07 smithi171 ceph-mon[37789]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T08:33:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:07 smithi171 ceph-mon[37789]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:33:07.615 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:08 smithi044 ceph-mon[32404]: pgmap v305: 97 pgs: 34 creating+peering, 63 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 664 B/s wr, 2 op/s 2024-07-26T08:33:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:08 smithi044 ceph-mon[32404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T08:33:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:08 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:33:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:08 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] up:active 2024-07-26T08:33:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:08 smithi044 ceph-mon[32404]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:08.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:08 smithi044 ceph-mon[32404]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-mon[37097]: pgmap v305: 97 pgs: 34 creating+peering, 63 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 664 B/s wr, 2 op/s 2024-07-26T08:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-mon[37097]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T08:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] up:active 2024-07-26T08:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-mon[37097]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-mon[37097]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:08 smithi171 ceph-mon[37789]: pgmap v305: 97 pgs: 34 creating+peering, 63 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 664 B/s wr, 2 op/s 2024-07-26T08:33:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:08 smithi171 ceph-mon[37789]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T08:33:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:08 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:33:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:08 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] up:active 2024-07-26T08:33:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:08 smithi171 ceph-mon[37789]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:08 smithi171 ceph-mon[37789]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:09.022 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:33:08 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:33:08.749+0000 7f91aabf3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T08:33:09.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 -- 172.21.15.44:0/196366794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d41057e0 msgr2=0x7fb7d4105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:09.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 --2- 172.21.15.44:0/196366794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d41057e0 0x7fb7d4105bf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009900 tx=0x7fb7c802a410 comp rx=0 tx=0).stop 2024-07-26T08:33:09.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 -- 172.21.15.44:0/196366794 shutdown_connections 2024-07-26T08:33:09.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 --2- 172.21.15.44:0/196366794 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d410b120 0x7fb7d410b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:09.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 --2- 172.21.15.44:0/196366794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410ba70 0x7fb7d41122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:09.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 --2- 172.21.15.44:0/196366794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d41057e0 0x7fb7d4105bf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:09.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 -- 172.21.15.44:0/196366794 >> 172.21.15.44:0/196366794 conn(0x7fb7d41011f0 msgr2=0x7fb7d4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:09.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 -- 172.21.15.44:0/196366794 shutdown_connections 2024-07-26T08:33:09.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.807+0000 7fb7dc800640 1 -- 172.21.15.44:0/196366794 wait complete. 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 Processor -- start 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 -- start start 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 0x7fb7d419c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410b120 0x7fb7d419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d410ba70 0x7fb7d41a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d4115720 con 0x7fb7d410ba70 2024-07-26T08:33:09.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41155a0 con 0x7fb7d410b120 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.809+0000 7fb7dc800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d41158a0 con 0x7fb7d41057e0 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 0x7fb7d419c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410b120 0x7fb7d419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 0x7fb7d419c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:55236/0 (socket says 172.21.15.44:55236) 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 -- 172.21.15.44:0/295418755 learned_addr learned my addr 172.21.15.44:0/295418755 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9400640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410b120 0x7fb7d419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:40578/0 (socket says 172.21.15.44:40578) 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7da800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d410ba70 0x7fb7d41a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:09.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 -- 172.21.15.44:0/295418755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410b120 msgr2=0x7fb7d419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:09.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410b120 0x7fb7d419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:09.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 -- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d410ba70 msgr2=0x7fb7d41a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:09.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d410ba70 0x7fb7d41a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:09.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.810+0000 7fb7d9e00640 1 -- 172.21.15.44:0/295418755 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d41a2060 con 0x7fb7d41057e0 2024-07-26T08:33:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.811+0000 7fb7d9e00640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 0x7fb7d419c0f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005940 tx=0x7fb7c80040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.811+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c803f030 con 0x7fb7d41057e0 2024-07-26T08:33:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.811+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c80044c0 con 0x7fb7d41057e0 2024-07-26T08:33:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.811+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d41a22f0 con 0x7fb7d41057e0 2024-07-26T08:33:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.811+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035070 con 0x7fb7d41057e0 2024-07-26T08:33:09.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.811+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d41a8ab0 con 0x7fb7d41057e0 2024-07-26T08:33:09.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.813+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d4106a60 con 0x7fb7d41057e0 2024-07-26T08:33:09.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.815+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fb7c8042030 con 0x7fb7d41057e0 2024-07-26T08:33:09.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.816+0000 7fb7cea00640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7bc0b3c20 0x7fb7bc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:09.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.817+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8194+0+0 (secure 0 0 0) 0x7fb7c8103fe0 con 0x7fb7d41057e0 2024-07-26T08:33:09.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.817+0000 7fb7d9400640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7bc0b3c20 0x7fb7bc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:09.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.818+0000 7fb7d9400640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7bc0b3c20 0x7fb7bc0b6130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb7d41039f0 tx=0x7fb7c4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:09.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:09.821+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7c80cafa0 con 0x7fb7d41057e0 2024-07-26T08:33:10.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.031+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fb7d41a8d10 con 0x7fb7d41057e0 2024-07-26T08:33:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:10 smithi044 ceph-mon[32404]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-07-26T08:33:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:10 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/295418755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T08:33:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:10 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T08:33:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:10 smithi116 ceph-mon[37097]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-07-26T08:33:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:10 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/295418755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T08:33:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:10 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T08:33:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:10 smithi171 ceph-mon[37789]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-07-26T08:33:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:10 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/295418755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T08:33:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:10 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T08:33:10.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.785+0000 7fb7cea00640 1 -- 172.21.15.44:0/295418755 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) ==== 157+0+0 (secure 0 0 0) 0x7fb7c80cf4f0 con 0x7fb7d41057e0 2024-07-26T08:33:10.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7bc0b3c20 msgr2=0x7fb7bc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7bc0b3c20 0x7fb7bc0b6130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb7d41039f0 tx=0x7fb7c4005f90 comp rx=0 tx=0).stop 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 msgr2=0x7fb7d419c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 0x7fb7d419c0f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005940 tx=0x7fb7c80040c0 comp rx=0 tx=0).stop 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 shutdown_connections 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb7bc0b3c20 0x7fb7bc0b6130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.790+0000 7fb7dc800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7d410ba70 0x7fb7d41a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.791+0000 7fb7dc800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d410b120 0x7fb7d419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:10.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.791+0000 7fb7dc800640 1 --2- 172.21.15.44:0/295418755 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d41057e0 0x7fb7d419c0f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:10.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.791+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 >> 172.21.15.44:0/295418755 conn(0x7fb7d41011f0 msgr2=0x7fb7d41098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:10.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.791+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 shutdown_connections 2024-07-26T08:33:10.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:10.791+0000 7fb7dc800640 1 -- 172.21.15.44:0/295418755 wait complete. 2024-07-26T08:33:11.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T08:33:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:11 smithi044 ceph-mon[32404]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-07-26T08:33:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:11 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T08:33:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:11 smithi044 ceph-mon[32404]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T08:33:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:11 smithi171 ceph-mon[37789]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-07-26T08:33:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:11 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T08:33:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:11 smithi171 ceph-mon[37789]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T08:33:12.193 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:11 smithi116 ceph-mon[37097]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-07-26T08:33:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:11 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T08:33:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:11 smithi116 ceph-mon[37097]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-07-26T08:33:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]: dispatch 2024-07-26T08:33:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T08:33:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-07-26T08:33:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T08:33:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-07-26T08:33:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-07-26T08:33:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-07-26T08:33:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]: dispatch 2024-07-26T08:33:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]: dispatch 2024-07-26T08:33:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-07-26T08:33:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-07-26T08:33:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-07-26T08:33:14.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T08:33:14.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]': finished 2024-07-26T08:33:14.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-07-26T08:33:14.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-07-26T08:33:14.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:13 smithi171 ceph-mon[37789]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-07-26T08:33:14.125 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:13 smithi044 ceph-mon[32404]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T08:33:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T08:33:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11, 8, 4]}]': finished 2024-07-26T08:33:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 7]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-07-26T08:33:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:13 smithi116 ceph-mon[37097]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T08:33:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 -- 172.21.15.44:0/3387534847 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40075800 msgr2=0x7f9c40075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 --2- 172.21.15.44:0/3387534847 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40075800 0x7f9c40075c10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009860 tx=0x7f9c2c02a340 comp rx=0 tx=0).stop 2024-07-26T08:33:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 -- 172.21.15.44:0/3387534847 shutdown_connections 2024-07-26T08:33:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 --2- 172.21.15.44:0/3387534847 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40075800 0x7f9c40075c10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 --2- 172.21.15.44:0/3387534847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40113c90 0x7f9c40116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 --2- 172.21.15.44:0/3387534847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40076db0 0x7f9c400751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:14.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 -- 172.21.15.44:0/3387534847 >> 172.21.15.44:0/3387534847 conn(0x7f9c400fee10 msgr2=0x7f9c40101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:14.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 -- 172.21.15.44:0/3387534847 shutdown_connections 2024-07-26T08:33:14.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.642+0000 7f9c46e00640 1 -- 172.21.15.44:0/3387534847 wait complete. 2024-07-26T08:33:14.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.643+0000 7f9c46e00640 1 Processor -- start 2024-07-26T08:33:14.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.643+0000 7f9c46e00640 1 -- start start 2024-07-26T08:33:14.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 0x7f9c401a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40076db0 0x7f9c401a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40113c90 0x7f9c401a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c401198d0 con 0x7f9c40076db0 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c40119750 con 0x7f9c40075800 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9c40119a50 con 0x7f9c40113c90 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c45a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 0x7f9c401a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c45000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40076db0 0x7f9c401a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c45a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 0x7f9c401a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:46162/0 (socket says 172.21.15.44:46162) 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c45a00640 1 -- 172.21.15.44:0/1179762113 learned_addr learned my addr 172.21.15.44:0/1179762113 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.644+0000 7f9c46400640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40113c90 0x7f9c401a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45000640 1 -- 172.21.15.44:0/1179762113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40113c90 msgr2=0x7f9c401a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:14.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45000640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40113c90 0x7f9c401a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:14.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45000640 1 -- 172.21.15.44:0/1179762113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 msgr2=0x7f9c401a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:14.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45000640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 0x7f9c401a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:14.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45000640 1 -- 172.21.15.44:0/1179762113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c401a62d0 con 0x7f9c40076db0 2024-07-26T08:33:14.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45a00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 0x7f9c401a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:33:14.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.645+0000 7f9c45000640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40076db0 0x7f9c401a0900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c02a860 tx=0x7f9c2c032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:14.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.646+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c037040 con 0x7f9c40076db0 2024-07-26T08:33:14.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.646+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c2c030c10 con 0x7f9c40076db0 2024-07-26T08:33:14.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.646+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0035f0 con 0x7f9c40076db0 2024-07-26T08:33:14.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.646+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c401a6560 con 0x7f9c40076db0 2024-07-26T08:33:14.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.646+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c401acd20 con 0x7f9c40076db0 2024-07-26T08:33:14.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.647+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c400766f0 con 0x7f9c40076db0 2024-07-26T08:33:14.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.650+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f9c2c043020 con 0x7f9c40076db0 2024-07-26T08:33:14.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.651+0000 7f9c3e000640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c280b3c90 0x7f9c280b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:14.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.652+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8761+0+0 (secure 0 0 0) 0x7f9c2c103330 con 0x7f9c40076db0 2024-07-26T08:33:14.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.652+0000 7f9c45a00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c280b3c90 0x7f9c280b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:14.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.652+0000 7f9c45a00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c280b3c90 0x7f9c280b61a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c34011500 tx=0x7f9c34005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:14.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.654+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9c2c04a060 con 0x7f9c40076db0 2024-07-26T08:33:14.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:14.866+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9c401129c0 con 0x7f9c40076db0 2024-07-26T08:33:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:14 smithi171 ceph-mon[37789]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T08:33:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:14 smithi044 ceph-mon[32404]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T08:33:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:14 smithi116 ceph-mon[37097]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T08:33:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:15 smithi171 ceph-mon[37789]: pgmap v312: 97 pgs: 3 peering, 94 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-07-26T08:33:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:15 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1179762113' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T08:33:16.139 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:15 smithi116 ceph-mon[37097]: pgmap v312: 97 pgs: 3 peering, 94 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-07-26T08:33:16.140 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:15 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1179762113' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T08:33:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:15 smithi044 ceph-mon[32404]: pgmap v312: 97 pgs: 3 peering, 94 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-07-26T08:33:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:15 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1179762113' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T08:33:16.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.851+0000 7f9c3e000640 1 -- 172.21.15.44:0/1179762113 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) ==== 228+0+0 (secure 0 0 0) 0x7f9c2c0328c0 con 0x7f9c40076db0 2024-07-26T08:33:16.853 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' created 2024-07-26T08:33:16.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c280b3c90 msgr2=0x7f9c280b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:16.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c280b3c90 0x7f9c280b61a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c34011500 tx=0x7f9c34005f90 comp rx=0 tx=0).stop 2024-07-26T08:33:16.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40076db0 msgr2=0x7f9c401a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:16.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40076db0 0x7f9c401a0900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c02a860 tx=0x7f9c2c032d50 comp rx=0 tx=0).stop 2024-07-26T08:33:16.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 shutdown_connections 2024-07-26T08:33:16.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c280b3c90 0x7f9c280b61a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:16.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c40113c90 0x7f9c401a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:16.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c40076db0 0x7f9c401a0900 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:16.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.856+0000 7f9c46e00640 1 --2- 172.21.15.44:0/1179762113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c40075800 0x7f9c401a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:16.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.857+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 >> 172.21.15.44:0/1179762113 conn(0x7f9c400fee10 msgr2=0x7f9c4010fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:16.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.857+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 shutdown_connections 2024-07-26T08:33:16.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:16.857+0000 7f9c46e00640 1 -- 172.21.15.44:0/1179762113 wait complete. 2024-07-26T08:33:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:16 smithi116 ceph-mon[37097]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T08:33:17.189 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:16 smithi171 ceph-mon[37789]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T08:33:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:16 smithi044 ceph-mon[32404]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T08:33:17.804 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-07-26T08:33:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:17 smithi044 ceph-mon[32404]: pgmap v314: 97 pgs: 3 peering, 94 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:17 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1179762113' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T08:33:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:17 smithi044 ceph-mon[32404]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T08:33:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:17 smithi171 ceph-mon[37789]: pgmap v314: 97 pgs: 3 peering, 94 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:17 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1179762113' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T08:33:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:17 smithi171 ceph-mon[37789]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T08:33:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:17 smithi116 ceph-mon[37097]: pgmap v314: 97 pgs: 3 peering, 94 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:17 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1179762113' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T08:33:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:17 smithi116 ceph-mon[37097]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T08:33:18.303 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:19.248 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:18 smithi044 ceph-mon[32404]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T08:33:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:18 smithi116 ceph-mon[37097]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T08:33:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:18 smithi171 ceph-mon[37789]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T08:33:20.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:19 smithi044 ceph-mon[32404]: pgmap v317: 129 pgs: 6 creating+peering, 19 unknown, 104 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:20.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:19 smithi044 ceph-mon[32404]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:20.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:19 smithi044 ceph-mon[32404]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T08:33:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:19 smithi116 ceph-mon[37097]: pgmap v317: 129 pgs: 6 creating+peering, 19 unknown, 104 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:19 smithi116 ceph-mon[37097]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:19 smithi116 ceph-mon[37097]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T08:33:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:19 smithi171 ceph-mon[37789]: pgmap v317: 129 pgs: 6 creating+peering, 19 unknown, 104 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:19 smithi171 ceph-mon[37789]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T08:33:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:19 smithi171 ceph-mon[37789]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T08:33:20.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.472+0000 7f3036800640 1 -- 172.21.15.44:0/2031565006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 msgr2=0x7f303010b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:20.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.472+0000 7f3036800640 1 --2- 172.21.15.44:0/2031565006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 0x7f303010b530 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3018009ca0 tx=0x7f301802a8c0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 -- 172.21.15.44:0/2031565006 shutdown_connections 2024-07-26T08:33:20.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 --2- 172.21.15.44:0/2031565006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f303010ba70 0x7f30301122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 --2- 172.21.15.44:0/2031565006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 0x7f303010b530 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 --2- 172.21.15.44:0/2031565006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f3030105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 -- 172.21.15.44:0/2031565006 >> 172.21.15.44:0/2031565006 conn(0x7f30301011f0 msgr2=0x7f3030103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:20.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 -- 172.21.15.44:0/2031565006 shutdown_connections 2024-07-26T08:33:20.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.473+0000 7f3036800640 1 -- 172.21.15.44:0/2031565006 wait complete. 2024-07-26T08:33:20.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.474+0000 7f3036800640 1 Processor -- start 2024-07-26T08:33:20.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.474+0000 7f3036800640 1 -- start start 2024-07-26T08:33:20.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3036800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f303019c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:20.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3036800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 0x7f303019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3036800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f303010ba70 0x7f30301a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3036800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3030115700 con 0x7f303010b120 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3036800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3030115580 con 0x7f30301057e0 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3036800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3030115880 con 0x7f303010ba70 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f3026a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 0x7f303019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f302f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f303019c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f302f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f303019c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:46174/0 (socket says 172.21.15.44:46174) 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f302f400640 1 -- 172.21.15.44:0/875258443 learned_addr learned my addr 172.21.15.44:0/875258443 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.475+0000 7f302fe00640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f303010ba70 0x7f30301a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302f400640 1 -- 172.21.15.44:0/875258443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f303010ba70 msgr2=0x7f30301a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:20.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302f400640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f303010ba70 0x7f30301a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302f400640 1 -- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 msgr2=0x7f303019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302f400640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 0x7f303019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302f400640 1 -- 172.21.15.44:0/875258443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30301a2160 con 0x7f30301057e0 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302f400640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f303019c1f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f302000e9f0 tx=0x7f302000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302000cd80 con 0x7f30301057e0 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.476+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30301a2450 con 0x7f30301057e0 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.477+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30200108e0 con 0x7f30301057e0 2024-07-26T08:33:20.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.477+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020018980 con 0x7f30301057e0 2024-07-26T08:33:20.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.477+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30301a8bb0 con 0x7f30301057e0 2024-07-26T08:33:20.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.478+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3030105bf0 con 0x7f30301057e0 2024-07-26T08:33:20.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.480+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f3020010430 con 0x7f30301057e0 2024-07-26T08:33:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.482+0000 7f302cc00640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f30140b3c20 0x7f30140b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:20.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.482+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9254+0+0 (secure 0 0 0) 0x7f30200e43c0 con 0x7f30301057e0 2024-07-26T08:33:20.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.482+0000 7f3026a00640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f30140b3c20 0x7f30140b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:20.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.483+0000 7f3026a00640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f30140b3c20 0x7f30140b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3018000f80 tx=0x7f3018009360 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:20.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.486+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f30200aaf60 con 0x7f30301057e0 2024-07-26T08:33:20.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.689+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f30301a8ea0 con 0x7f30301057e0 2024-07-26T08:33:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.903+0000 7f302cc00640 1 -- 172.21.15.44:0/875258443 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f30200af4b0 con 0x7f30301057e0 2024-07-26T08:33:20.905 INFO:teuthology.orchestra.run.smithi044.stderr:set pool 4 allow_ec_overwrites to true 2024-07-26T08:33:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.907+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f30140b3c20 msgr2=0x7f30140b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f30140b3c20 0x7f30140b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3018000f80 tx=0x7f3018009360 comp rx=0 tx=0).stop 2024-07-26T08:33:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 msgr2=0x7f303019c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f303019c1f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f302000e9f0 tx=0x7f302000eeb0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 shutdown_connections 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f30140b3c20 0x7f30140b6130 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f303010ba70 0x7f30301a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f303010b120 0x7f303019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 --2- 172.21.15.44:0/875258443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30301057e0 0x7f303019c1f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 >> 172.21.15.44:0/875258443 conn(0x7f30301011f0 msgr2=0x7f30301098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.908+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 shutdown_connections 2024-07-26T08:33:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:20.909+0000 7f3036800640 1 -- 172.21.15.44:0/875258443 wait complete. 2024-07-26T08:33:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:20 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/875258443' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T08:33:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:20 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T08:33:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:20 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/875258443' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T08:33:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:20 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T08:33:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:20 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/875258443' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T08:33:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:20 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T08:33:21.780 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-07-26T08:33:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:21 smithi044 ceph-mon[32404]: pgmap v319: 129 pgs: 6 creating+peering, 14 unknown, 109 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:21 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T08:33:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:21 smithi044 ceph-mon[32404]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T08:33:22.266 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:21 smithi116 ceph-mon[37097]: pgmap v319: 129 pgs: 6 creating+peering, 14 unknown, 109 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:21 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T08:33:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:21 smithi116 ceph-mon[37097]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T08:33:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:21 smithi171 ceph-mon[37789]: pgmap v319: 129 pgs: 6 creating+peering, 14 unknown, 109 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:33:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:21 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T08:33:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:21 smithi171 ceph-mon[37789]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T08:33:24.107 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:23 smithi044 ceph-mon[32404]: pgmap v321: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:23 smithi116 ceph-mon[37097]: pgmap v321: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:23 smithi171 ceph-mon[37789]: pgmap v321: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 -- 172.21.15.44:0/2094302644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 msgr2=0x7f095c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 --2- 172.21.15.44:0/2094302644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 0x7f095c109300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0944009900 tx=0x7f094402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 -- 172.21.15.44:0/2094302644 shutdown_connections 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 --2- 172.21.15.44:0/2094302644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f095c109840 0x7f095c110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 --2- 172.21.15.44:0/2094302644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 0x7f095c109300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 --2- 172.21.15.44:0/2094302644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1035b0 0x7f095c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 -- 172.21.15.44:0/2094302644 >> 172.21.15.44:0/2094302644 conn(0x7f095c0ff000 msgr2=0x7f095c101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:24.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 -- 172.21.15.44:0/2094302644 shutdown_connections 2024-07-26T08:33:24.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.438+0000 7f0961800640 1 -- 172.21.15.44:0/2094302644 wait complete. 2024-07-26T08:33:24.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.440+0000 7f0961800640 1 Processor -- start 2024-07-26T08:33:24.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.440+0000 7f0961800640 1 -- start start 2024-07-26T08:33:24.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.441+0000 7f0961800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f095c1035b0 0x7f095c199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:24.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.441+0000 7f0961800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 0x7f095c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:24.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.441+0000 7f0961800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 0x7f095c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:24.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.441+0000 7f0961800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1134b0 con 0x7f095c108ef0 2024-07-26T08:33:24.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.441+0000 7f0961800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f095c113330 con 0x7f095c109840 2024-07-26T08:33:24.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.441+0000 7f0961800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f095c113630 con 0x7f095c1035b0 2024-07-26T08:33:24.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f0952000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 0x7f095c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:24.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 0x7f095c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:24.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f095c1035b0 0x7f095c199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:24.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 0x7f095c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:37052/0 (socket says 172.21.15.44:37052) 2024-07-26T08:33:24.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 -- 172.21.15.44:0/3874689118 learned_addr learned my addr 172.21.15.44:0/3874689118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:24.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 -- 172.21.15.44:0/3874689118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f095c1035b0 msgr2=0x7f095c199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f095c1035b0 0x7f095c199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 -- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 msgr2=0x7f095c19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 0x7f095c19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.442+0000 7f095b400640 1 -- 172.21.15.44:0/3874689118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c19fe50 con 0x7f095c109840 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.443+0000 7f095b400640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 0x7f095c19f720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0954010ea0 tx=0x7f0954010f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.443+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954004b20 con 0x7f095c109840 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.443+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c1a0140 con 0x7f095c109840 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.443+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0954004cc0 con 0x7f095c109840 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.443+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954019740 con 0x7f095c109840 2024-07-26T08:33:24.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.443+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c075020 con 0x7f095c109840 2024-07-26T08:33:24.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.444+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c104830 con 0x7f095c109840 2024-07-26T08:33:24.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.447+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f0954005000 con 0x7f095c109840 2024-07-26T08:33:24.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.449+0000 7f0953e00640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09400b3c20 0x7f09400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:24.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.449+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9254+0+0 (secure 0 0 0) 0x7f09540e84f0 con 0x7f095c109840 2024-07-26T08:33:24.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.449+0000 7f095aa00640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09400b3c20 0x7f09400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.450+0000 7f095aa00640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09400b3c20 0x7f09400b6130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f094c009bd0 tx=0x7f094c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:24.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.450+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f09540af090 con 0x7f095c109840 2024-07-26T08:33:24.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.698+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f095c1039c0 con 0x7f095c109840 2024-07-26T08:33:24.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.923+0000 7f0953e00640 1 -- 172.21.15.44:0/3874689118 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f09540b35e0 con 0x7f095c109840 2024-07-26T08:33:24.925 INFO:teuthology.orchestra.run.smithi044.stderr:added data pool 4 to fsmap 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09400b3c20 msgr2=0x7f09400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09400b3c20 0x7f09400b6130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f094c009bd0 tx=0x7f094c005f90 comp rx=0 tx=0).stop 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 msgr2=0x7f095c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 0x7f095c19f720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0954010ea0 tx=0x7f0954010f90 comp rx=0 tx=0).stop 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 shutdown_connections 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f09400b3c20 0x7f09400b6130 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c109840 0x7f095c19f720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f095c108ef0 0x7f095c19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 --2- 172.21.15.44:0/3874689118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f095c1035b0 0x7f095c199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:24.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 >> 172.21.15.44:0/3874689118 conn(0x7f095c0ff000 msgr2=0x7f095c101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:24.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.928+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 shutdown_connections 2024-07-26T08:33:24.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:24.929+0000 7f0961800640 1 -- 172.21.15.44:0/3874689118 wait complete. 2024-07-26T08:33:24.957 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:24 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3874689118' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T08:33:24.957 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:24 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T08:33:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:24 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3874689118' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T08:33:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:24 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T08:33:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:24 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3874689118' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T08:33:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:24 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T08:33:25.814 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:33:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:25 smithi044 ceph-mon[32404]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:25 smithi044 ceph-mon[32404]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T08:33:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:25 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T08:33:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:25 smithi044 ceph-mon[32404]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:25 smithi116 ceph-mon[37097]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:25 smithi116 ceph-mon[37097]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T08:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:25 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T08:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:25 smithi116 ceph-mon[37097]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:26.302 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:26.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:25 smithi171 ceph-mon[37789]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:26.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:25 smithi171 ceph-mon[37789]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T08:33:26.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:25 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T08:33:26.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:25 smithi171 ceph-mon[37789]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:33:27.237 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:26 smithi044 ceph-mon[32404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T08:33:27.237 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:26 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:33:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:26 smithi116 ceph-mon[37097]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T08:33:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:26 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:33:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:26 smithi171 ceph-mon[37789]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T08:33:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:26 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:33:28.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:27 smithi044 ceph-mon[32404]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:27 smithi116 ceph-mon[37097]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:27 smithi171 ceph-mon[37789]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:28.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 -- 172.21.15.44:0/3250379307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077cc0 msgr2=0x7f4fb4113c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:28.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 --2- 172.21.15.44:0/3250379307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077cc0 0x7f4fb4113c60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009900 tx=0x7f4fa002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 -- 172.21.15.44:0/3250379307 shutdown_connections 2024-07-26T08:33:28.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 --2- 172.21.15.44:0/3250379307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fb41141a0 0x7f4fb41165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 --2- 172.21.15.44:0/3250379307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077cc0 0x7f4fb4113c60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 --2- 172.21.15.44:0/3250379307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4fb4077370 0x7f4fb4077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 -- 172.21.15.44:0/3250379307 >> 172.21.15.44:0/3250379307 conn(0x7f4fb41011f0 msgr2=0x7f4fb4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:28.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.454+0000 7f4fbb200640 1 -- 172.21.15.44:0/3250379307 shutdown_connections 2024-07-26T08:33:28.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.455+0000 7f4fbb200640 1 -- 172.21.15.44:0/3250379307 wait complete. 2024-07-26T08:33:28.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.456+0000 7f4fbb200640 1 Processor -- start 2024-07-26T08:33:28.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.456+0000 7f4fbb200640 1 -- start start 2024-07-26T08:33:28.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fbb200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 0x7f4fb41a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:28.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fbb200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fb4077cc0 0x7f4fb41a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:28.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fbb200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4fb41141a0 0x7f4fb41a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:28.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fbb200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4119b00 con 0x7f4fb4077cc0 2024-07-26T08:33:28.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fbb200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4119980 con 0x7f4fb4077370 2024-07-26T08:33:28.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fbb200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4119c80 con 0x7f4fb41141a0 2024-07-26T08:33:28.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 0x7f4fb41a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:28.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 0x7f4fb41a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:37080/0 (socket says 172.21.15.44:37080) 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fb3e00640 1 -- 172.21.15.44:0/976117126 learned_addr learned my addr 172.21.15.44:0/976117126 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fb3400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fb4077cc0 0x7f4fb41a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.457+0000 7f4fb9200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4fb41141a0 0x7f4fb41a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb3e00640 1 -- 172.21.15.44:0/976117126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4fb41141a0 msgr2=0x7f4fb41a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb3e00640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4fb41141a0 0x7f4fb41a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb3e00640 1 -- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fb4077cc0 msgr2=0x7f4fb41a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb3e00640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fb4077cc0 0x7f4fb41a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb3e00640 1 -- 172.21.15.44:0/976117126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb41a6480 con 0x7f4fb4077370 2024-07-26T08:33:28.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb3e00640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 0x7f4fb41a0510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400d950 tx=0x7f4fa400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:28.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.458+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4017070 con 0x7f4fb4077370 2024-07-26T08:33:28.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.459+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fa4009c30 con 0x7f4fb4077370 2024-07-26T08:33:28.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.459+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb41a6770 con 0x7f4fb4077370 2024-07-26T08:33:28.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.459+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400f620 con 0x7f4fb4077370 2024-07-26T08:33:28.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.459+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb41aced0 con 0x7f4fb4077370 2024-07-26T08:33:28.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.460+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb4077780 con 0x7f4fb4077370 2024-07-26T08:33:28.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.462+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f4fa4005030 con 0x7f4fb4077370 2024-07-26T08:33:28.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.464+0000 7f4fb0c00640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4f980b3c20 0x7f4f980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:28.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.464+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f4fa40e3c40 con 0x7f4fb4077370 2024-07-26T08:33:28.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.464+0000 7f4fb3400640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4f980b3c20 0x7f4f980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:28.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.465+0000 7f4fb3400640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4f980b3c20 0x7f4f980b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009230 tx=0x7f4fa0031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:28.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.465+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4fa4009de0 con 0x7f4fb4077370 2024-07-26T08:33:28.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.712+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fb41a1640 con 0x7f4fb4077370 2024-07-26T08:33:28.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.713+0000 7f4fb0c00640 1 -- 172.21.15.44:0/976117126 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11854 (secure 0 0 0) 0x7f4fb41a1640 con 0x7f4fb4077370 2024-07-26T08:33:28.716 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:33:28.716 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"btime":"2024-07-26T08:33:24:904316+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24650,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24655,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24668,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24679,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24686,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:33:24.701218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24701,"qdb_cluster":[24701]},"id":1}]} 2024-07-26T08:33:28.716 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4f980b3c20 msgr2=0x7f4f980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4f980b3c20 0x7f4f980b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0009230 tx=0x7f4fa0031400 comp rx=0 tx=0).stop 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 msgr2=0x7f4fb41a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 0x7f4fb41a0510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400d950 tx=0x7f4fa400de10 comp rx=0 tx=0).stop 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 shutdown_connections 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4f980b3c20 0x7f4f980b6130 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4fb41141a0 0x7f4fb41a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4fb4077cc0 0x7f4fb41a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 --2- 172.21.15.44:0/976117126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fb4077370 0x7f4fb41a0510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:28.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 >> 172.21.15.44:0/976117126 conn(0x7f4fb41011f0 msgr2=0x7f4fb4103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:28.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 shutdown_connections 2024-07-26T08:33:28.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:28.716+0000 7f4fbb200640 1 -- 172.21.15.44:0/976117126 wait complete. 2024-07-26T08:33:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:28 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/976117126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:33:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:28 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/976117126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:33:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:28 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/976117126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:33:29.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:33:30.012 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:29 smithi044 ceph-mon[32404]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:29 smithi116 ceph-mon[37097]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:29 smithi171 ceph-mon[37789]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:31 smithi044 ceph-mon[32404]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:32.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.266+0000 7f86e6600640 1 -- 172.21.15.44:0/1504474960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 msgr2=0x7f86e0105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:32.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.266+0000 7f86e6600640 1 --2- 172.21.15.44:0/1504474960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e0105bf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f86d0009900 tx=0x7f86d002a400 comp rx=0 tx=0).stop 2024-07-26T08:33:32.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 -- 172.21.15.44:0/1504474960 shutdown_connections 2024-07-26T08:33:32.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 --2- 172.21.15.44:0/1504474960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010ba70 0x7f86e01122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 --2- 172.21.15.44:0/1504474960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86e010b120 0x7f86e010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 --2- 172.21.15.44:0/1504474960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e0105bf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 -- 172.21.15.44:0/1504474960 >> 172.21.15.44:0/1504474960 conn(0x7f86e01011f0 msgr2=0x7f86e0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:32.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 -- 172.21.15.44:0/1504474960 shutdown_connections 2024-07-26T08:33:32.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.267+0000 7f86e6600640 1 -- 172.21.15.44:0/1504474960 wait complete. 2024-07-26T08:33:32.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.268+0000 7f86e6600640 1 Processor -- start 2024-07-26T08:33:32.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.268+0000 7f86e6600640 1 -- start start 2024-07-26T08:33:32.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86e6600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e019c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:32.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86e6600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010b120 0x7f86e019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:32.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86e6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86e010ba70 0x7f86e01a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:32.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86e6600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f86e0115770 con 0x7f86e01057e0 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86e6600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86e01155f0 con 0x7f86e010ba70 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86e6600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f86e01158f0 con 0x7f86e010b120 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86df400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e019c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86df400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e019c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47080/0 (socket says 172.21.15.44:47080) 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86dea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010b120 0x7f86e019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86df400640 1 -- 172.21.15.44:0/3428848671 learned_addr learned my addr 172.21.15.44:0/3428848671 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:32.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.269+0000 7f86dfe00640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86e010ba70 0x7f86e01a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86dea00640 1 -- 172.21.15.44:0/3428848671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86e010ba70 msgr2=0x7f86e01a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86dea00640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86e010ba70 0x7f86e01a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86dea00640 1 -- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 msgr2=0x7f86e019c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86dea00640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e019c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86dea00640 1 -- 172.21.15.44:0/3428848671 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e01a20b0 con 0x7f86e010b120 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86dea00640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010b120 0x7f86e019c6e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f86e006c3a0 tx=0x7f86cc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc00cc40 con 0x7f86e010b120 2024-07-26T08:33:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e01a23a0 con 0x7f86e010b120 2024-07-26T08:33:32.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86cc00cde0 con 0x7f86e010b120 2024-07-26T08:33:32.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc018760 con 0x7f86e010b120 2024-07-26T08:33:32.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.270+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e01a8b00 con 0x7f86e010b120 2024-07-26T08:33:32.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:31 smithi116 ceph-mon[37097]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:32.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.271+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e0105bf0 con 0x7f86e010b120 2024-07-26T08:33:32.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.275+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f86cc010a90 con 0x7f86e010b120 2024-07-26T08:33:32.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.276+0000 7f86d7e00640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f86c40b3c20 0x7f86c40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:32.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.277+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f86cc0e4860 con 0x7f86e010b120 2024-07-26T08:33:32.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.277+0000 7f86df400640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f86c40b3c20 0x7f86c40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:32.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.277+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f86cc0ab3e0 con 0x7f86e010b120 2024-07-26T08:33:32.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.278+0000 7f86df400640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f86c40b3c20 0x7f86c40b6130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86d00096c0 tx=0x7f86d0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:31 smithi171 ceph-mon[37789]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:32.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.483+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86e01a8da0 con 0x7f86e010b120 2024-07-26T08:33:32.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.485+0000 7f86d7e00640 1 -- 172.21.15.44:0/3428848671 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24881 (secure 0 0 0) 0x7f86cc0af930 con 0x7f86e010b120 2024-07-26T08:33:32.487 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:33:32.487 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:33:24.700706+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:33:32.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f86c40b3c20 msgr2=0x7f86c40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f86c40b3c20 0x7f86c40b6130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86d00096c0 tx=0x7f86d0005a20 comp rx=0 tx=0).stop 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010b120 msgr2=0x7f86e019c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010b120 0x7f86e019c6e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f86e006c3a0 tx=0x7f86cc00ee50 comp rx=0 tx=0).stop 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 shutdown_connections 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f86c40b3c20 0x7f86c40b6130 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86e010b120 0x7f86e019c6e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86e010ba70 0x7f86e01a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 --2- 172.21.15.44:0/3428848671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f86e01057e0 0x7f86e019c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:32.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.488+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 >> 172.21.15.44:0/3428848671 conn(0x7f86e01011f0 msgr2=0x7f86e01098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:32.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.489+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 shutdown_connections 2024-07-26T08:33:32.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:32.489+0000 7f86e6600640 1 -- 172.21.15.44:0/3428848671 wait complete. 2024-07-26T08:33:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:33 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3428848671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:33 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3428848671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:33 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3428848671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:33.406 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:33:33.883 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:34 smithi044 ceph-mon[32404]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:34 smithi116 ceph-mon[37097]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:34 smithi171 ceph-mon[37789]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 -- 172.21.15.44:0/2028630740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f223810b120 msgr2=0x7f223810b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 --2- 172.21.15.44:0/2028630740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f223810b120 0x7f223810b530 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2220009900 tx=0x7f222002a410 comp rx=0 tx=0).stop 2024-07-26T08:33:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 -- 172.21.15.44:0/2028630740 shutdown_connections 2024-07-26T08:33:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 --2- 172.21.15.44:0/2028630740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f223810ba70 0x7f22381122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 --2- 172.21.15.44:0/2028630740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f223810b120 0x7f223810b530 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 --2- 172.21.15.44:0/2028630740 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22381057e0 0x7f2238105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 -- 172.21.15.44:0/2028630740 >> 172.21.15.44:0/2028630740 conn(0x7f22381011f0 msgr2=0x7f2238103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.127+0000 7f223e400640 1 -- 172.21.15.44:0/2028630740 shutdown_connections 2024-07-26T08:33:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.128+0000 7f223e400640 1 -- 172.21.15.44:0/2028630740 wait complete. 2024-07-26T08:33:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 Processor -- start 2024-07-26T08:33:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 -- start start 2024-07-26T08:33:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 0x7f223819c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223810b120 0x7f223819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f223810ba70 0x7f22381a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2238115750 con 0x7f22381057e0 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22381155d0 con 0x7f223810ba70 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.129+0000 7f223e400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f22381158d0 con 0x7f223810b120 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 0x7f223819c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 0x7f223819c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47096/0 (socket says 172.21.15.44:47096) 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 -- 172.21.15.44:0/1242037644 learned_addr learned my addr 172.21.15.44:0/1242037644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f222fe00640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223810b120 0x7f223819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237e00640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f223810ba70 0x7f22381a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 -- 172.21.15.44:0/1242037644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223810b120 msgr2=0x7f223819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223810b120 0x7f223819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 -- 172.21.15.44:0/1242037644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f223810ba70 msgr2=0x7f22381a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f223810ba70 0x7f22381a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 -- 172.21.15.44:0/1242037644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22381a20d0 con 0x7f22381057e0 2024-07-26T08:33:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.130+0000 7f2237400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 0x7f223819c160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f222800e9f0 tx=0x7f222800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.131+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222800cd80 con 0x7f22381057e0 2024-07-26T08:33:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.131+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22381a23c0 con 0x7f22381057e0 2024-07-26T08:33:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.131+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22280054d0 con 0x7f22381057e0 2024-07-26T08:33:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.131+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228010690 con 0x7f22381057e0 2024-07-26T08:33:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.131+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22381a8b20 con 0x7f22381057e0 2024-07-26T08:33:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.132+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2238105bf0 con 0x7f22381057e0 2024-07-26T08:33:36.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.135+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f2228004e30 con 0x7f22381057e0 2024-07-26T08:33:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.138+0000 7f2234c00640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f221c0b3c20 0x7f221c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.138+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f22280e4670 con 0x7f22381057e0 2024-07-26T08:33:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.138+0000 7f222fe00640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f221c0b3c20 0x7f221c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.139+0000 7f222fe00640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f221c0b3c20 0x7f221c0b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f222002aa00 tx=0x7f2220005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.140+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2228055600 con 0x7f22381057e0 2024-07-26T08:33:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:36 smithi044 ceph-mon[32404]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:36 smithi116 ceph-mon[37097]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:36 smithi171 ceph-mon[37789]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:36.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.348+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f223802e0c0 con 0x7f22381057e0 2024-07-26T08:33:36.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.353+0000 7f2234c00640 1 -- 172.21.15.44:0/1242037644 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24881 (secure 0 0 0) 0x7f22280ab840 con 0x7f22381057e0 2024-07-26T08:33:36.355 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:33:36.355 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:33:24.700706+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:33:36.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f221c0b3c20 msgr2=0x7f221c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:36.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f221c0b3c20 0x7f221c0b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f222002aa00 tx=0x7f2220005a20 comp rx=0 tx=0).stop 2024-07-26T08:33:36.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 msgr2=0x7f223819c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:36.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 0x7f223819c160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f222800e9f0 tx=0x7f222800edf0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 shutdown_connections 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f221c0b3c20 0x7f221c0b6130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223810b120 0x7f223819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f223810ba70 0x7f22381a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 --2- 172.21.15.44:0/1242037644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22381057e0 0x7f223819c160 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.356+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 >> 172.21.15.44:0/1242037644 conn(0x7f22381011f0 msgr2=0x7f22381046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:36.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.357+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 shutdown_connections 2024-07-26T08:33:36.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:36.357+0000 7f223e400640 1 -- 172.21.15.44:0/1242037644 wait complete. 2024-07-26T08:33:37.262 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:33:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:37 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1242037644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:37.342 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:37 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1242037644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:37 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1242037644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:37.769 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:38.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:38 smithi044 ceph-mon[32404]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:38 smithi171 ceph-mon[37789]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:38 smithi116 ceph-mon[37097]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:39.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 -- 172.21.15.44:0/4046281234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac108d20 msgr2=0x7f0eac109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:39.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 --2- 172.21.15.44:0/4046281234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac108d20 0x7f0eac109130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009900 tx=0x7f0e9402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:33:39.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 -- 172.21.15.44:0/4046281234 shutdown_connections 2024-07-26T08:33:39.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 --2- 172.21.15.44:0/4046281234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac108d20 0x7f0eac109130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:39.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 --2- 172.21.15.44:0/4046281234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac109670 0x7f0eac10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:39.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 --2- 172.21.15.44:0/4046281234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eac1033e0 0x7f0eac1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:39.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 -- 172.21.15.44:0/4046281234 >> 172.21.15.44:0/4046281234 conn(0x7f0eac0fee30 msgr2=0x7f0eac101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:39.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.874+0000 7f0eb1800640 1 -- 172.21.15.44:0/4046281234 shutdown_connections 2024-07-26T08:33:39.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.875+0000 7f0eb1800640 1 -- 172.21.15.44:0/4046281234 wait complete. 2024-07-26T08:33:39.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 Processor -- start 2024-07-26T08:33:39.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 -- start start 2024-07-26T08:33:39.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac1033e0 0x7f0eac075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:39.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 0x7f0eac075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:39.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eac109670 0x7f0eac075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:39.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac113300 con 0x7f0eac109670 2024-07-26T08:33:39.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac113180 con 0x7f0eac108d20 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.876+0000 7f0eb1800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac113480 con 0x7f0eac1033e0 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 0x7f0eac075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 0x7f0eac075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:37416/0 (socket says 172.21.15.44:37416) 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 -- 172.21.15.44:0/1221370892 learned_addr learned my addr 172.21.15.44:0/1221370892 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0eabe00640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac1033e0 0x7f0eac075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0eb0e00640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eac109670 0x7f0eac075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 -- 172.21.15.44:0/1221370892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac1033e0 msgr2=0x7f0eac075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac1033e0 0x7f0eac075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 -- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eac109670 msgr2=0x7f0eac075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eac109670 0x7f0eac075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.877+0000 7f0ea3400640 1 -- 172.21.15.44:0/1221370892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac1a7fd0 con 0x7f0eac108d20 2024-07-26T08:33:39.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.878+0000 7f0ea3400640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 0x7f0eac075680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e9402a910 tx=0x7f0e94031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:39.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.878+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9403f030 con 0x7f0eac108d20 2024-07-26T08:33:39.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.878+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e94004490 con 0x7f0eac108d20 2024-07-26T08:33:39.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.878+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94035070 con 0x7f0eac108d20 2024-07-26T08:33:39.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.878+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac1a8260 con 0x7f0eac108d20 2024-07-26T08:33:39.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.878+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1a86f0 con 0x7f0eac108d20 2024-07-26T08:33:39.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.879+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac1037f0 con 0x7f0eac108d20 2024-07-26T08:33:39.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.882+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f0e94042030 con 0x7f0eac108d20 2024-07-26T08:33:39.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.883+0000 7f0ea9600640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0e900b3c20 0x7f0e900b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:39.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.884+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f0e941048f0 con 0x7f0eac108d20 2024-07-26T08:33:39.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.884+0000 7f0eabe00640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0e900b3c20 0x7f0e900b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:39.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.884+0000 7f0eabe00640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0e900b3c20 0x7f0e900b6130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0eac076b90 tx=0x7f0e9c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:39.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:39.885+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0e940cbac0 con 0x7f0eac108d20 2024-07-26T08:33:40.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.129+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eac02e0c0 con 0x7f0eac108d20 2024-07-26T08:33:40.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.130+0000 7f0ea9600640 1 -- 172.21.15.44:0/1221370892 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11854 (secure 0 0 0) 0x7f0e940cb3c0 con 0x7f0eac108d20 2024-07-26T08:33:40.132 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:33:40.132 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"btime":"2024-07-26T08:33:24:904316+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24620,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24638,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24650,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24655,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24668,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24679,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24686,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:33:24.701218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24701},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24701,"qdb_cluster":[24701]},"id":1}]} 2024-07-26T08:33:40.132 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2024-07-26T08:33:40.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0e900b3c20 msgr2=0x7f0e900b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:40.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0e900b3c20 0x7f0e900b6130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0eac076b90 tx=0x7f0e9c005f90 comp rx=0 tx=0).stop 2024-07-26T08:33:40.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 msgr2=0x7f0eac075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:40.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 0x7f0eac075680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e9402a910 tx=0x7f0e94031a20 comp rx=0 tx=0).stop 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 shutdown_connections 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0e900b3c20 0x7f0e900b6130 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0eac109670 0x7f0eac075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac108d20 0x7f0eac075680 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 --2- 172.21.15.44:0/1221370892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0eac1033e0 0x7f0eac075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 >> 172.21.15.44:0/1221370892 conn(0x7f0eac0fee30 msgr2=0x7f0eac101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 shutdown_connections 2024-07-26T08:33:40.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:40.133+0000 7f0eb1800640 1 -- 172.21.15.44:0/1221370892 wait complete. 2024-07-26T08:33:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:40 smithi171 ceph-mon[37789]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:40 smithi044 ceph-mon[32404]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:40 smithi116 ceph-mon[37097]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:40.989 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:33:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:41 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1221370892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:33:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:41 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1221370892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:33:41.470 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:41 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1221370892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:33:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:42 smithi044 ceph-mon[32404]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:42 smithi171 ceph-mon[37789]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:42 smithi116 ceph-mon[37097]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 -- 172.21.15.44:0/2071144606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810af10 msgr2=0x7f840810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 --2- 172.21.15.44:0/2071144606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810af10 0x7f840810b320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009900 tx=0x7f83f402a410 comp rx=0 tx=0).stop 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 -- 172.21.15.44:0/2071144606 shutdown_connections 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 --2- 172.21.15.44:0/2071144606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840810b860 0x7f8408112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 --2- 172.21.15.44:0/2071144606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810af10 0x7f840810b320 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 --2- 172.21.15.44:0/2071144606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84081055d0 0x7f84081059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 -- 172.21.15.44:0/2071144606 >> 172.21.15.44:0/2071144606 conn(0x7f8408101040 msgr2=0x7f8408103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:43.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 -- 172.21.15.44:0/2071144606 shutdown_connections 2024-07-26T08:33:43.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.596+0000 7f840e600640 1 -- 172.21.15.44:0/2071144606 wait complete. 2024-07-26T08:33:43.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.598+0000 7f840e600640 1 Processor -- start 2024-07-26T08:33:43.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.598+0000 7f840e600640 1 -- start start 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 0x7f840819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840e600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840810af10 0x7f840819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840e600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810b860 0x7f84081a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840e600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8408115520 con 0x7f840810b860 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84081153a0 con 0x7f84081055d0 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840e600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f84081156a0 con 0x7f840810af10 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 0x7f840819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 0x7f840819bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:36604/0 (socket says 172.21.15.44:36604) 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840dc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810b860 0x7f84081a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f840d200640 1 -- 172.21.15.44:0/876825812 learned_addr learned my addr 172.21.15.44:0/876825812 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:43.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.599+0000 7f8407e00640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840810af10 0x7f840819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:43.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.600+0000 7f840d200640 1 -- 172.21.15.44:0/876825812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840810af10 msgr2=0x7f840819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:43.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.600+0000 7f840d200640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840810af10 0x7f840819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.600+0000 7f840d200640 1 -- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810b860 msgr2=0x7f84081a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.600+0000 7f840d200640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810b860 0x7f84081a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.600+0000 7f840d200640 1 -- 172.21.15.44:0/876825812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84081a1f20 con 0x7f84081055d0 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.600+0000 7f840d200640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 0x7f840819bfb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e9f0 tx=0x7f83fc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.601+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00ccc0 con 0x7f84081055d0 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.601+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84081a2210 con 0x7f84081055d0 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.601+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83fc0108e0 con 0x7f84081055d0 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.601+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc018880 con 0x7f84081055d0 2024-07-26T08:33:43.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.601+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84081a8970 con 0x7f84081055d0 2024-07-26T08:33:43.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.602+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84081059e0 con 0x7f84081055d0 2024-07-26T08:33:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.605+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f83fc010430 con 0x7f84081055d0 2024-07-26T08:33:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.606+0000 7f8405600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83f00b3ca0 0x7f83f00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.606+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f83fc0e48d0 con 0x7f84081055d0 2024-07-26T08:33:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.606+0000 7f8407e00640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83f00b3ca0 0x7f83f00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.607+0000 7f8407e00640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83f00b3ca0 0x7f83f00b61b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f83f402aa00 tx=0x7f83f4005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.608+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f83fc0abaa0 con 0x7f84081055d0 2024-07-26T08:33:43.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.817+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f840802e0c0 con 0x7f84081055d0 2024-07-26T08:33:43.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.819+0000 7f8405600640 1 -- 172.21.15.44:0/876825812 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24881 (secure 0 0 0) 0x7f83fc0ab3a0 con 0x7f84081055d0 2024-07-26T08:33:43.822 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:33:43.822 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":89,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:33:24.700706+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83f00b3ca0 msgr2=0x7f83f00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83f00b3ca0 0x7f83f00b61b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f83f402aa00 tx=0x7f83f4005a20 comp rx=0 tx=0).stop 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 msgr2=0x7f840819bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 0x7f840819bfb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e9f0 tx=0x7f83fc00edf0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 shutdown_connections 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83f00b3ca0 0x7f83f00b61b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840810af10 0x7f840819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840810b860 0x7f84081a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 --2- 172.21.15.44:0/876825812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081055d0 0x7f840819bfb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 >> 172.21.15.44:0/876825812 conn(0x7f8408101040 msgr2=0x7f8408103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.823+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 shutdown_connections 2024-07-26T08:33:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:43.824+0000 7f840e600640 1 -- 172.21.15.44:0/876825812 wait complete. 2024-07-26T08:33:44.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:44 smithi044 ceph-mon[32404]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:44.223 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:44 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/876825812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:44 smithi171 ceph-mon[37789]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:44 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/876825812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:44 smithi116 ceph-mon[37097]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:44 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/876825812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:33:44.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T08:33:06.110522+0000', 'modified': '2024-07-26T08:33:24.701218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24701}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24701': {'gid': 24701, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6839/3153223603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 3153223603}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 3153223603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24701, 'qdb_cluster': [24701]} 2024-07-26T08:33:44.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T08:33:44.673 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T08:33:45.149 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:46 smithi171 ceph-mon[37789]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:46.337 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:46 smithi044 ceph-mon[32404]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:46 smithi116 ceph-mon[37097]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 -- 172.21.15.44:0/4088309129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810ba50 msgr2=0x7fd528112280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 --2- 172.21.15.44:0/4088309129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810ba50 0x7fd528112280 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd520009900 tx=0x7fd52002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 -- 172.21.15.44:0/4088309129 shutdown_connections 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 --2- 172.21.15.44:0/4088309129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd52810b100 0x7fd52810b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 --2- 172.21.15.44:0/4088309129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810ba50 0x7fd528112280 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 --2- 172.21.15.44:0/4088309129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd528105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 -- 172.21.15.44:0/4088309129 >> 172.21.15.44:0/4088309129 conn(0x7fd528101290 msgr2=0x7fd5281036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:47.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 -- 172.21.15.44:0/4088309129 shutdown_connections 2024-07-26T08:33:47.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.505+0000 7fd52d400640 1 -- 172.21.15.44:0/4088309129 wait complete. 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.506+0000 7fd52d400640 1 Processor -- start 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.506+0000 7fd52d400640 1 -- start start 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd52d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd52819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd52d400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810b100 0x7fd52819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd52d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd52810ba50 0x7fd5281a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd52d400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd528115700 con 0x7fd52810b100 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd52d400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd528115580 con 0x7fd5281057c0 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd52d400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd528115880 con 0x7fd52810ba50 2024-07-26T08:33:47.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd526800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd52819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd526800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd52819c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:36610/0 (socket says 172.21.15.44:36610) 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd526800640 1 -- 172.21.15.44:0/2924235803 learned_addr learned my addr 172.21.15.44:0/2924235803 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd527200640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd52810ba50 0x7fd5281a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.507+0000 7fd525e00640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810b100 0x7fd52819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd526800640 1 -- 172.21.15.44:0/2924235803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd52810ba50 msgr2=0x7fd5281a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd526800640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd52810ba50 0x7fd5281a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd526800640 1 -- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810b100 msgr2=0x7fd52819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd526800640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810b100 0x7fd52819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd526800640 1 -- 172.21.15.44:0/2924235803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5281a2020 con 0x7fd5281057c0 2024-07-26T08:33:47.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd525e00640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810b100 0x7fd52819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:33:47.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd526800640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd52819c0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd51800ebc0 tx=0x7fd51800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:47.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd51f400640 1 -- 172.21.15.44:0/2924235803 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518010630 con 0x7fd5281057c0 2024-07-26T08:33:47.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd51f400640 1 -- 172.21.15.44:0/2924235803 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd518010c80 con 0x7fd5281057c0 2024-07-26T08:33:47.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5281a2310 con 0x7fd5281057c0 2024-07-26T08:33:47.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd51f400640 1 -- 172.21.15.44:0/2924235803 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518018ac0 con 0x7fd5281057c0 2024-07-26T08:33:47.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.508+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5281a8a70 con 0x7fd5281057c0 2024-07-26T08:33:47.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.512+0000 7fd534063740 1 Processor -- start 2024-07-26T08:33:47.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.512+0000 7fd534063740 1 -- start start 2024-07-26T08:33:47.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.512+0000 7fd51f400640 1 -- 172.21.15.44:0/2924235803 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fd518018c60 con 0x7fd5281057c0 2024-07-26T08:33:47.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd534063740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc134550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd534063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc134a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd534063740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd534063740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5581dc0aa950 con 0x5581dbd8ef60 2024-07-26T08:33:47.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd534063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581dc099e50 con 0x5581dbd660a0 2024-07-26T08:33:47.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd534063740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5581dc09de90 con 0x5581dbd6e280 2024-07-26T08:33:47.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd526800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc134a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd527200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd526800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc134a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:36612/0 (socket says 172.21.15.44:36612) 2024-07-26T08:33:47.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd526800640 1 -- 172.21.15.44:0/2141501124 learned_addr learned my addr 172.21.15.44:0/2141501124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:47.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.513+0000 7fd525e00640 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd526800640 1 -- 172.21.15.44:0/2141501124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 msgr2=0x5581dc134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd526800640 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc134fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd526800640 1 -- 172.21.15.44:0/2141501124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 msgr2=0x5581dc134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd526800640 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc134550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd526800640 1 -- 172.21.15.44:0/2141501124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581dc0410f0 con 0x5581dbd660a0 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd527200640 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd526800640 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc134a90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd5180eac20 tx=0x7fd5180ec670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94953178cb739bee server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.514+0000 7fd513400640 1 -- 172.21.15.44:0/2141501124 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180f3070 con 0x5581dbd660a0 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.515+0000 7fd513400640 1 -- 172.21.15.44:0/2141501124 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5180e6580 con 0x5581dbd660a0 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.515+0000 7fd513400640 1 -- 172.21.15.44:0/2141501124 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180ef840 con 0x5581dbd660a0 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.515+0000 7fd51f400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd50c0b3c20 0x7fd50c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.515+0000 7fd51f400640 1 -- 172.21.15.44:0/2924235803 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7fd5180e4880 con 0x7fd5281057c0 2024-07-26T08:33:47.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.515+0000 7fd527200640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd50c0b3c20 0x7fd50c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 -- 172.21.15.44:0/2141501124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 msgr2=0x5581dc134a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc134a90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd5180eac20 tx=0x7fd5180ec670 comp rx=0 tx=0).stop 2024-07-26T08:33:47.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 -- 172.21.15.44:0/2141501124 shutdown_connections 2024-07-26T08:33:47.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc134fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc134a90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 --2- 172.21.15.44:0/2141501124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 -- 172.21.15.44:0/2141501124 >> 172.21.15.44:0/2141501124 conn(0x5581dbd83350 msgr2=0x5581dc132440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd527200640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd50c0b3c20 0x7fd50c0b6130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd520006010 tx=0x7fd5200058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 -- 172.21.15.44:0/2141501124 shutdown_connections 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.516+0000 7fd534063740 1 -- 172.21.15.44:0/2141501124 wait complete. 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.517+0000 7fd534063740 1 -- start start 2024-07-26T08:33:47.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.517+0000 7fd534063740 1 Processor -- start 2024-07-26T08:33:47.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc13fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc1403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc14c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5581dbefd5f0 con 0x5581dbd8ef60 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581dc036520 con 0x5581dbd660a0 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5581dc142d30 con 0x5581dbd6e280 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd534063740 10 client.0 _ll_register_callbacks cb 0x5581dc045ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd527200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc13fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd525e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc1403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc13fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:36614/0 (socket says 172.21.15.44:36614) 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 -- 172.21.15.44:0/483167902 learned_addr learned my addr 172.21.15.44:0/483167902 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:47.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 -- 172.21.15.44:0/483167902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 msgr2=0x5581dc1403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc1403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 -- 172.21.15.44:0/483167902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 msgr2=0x5581dc14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc14c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.518+0000 7fd526800640 1 -- 172.21.15.44:0/483167902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581dbdfa540 con 0x5581dbd660a0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd526800640 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc13fe10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5180a73d0 tx=0x7fd5180ea6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=270115ca547755b2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd507e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd507e00640 1 -- 172.21.15.44:0/483167902 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180f3030 con 0x5581dbd660a0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd507e00640 1 -- 172.21.15.44:0/483167902 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7fd5180ead10 con 0x5581dbd660a0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5180e6580 con 0x5581dbd660a0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd507e00640 1 -- 172.21.15.44:0/483167902 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd5180a7210 con 0x5581dbd660a0 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd534063740 10 client.24797 fetch_fsmap learned FSMap version 17 2024-07-26T08:33:47.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.519+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5581dc14ebf0 con 0x5581dbd660a0 2024-07-26T08:33:47.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd507e00640 1 -- 172.21.15.44:0/483167902 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5180e6580 con 0x5581dbd660a0 2024-07-26T08:33:47.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd507e00640 1 -- 172.21.15.44:0/483167902 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7fd5180b3f70 con 0x5581dbd660a0 2024-07-26T08:33:47.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd534063740 10 client.24797 fetch_fsmap finished waiting for FSMap version 17 2024-07-26T08:33:47.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd534063740 10 client.24797 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T08:33:47.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd534063740 20 client.24797 populate_metadata read hostname 'smithi044' 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd534063740 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd4d8e0 0x5581dc1520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd534063740 4 client.24797 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.520+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581dc0ac8a0 con 0x5581dbd4d8e0 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.521+0000 7fd525e00640 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd4d8e0 0x5581dc1520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.521+0000 7fd525e00640 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd4d8e0 0x5581dc1520d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9afcf21e3d0602d7 server_cookie=534519c6ee2ba489 in_seq=0 out_seq=0 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.521+0000 7fd507e00640 10 client.24797 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.522+0000 7fd507e00640 1 -- 172.21.15.44:0/483167902 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581dc0ac8a0 con 0x5581dbd4d8e0 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.522+0000 7fd507e00640 10 client.24797 handle_command_reply: tid=0 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.522+0000 7fd534063740 1 client.24797 shutdown 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.522+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 msgr2=0x5581dc13fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.522+0000 7fd534063740 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc13fe10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5180a73d0 tx=0x7fd5180ea6d0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 shutdown_connections 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd4d8e0 0x5581dc1520d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd6e280 0x5581dc1403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd8ef60 0x5581dc14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 --2- 172.21.15.44:0/483167902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc13fe10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 >> 172.21.15.44:0/483167902 conn(0x5581dbd83350 msgr2=0x5581dc137730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd507e00640 0 client.24797 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 shutdown_connections 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 1 -- 172.21.15.44:0/483167902 wait complete. 2024-07-26T08:33:47.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.523+0000 7fd534063740 20 client.24797 trim_cache size 0 max 16384 2024-07-26T08:33:47.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 Processor -- start 2024-07-26T08:33:47.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 -- start start 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc1520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc138010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd8ef60 0x5581dc141a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5581dc150410 con 0x5581dbd6e280 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581dc150ae0 con 0x5581dbd660a0 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.543+0000 7fd534063740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5581dc1511b0 con 0x5581dbd8ef60 2024-07-26T08:33:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd525e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc138010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd525e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc138010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55164/0 (socket says 172.21.15.44:55164) 2024-07-26T08:33:47.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd525e00640 1 -- 172.21.15.44:0/949486386 learned_addr learned my addr 172.21.15.44:0/949486386 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:47.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd526800640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc1520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd527200640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd8ef60 0x5581dc141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd526800640 1 -- 172.21.15.44:0/949486386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd8ef60 msgr2=0x5581dc141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd526800640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd8ef60 0x5581dc141a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.544+0000 7fd526800640 1 -- 172.21.15.44:0/949486386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 msgr2=0x5581dc138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd526800640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc138010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd526800640 1 -- 172.21.15.44:0/949486386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581dc1410d0 con 0x5581dbd660a0 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd527200640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd8ef60 0x5581dc141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd525e00640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc138010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd526800640 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc1520d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd5180ea210 tx=0x7fd518000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b4381354ca9770 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd512a00640 1 -- 172.21.15.44:0/949486386 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180f3020 con 0x5581dbd660a0 2024-07-26T08:33:47.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd512a00640 1 -- 172.21.15.44:0/949486386 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5180ecb70 con 0x5581dbd660a0 2024-07-26T08:33:47.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.545+0000 7fd512a00640 1 -- 172.21.15.44:0/949486386 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180ecea0 con 0x5581dbd660a0 2024-07-26T08:33:47.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 -- 172.21.15.44:0/949486386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 msgr2=0x5581dc1520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc1520d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd5180ea210 tx=0x7fd518000f40 comp rx=0 tx=0).stop 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 -- 172.21.15.44:0/949486386 shutdown_connections 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd8ef60 0x5581dc141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc138010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 --2- 172.21.15.44:0/949486386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd660a0 0x5581dc1520d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.546+0000 7fd534063740 1 -- 172.21.15.44:0/949486386 >> 172.21.15.44:0/949486386 conn(0x5581dbd83350 msgr2=0x5581dc141330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.547+0000 7fd534063740 1 -- 172.21.15.44:0/949486386 shutdown_connections 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.547+0000 7fd534063740 1 -- 172.21.15.44:0/949486386 wait complete. 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.547+0000 7fd534063740 1 -- start start 2024-07-26T08:33:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.547+0000 7fd534063740 1 Processor -- start 2024-07-26T08:33:47.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 0x5581dc153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd8ef60 0x5581dc158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5581dc151f50 con 0x5581dbd6e280 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581dc151880 con 0x5581dbd8ef60 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5581dc152620 con 0x5581dbd660a0 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd534063740 10 client.0 _ll_register_callbacks cb 0x7fd5180eaa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd525e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd526800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 0x5581dc153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd525e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55180/0 (socket says 172.21.15.44:55180) 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd526800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 0x5581dc153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48418/0 (socket says 172.21.15.44:48418) 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd525e00640 1 -- 172.21.15.44:0/3941279052 learned_addr learned my addr 172.21.15.44:0/3941279052 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.548+0000 7fd527200640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd8ef60 0x5581dc158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd525e00640 1 -- 172.21.15.44:0/3941279052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 msgr2=0x5581dc153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd525e00640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 0x5581dc153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd525e00640 1 -- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd8ef60 msgr2=0x5581dc158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd525e00640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd8ef60 0x5581dc158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd525e00640 1 -- 172.21.15.44:0/3941279052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581dc159520 con 0x5581dbd6e280 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd526800640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 0x5581dc153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd525e00640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc154040 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd52806c220 tx=0x7fd514005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4250251d2e29dc00 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:47.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.549+0000 7fd507e00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:33:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.550+0000 7fd507e00640 1 -- 172.21.15.44:0/3941279052 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514004220 con 0x5581dbd6e280 2024-07-26T08:33:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.550+0000 7fd507e00640 1 -- 172.21.15.44:0/3941279052 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7fd5140043c0 con 0x5581dbd6e280 2024-07-26T08:33:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.550+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5180ecb70 con 0x5581dbd6e280 2024-07-26T08:33:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.550+0000 7fd507e00640 1 -- 172.21.15.44:0/3941279052 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd51402f050 con 0x5581dbd6e280 2024-07-26T08:33:47.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.550+0000 7fd534063740 10 client.14676 fetch_fsmap learned FSMap version 17 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.550+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5581dc15a940 con 0x5581dbd6e280 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd507e00640 1 -- 172.21.15.44:0/3941279052 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5180ecb70 con 0x5581dbd6e280 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd507e00640 1 -- 172.21.15.44:0/3941279052 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7fd5140039c0 con 0x5581dbd6e280 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd534063740 10 client.14676 fetch_fsmap finished waiting for FSMap version 17 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd534063740 10 client.14676 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd534063740 20 client.14676 populate_metadata read hostname 'smithi044' 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd534063740 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd22c90 0x5581dc1be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd534063740 4 client.14676 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.551+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "status"}) -- 0x5581dc0a8340 con 0x5581dbd22c90 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.552+0000 7fd526800640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd22c90 0x5581dc1be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.552+0000 7fd526800640 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd22c90 0x5581dc1be3d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0e3b0fce57fabc server_cookie=6e5bd496e4567218 in_seq=0 out_seq=0 2024-07-26T08:33:47.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.552+0000 7fd507e00640 10 client.14676 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:33:47.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.552+0000 7fd507e00640 1 -- 172.21.15.44:0/3941279052 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5581dc0a8340 con 0x5581dbd22c90 2024-07-26T08:33:47.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.552+0000 7fd507e00640 10 client.14676 handle_command_reply: tid=0 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.552+0000 7fd534063740 1 client.14676 shutdown 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 msgr2=0x5581dc154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc154040 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd52806c220 tx=0x7fd514005770 comp rx=0 tx=0).stop 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 shutdown_connections 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5581dbd22c90 0x5581dc1be3d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581dbd8ef60 0x5581dc158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581dbd6e280 0x5581dc154040 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 --2- 172.21.15.44:0/3941279052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5581dbd660a0 0x5581dc153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 >> 172.21.15.44:0/3941279052 conn(0x5581dbd83350 msgr2=0x5581dc137de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd507e00640 0 client.14676 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 shutdown_connections 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 1 -- 172.21.15.44:0/3941279052 wait complete. 2024-07-26T08:33:47.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.553+0000 7fd534063740 20 client.14676 trim_cache size 0 max 16384 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24701, 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 41.387986701000003, 2024-07-26T08:33:47.557 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 17, 2024-07-26T08:33:47.558 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 89, 2024-07-26T08:33:47.558 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 89, 2024-07-26T08:33:47.558 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 64.317220301000006 2024-07-26T08:33:47.558 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:33:47.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd50c0b3c20 msgr2=0x7fd50c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd50c0b3c20 0x7fd50c0b6130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd520006010 tx=0x7fd5200058d0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 msgr2=0x7fd52819c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd52819c0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd51800ebc0 tx=0x7fd51800c6a0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 shutdown_connections 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd50c0b3c20 0x7fd50c0b6130 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd52810ba50 0x7fd5281a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd52810b100 0x7fd52819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 --2- 172.21.15.44:0/2924235803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281057c0 0x7fd52819c0b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.554+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 >> 172.21.15.44:0/2924235803 conn(0x7fd528101290 msgr2=0x7fd528109720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.555+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 shutdown_connections 2024-07-26T08:33:47.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:47.555+0000 7fd52d400640 1 -- 172.21.15.44:0/2924235803 wait complete. 2024-07-26T08:33:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:48 smithi044 ceph-mon[32404]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:48 smithi171 ceph-mon[37789]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:48.397 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T08:33:48.397 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:48.397 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T08:33:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:48 smithi116 ceph-mon[37097]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:48.903 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:50.248 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:50 smithi044 ceph-mon[32404]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:50 smithi171 ceph-mon[37789]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:50 smithi116 ceph-mon[37097]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:51.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 -- 172.21.15.44:0/1473089239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810ba70 msgr2=0x7f8ae81122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:51.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 --2- 172.21.15.44:0/1473089239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810ba70 0x7f8ae81122a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009ca0 tx=0x7f8ae002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 -- 172.21.15.44:0/1473089239 shutdown_connections 2024-07-26T08:33:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 --2- 172.21.15.44:0/1473089239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae810b120 0x7f8ae810b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 --2- 172.21.15.44:0/1473089239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810ba70 0x7f8ae81122a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 --2- 172.21.15.44:0/1473089239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae81057e0 0x7f8ae8105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 -- 172.21.15.44:0/1473089239 >> 172.21.15.44:0/1473089239 conn(0x7f8ae81011f0 msgr2=0x7f8ae8103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 -- 172.21.15.44:0/1473089239 shutdown_connections 2024-07-26T08:33:51.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.120+0000 7f8af0000640 1 -- 172.21.15.44:0/1473089239 wait complete. 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.122+0000 7f8af0000640 1 Processor -- start 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.122+0000 7f8af0000640 1 -- start start 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.122+0000 7f8af0000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 0x7f8ae819c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.122+0000 7f8af0000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810b120 0x7f8ae819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8af0000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae810ba70 0x7f8ae81a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8af0000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae8115720 con 0x7f8ae810b120 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8af0000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae81155a0 con 0x7f8ae810ba70 2024-07-26T08:33:51.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8af0000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae81158a0 con 0x7f8ae81057e0 2024-07-26T08:33:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 0x7f8ae819c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aee000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae810ba70 0x7f8ae81a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 0x7f8ae819c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48444/0 (socket says 172.21.15.44:48444) 2024-07-26T08:33:51.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 -- 172.21.15.44:0/3747300475 learned_addr learned my addr 172.21.15.44:0/3747300475 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aecc00640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810b120 0x7f8ae819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 -- 172.21.15.44:0/3747300475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae810ba70 msgr2=0x7f8ae81a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae810ba70 0x7f8ae81a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 -- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810b120 msgr2=0x7f8ae819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810b120 0x7f8ae819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.123+0000 7f8aed600640 1 -- 172.21.15.44:0/3747300475 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae81a2080 con 0x7f8ae81057e0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.124+0000 7f8aed600640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 0x7f8ae819c110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00ea60 tx=0x7f8adc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.124+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc00cde0 con 0x7f8ae81057e0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.124+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae81a2370 con 0x7f8ae81057e0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.124+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8adc0035f0 con 0x7f8ae81057e0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.124+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc010690 con 0x7f8ae81057e0 2024-07-26T08:33:51.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.124+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae81a8ad0 con 0x7f8ae81057e0 2024-07-26T08:33:51.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.126+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae8105bf0 con 0x7f8ae81057e0 2024-07-26T08:33:51.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.128+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f8adc0037a0 con 0x7f8ae81057e0 2024-07-26T08:33:51.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.129+0000 7f8ae6000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8ad00b3c20 0x7f8ad00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:33:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.129+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f8adc0e4390 con 0x7f8ae81057e0 2024-07-26T08:33:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.130+0000 7f8aecc00640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8ad00b3c20 0x7f8ad00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:33:51.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.130+0000 7f8aecc00640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8ad00b3c20 0x7f8ad00b6130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8103a90 tx=0x7f8ad4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:33:51.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.134+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8adc0ab5c0 con 0x7f8ae81057e0 2024-07-26T08:33:51.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.405+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8ae802e0c0 con 0x7f8ae81057e0 2024-07-26T08:33:51.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.406+0000 7f8ae6000640 1 -- 172.21.15.44:0/3747300475 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f8adc0aaec0 con 0x7f8ae81057e0 2024-07-26T08:33:51.408 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8ad00b3c20 msgr2=0x7f8ad00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8ad00b3c20 0x7f8ad00b6130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8103a90 tx=0x7f8ad4005f90 comp rx=0 tx=0).stop 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 msgr2=0x7f8ae819c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 0x7f8ae819c110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00ea60 tx=0x7f8adc00ef20 comp rx=0 tx=0).stop 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 shutdown_connections 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8ad00b3c20 0x7f8ad00b6130 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae810ba70 0x7f8ae81a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8ae810b120 0x7f8ae819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 --2- 172.21.15.44:0/3747300475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ae81057e0 0x7f8ae819c110 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:33:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 >> 172.21.15.44:0/3747300475 conn(0x7f8ae81011f0 msgr2=0x7f8ae810a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:33:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 shutdown_connections 2024-07-26T08:33:51.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:51.409+0000 7f8af0000640 1 -- 172.21.15.44:0/3747300475 wait complete. 2024-07-26T08:33:52.232 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T08:33:52.232 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T08:33:52.232 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-07-26T08:33:52.232 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-07-26T08:33:52.232 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:52.232 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T08:33:52.233 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T08:33:52.233 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T08:33:52.233 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T08:33:52.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:52.233 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-07-26T08:33:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:52 smithi116 ceph-mon[37097]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:52.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:52.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:52 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3747300475' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T08:33:52.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:52.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:52.279 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T08:33:52.279 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T08:33:52.279 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T08:33:52.279 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-07-26T08:33:52.279 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-07-26T08:33:52.279 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 41845sec preferred_lft 41845sec 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-07-26T08:33:52.280 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-07-26T08:33:52.281 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T08:33:52.281 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-07-26T08:33:52.281 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-07-26T08:33:52.282 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T08:33:52.283 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:33:52.283 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:33:52.283 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-07-26T08:33:52.283 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-07-26T08:33:52.283 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-07-26T08:33:52.284 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T08:33:52.284 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:33:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:52 smithi171 ceph-mon[37789]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:52 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3747300475' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T08:33:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:33:52.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:52 smithi044 ceph-mon[32404]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:52.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:52.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:52 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3747300475' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T08:33:52.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:52.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:52.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:52.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:52.786 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T08:33:52.870 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-07-26T08:33:52.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:52.875 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-07-26T08:33:52.958 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-07-26T08:33:52.958 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T08:33:52.958 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-07-26T08:33:52.958 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-07-26T08:33:52.959 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T08:33:52.960 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:33:52.960 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:33:52.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-07-26T08:33:52.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-07-26T08:33:52.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-07-26T08:33:52.960 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:33:53.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:33:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:53 smithi044 ceph-mon[32404]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:53 smithi116 ceph-mon[37097]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:53 smithi171 ceph-mon[37789]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:54.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:54.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:54.128 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-07-26T08:33:54.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:54.172 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-07-26T08:33:54.254 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:33:54.254 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:33:54.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:54.254 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-07-26T08:33:54.254 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:33:54.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:33:54.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:54.458 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-07-26T08:33:54.458 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:33:54.458 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:33:54.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-07-26T08:33:54.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T08:33:54.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-07-26T08:33:54.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-07-26T08:33:54.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-07-26T08:33:54.459 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:33:54.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:33:54.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:54.792 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:33:54.792 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:33:54.793 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-07-26T08:33:54.793 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T08:33:54.793 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:33:54.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:33:54.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:54.998 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-07-26T08:33:54.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T08:33:54.998 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:55.035 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T08:33:55.035 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T08:33:55.035 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:55.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-07-26T08:33:55.235 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T08:33:55.284 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-07-26T08:33:55.285 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T08:33:55.286 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/5dea4f29725b3a66c0ed4096e354feea673380a062d6068bce380ebc68000dee/merged 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/3e3ec0e0aa68bd27202ad520d81258523521e5f577e98d9943bafa7984260932/merged 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/24d170910683186720ce595fccfac7b4a374a3dd9441fe814364fc96f309b6d2/merged 2024-07-26T08:33:55.287 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/79f0668f2c9cf34e45b58c03fee5dd6e6bef0395e653e7709d749e3eb8e24559/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/28dde9ff32fced346914caaf3ac6b66072aa1746e46248d13f47a0b5e4141a61/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/1af392e5f4258b69c9a096274a25cbd518d41c8f9708c1c3cf912037d73d3a71/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/aacfbc611ffe048e07c228c18b0173dc63405b6a89f5d84a035b788bc068d89c/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/71fb1583cbc3f8b2453255484595518cbf4460c7202d3ead853e52d54b2456a2/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/fe282fb8373abebe71952f7c9a0d0f5864e022f22840115d1c892172c5eefb26/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/32e1124880a601f7ef899becd70fd711f5179029ad0e8846534099ab3a324d10/merged 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:55.288 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:55.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:55.290 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-07-26T08:33:55.379 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-07-26T08:33:55.379 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-07-26T08:33:55.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-07-26T08:33:55.451 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-07-26T08:33:55.493 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T08:33:55.494 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/5dea4f29725b3a66c0ed4096e354feea673380a062d6068bce380ebc68000dee/merged 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/3e3ec0e0aa68bd27202ad520d81258523521e5f577e98d9943bafa7984260932/merged 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/24d170910683186720ce595fccfac7b4a374a3dd9441fe814364fc96f309b6d2/merged 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/79f0668f2c9cf34e45b58c03fee5dd6e6bef0395e653e7709d749e3eb8e24559/merged 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/28dde9ff32fced346914caaf3ac6b66072aa1746e46248d13f47a0b5e4141a61/merged 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/1af392e5f4258b69c9a096274a25cbd518d41c8f9708c1c3cf912037d73d3a71/merged 2024-07-26T08:33:55.495 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/aacfbc611ffe048e07c228c18b0173dc63405b6a89f5d84a035b788bc068d89c/merged 2024-07-26T08:33:55.496 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/71fb1583cbc3f8b2453255484595518cbf4460c7202d3ead853e52d54b2456a2/merged 2024-07-26T08:33:55.496 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/fe282fb8373abebe71952f7c9a0d0f5864e022f22840115d1c892172c5eefb26/merged 2024-07-26T08:33:55.496 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/32e1124880a601f7ef899becd70fd711f5179029ad0e8846534099ab3a324d10/merged 2024-07-26T08:33:55.496 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-07-26T08:33:55.496 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:55.496 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:55.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:55.497 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-07-26T08:33:55.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:2024-07-26T08:33:55.647+0000 7f9220592480 -1 init, newargv = 0x7f9204004ba0 newargc=15 2024-07-26T08:33:55.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[83901]: starting ceph client 2024-07-26T08:33:55.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[83901]: starting fuse 2024-07-26T08:33:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:55 smithi044 ceph-mon[32404]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:55 smithi116 ceph-mon[37097]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:55 smithi171 ceph-mon[37789]: pgmap v338: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:33:56.681 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-07-26T08:33:56.779 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T08:33:56.838 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-07-26T08:33:56.839 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-07-26T08:33:56.840 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/5dea4f29725b3a66c0ed4096e354feea673380a062d6068bce380ebc68000dee/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/3e3ec0e0aa68bd27202ad520d81258523521e5f577e98d9943bafa7984260932/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/24d170910683186720ce595fccfac7b4a374a3dd9441fe814364fc96f309b6d2/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/79f0668f2c9cf34e45b58c03fee5dd6e6bef0395e653e7709d749e3eb8e24559/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/28dde9ff32fced346914caaf3ac6b66072aa1746e46248d13f47a0b5e4141a61/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/1af392e5f4258b69c9a096274a25cbd518d41c8f9708c1c3cf912037d73d3a71/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/aacfbc611ffe048e07c228c18b0173dc63405b6a89f5d84a035b788bc068d89c/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/71fb1583cbc3f8b2453255484595518cbf4460c7202d3ead853e52d54b2456a2/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/fe282fb8373abebe71952f7c9a0d0f5864e022f22840115d1c892172c5eefb26/merged 2024-07-26T08:33:56.841 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/32e1124880a601f7ef899becd70fd711f5179029ad0e8846534099ab3a324d10/merged 2024-07-26T08:33:56.842 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-07-26T08:33:56.842 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:56.842 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:33:56.842 INFO:teuthology.orchestra.run.smithi044.stdout:/home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:56.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:56.843 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-07-26T08:33:56.898 INFO:teuthology.orchestra.run.smithi044.stdout:119 2024-07-26T08:33:56.898 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-07-26T08:33:56.898 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-07-26T08:33:56.898 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-07-26T08:33:56.898 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-07-26T08:33:56.898 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-07-26T08:33:56.898 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-07-26T08:33:56.899 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.admin")) 2024-07-26T08:33:56.900 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-07-26T08:33:57.061 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.admin.83901.asok 2024-07-26T08:33:57.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:57.079 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.83901.asok 2024-07-26T08:33:57.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:57.080 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.83901.asok status 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528", 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "admin", 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "83901", 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-07-26T08:33:57.393 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 14688, 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 14688 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-07-26T08:33:57.394 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 674756318 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 674756318 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.14688 192.168.0.1:0/674756318", 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/674756318", 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 17, 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 89, 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-07-26T08:33:57.395 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-07-26T08:33:57.396 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-07-26T08:33:57.396 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:33:57.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:57.415 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-07-26T08:33:57.415 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:57.467 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-07-26T08:33:57.468 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:57.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:33:57.468 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:33:57.561 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-07-26T08:33:57.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T08:33:57.699 INFO:teuthology.orchestra.run.smithi044.stdout:# file: . 2024-07-26T08:33:57.699 INFO:teuthology.orchestra.run.smithi044.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-07-26T08:33:57.699 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:33:57.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:33:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:33:57.709 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:33:57.709 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T08:33:57.759 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T08:33:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:57 smithi044 ceph-mon[32404]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:57 smithi116 ceph-mon[37097]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:33:58.283 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:33:58.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:57 smithi171 ceph-mon[37789]: pgmap v339: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T08:34:00.258 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:33:59 smithi044 ceph-mon[32404]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:33:59 smithi116 ceph-mon[37097]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:33:59 smithi171 ceph-mon[37789]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:00.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 -- 172.21.15.44:0/3587193420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf410f9c0 msgr2=0x7f8bf4111de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:00.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3587193420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf410f9c0 0x7f8bf4111de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ca0 tx=0x7f8bec02a950 comp rx=0 tx=0).stop 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 -- 172.21.15.44:0/3587193420 shutdown_connections 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3587193420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4102760 0x7f8bf410f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3587193420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf410f9c0 0x7f8bf4111de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3587193420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf4101e10 0x7f8bf4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.851+0000 7f8bfae00640 1 -- 172.21.15.44:0/3587193420 >> 172.21.15.44:0/3587193420 conn(0x7f8bf40fdcf0 msgr2=0x7f8bf4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.852+0000 7f8bfae00640 1 -- 172.21.15.44:0/3587193420 shutdown_connections 2024-07-26T08:34:00.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.852+0000 7f8bfae00640 1 -- 172.21.15.44:0/3587193420 wait complete. 2024-07-26T08:34:00.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.853+0000 7f8bfae00640 1 Processor -- start 2024-07-26T08:34:00.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.853+0000 7f8bfae00640 1 -- start start 2024-07-26T08:34:00.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfae00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4101e10 0x7f8bf419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:00.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfae00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf4102760 0x7f8bf419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:00.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 0x7f8bf41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:00.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfae00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4115480 con 0x7f8bf4102760 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4115300 con 0x7f8bf410f9c0 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfae00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4115600 con 0x7f8bf4101e10 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfa400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 0x7f8bf41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfa400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 0x7f8bf41a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:52726/0 (socket says 172.21.15.44:52726) 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bfa400640 1 -- 172.21.15.44:0/3666056520 learned_addr learned my addr 172.21.15.44:0/3666056520 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bf9000640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf4102760 0x7f8bf419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:00.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.854+0000 7f8bf9a00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4101e10 0x7f8bf419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9000640 1 -- 172.21.15.44:0/3666056520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4101e10 msgr2=0x7f8bf419be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9000640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4101e10 0x7f8bf419be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9000640 1 -- 172.21.15.44:0/3666056520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 msgr2=0x7f8bf41a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9000640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 0x7f8bf41a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9000640 1 -- 172.21.15.44:0/3666056520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf41a1de0 con 0x7f8bf4102760 2024-07-26T08:34:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9a00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4101e10 0x7f8bf419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bfa400640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 0x7f8bf41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf9000640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf4102760 0x7f8bf419c410 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8be000e9f0 tx=0x7f8be000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be000ccc0 con 0x7f8bf4102760 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf41a20d0 con 0x7f8bf4102760 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be00108e0 con 0x7f8bf4102760 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.855+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0018880 con 0x7f8bf4102760 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.856+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf41a8830 con 0x7f8bf4102760 2024-07-26T08:34:00.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.857+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf4102220 con 0x7f8bf4102760 2024-07-26T08:34:00.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.860+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f8be0010430 con 0x7f8bf4102760 2024-07-26T08:34:00.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.861+0000 7f8bf2000640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8bd80b3c20 0x7f8bd80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:00.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.861+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7f8be00e48c0 con 0x7f8bf4102760 2024-07-26T08:34:00.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.861+0000 7f8bf9a00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8bd80b3c20 0x7f8bd80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:00.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.862+0000 7f8bf9a00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8bd80b3c20 0x7f8bd80b6130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009fc0 tx=0x7f8be4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:00.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:00.863+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8be00ab440 con 0x7f8bf4102760 2024-07-26T08:34:01.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.081+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8bf419cfa0 con 0x7f8bf4102760 2024-07-26T08:34:01.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.082+0000 7f8bf2000640 1 -- 172.21.15.44:0/3666056520 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v89) ==== 81+0+883 (secure 0 0 0) 0x7f8be00af990 con 0x7f8bf4102760 2024-07-26T08:34:01.085 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2545437880 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:01.085 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/3951671566 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:01.085 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1988892875 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:01.085 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1840681309 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:01.085 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/3951671566 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:01.085 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/475987513 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1872480755 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1673664031 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2363513350 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1673664031 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2833279364 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3302011051 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/487473 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3491826252 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1840681309 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:01.086 INFO:teuthology.orchestra.run.smithi044.stderr:listed 15 entries 2024-07-26T08:34:01.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.085+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8bd80b3c20 msgr2=0x7f8bd80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:01.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.085+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8bd80b3c20 0x7f8bd80b6130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009fc0 tx=0x7f8be4005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:01.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.085+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf4102760 msgr2=0x7f8bf419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:01.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.085+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf4102760 0x7f8bf419c410 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8be000e9f0 tx=0x7f8be000eeb0 comp rx=0 tx=0).stop 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 shutdown_connections 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8bd80b3c20 0x7f8bd80b6130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf410f9c0 0x7f8bf41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8bf4102760 0x7f8bf419c410 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 --2- 172.21.15.44:0/3666056520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8bf4101e10 0x7f8bf419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 >> 172.21.15.44:0/3666056520 conn(0x7f8bf40fdcf0 msgr2=0x7f8bf4111b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 shutdown_connections 2024-07-26T08:34:01.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:01.086+0000 7f8bfae00640 1 -- 172.21.15.44:0/3666056520 wait complete. 2024-07-26T08:34:01.947 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:34:01.947 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T08:34:02.011 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T08:34:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:01 smithi044 ceph-mon[32404]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:01 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3666056520' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T08:34:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:01 smithi116 ceph-mon[37097]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:01 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3666056520' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T08:34:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:01 smithi171 ceph-mon[37789]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:01 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3666056520' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T08:34:02.499 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:03 smithi044 ceph-mon[32404]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:03 smithi116 ceph-mon[37097]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:03 smithi171 ceph-mon[37789]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 -- 172.21.15.44:0/4208291778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 msgr2=0x7fa3e010fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 --2- 172.21.15.44:0/4208291778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 0x7fa3e010fb80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8009ca0 tx=0x7fa3d802a8c0 comp rx=0 tx=0).stop 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 -- 172.21.15.44:0/4208291778 shutdown_connections 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 --2- 172.21.15.44:0/4208291778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 0x7fa3e010fb80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 --2- 172.21.15.44:0/4208291778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0104b80 0x7fa3e010d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 --2- 172.21.15.44:0/4208291778 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e0104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 -- 172.21.15.44:0/4208291778 >> 172.21.15.44:0/4208291778 conn(0x7fa3e00fdd50 msgr2=0x7fa3e01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 -- 172.21.15.44:0/4208291778 shutdown_connections 2024-07-26T08:34:04.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.930+0000 7fa3e6200640 1 -- 172.21.15.44:0/4208291778 wait complete. 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.931+0000 7fa3e6200640 1 Processor -- start 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.931+0000 7fa3e6200640 1 -- start start 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e6200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e019be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e6200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0104b80 0x7fa3e019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e6200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 0x7fa3e01a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e6200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0113300 con 0x7fa3e010d690 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e6200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0113180 con 0x7fa3e0104b80 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e6200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e0113480 con 0x7fa3e0104230 2024-07-26T08:34:04.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e4e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e019be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e4e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e019be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:34364/0 (socket says 172.21.15.44:34364) 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e4e00640 1 -- 172.21.15.44:0/1539131791 learned_addr learned my addr 172.21.15.44:0/1539131791 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.932+0000 7fa3e5800640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 0x7fa3e01a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3dfe00640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0104b80 0x7fa3e019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3e4e00640 1 -- 172.21.15.44:0/1539131791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0104b80 msgr2=0x7fa3e019c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3e4e00640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0104b80 0x7fa3e019c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3e4e00640 1 -- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 msgr2=0x7fa3e01a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3e4e00640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 0x7fa3e01a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:04.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3e4e00640 1 -- 172.21.15.44:0/1539131791 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e01a1e00 con 0x7fa3e0104230 2024-07-26T08:34:04.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.933+0000 7fa3e4e00640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e019be90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400e9f0 tx=0x7fa3d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:04.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.934+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d400cd80 con 0x7fa3e0104230 2024-07-26T08:34:04.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.934+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3d40108e0 con 0x7fa3e0104230 2024-07-26T08:34:04.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.934+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e01a20f0 con 0x7fa3e0104230 2024-07-26T08:34:04.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.934+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4018980 con 0x7fa3e0104230 2024-07-26T08:34:04.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.934+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e01a8850 con 0x7fa3e0104230 2024-07-26T08:34:04.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.935+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e0104640 con 0x7fa3e0104230 2024-07-26T08:34:04.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.938+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fa3d4010430 con 0x7fa3e0104230 2024-07-26T08:34:04.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.940+0000 7fa3dd600640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa3c80b3ca0 0x7fa3c80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:04.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.940+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9286+0+0 (secure 0 0 0) 0x7fa3d40e4690 con 0x7fa3e0104230 2024-07-26T08:34:04.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.940+0000 7fa3dfe00640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa3c80b3ca0 0x7fa3c80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:04.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.941+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa3d40ab860 con 0x7fa3e0104230 2024-07-26T08:34:04.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:04.941+0000 7fa3dfe00640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa3c80b3ca0 0x7fa3c80b61b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc005fd0 tx=0x7fa3cc005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:05.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.164+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa3e019cfc0 con 0x7fa3e0104230 2024-07-26T08:34:05.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.165+0000 7fa3dd600640 1 -- 172.21.15.44:0/1539131791 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v89) ==== 81+0+883 (secure 0 0 0) 0x7fa3d40ab160 con 0x7fa3e0104230 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2545437880 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/3951671566 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1988892875 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1840681309 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/3951671566 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/475987513 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:05.168 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1872480755 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1673664031 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2363513350 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1673664031 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2833279364 2024-07-27T08:23:50.924919+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3302011051 2024-07-27T08:23:20.144063+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/487473 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3491826252 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1840681309 2024-07-27T08:25:12.575949+0000 2024-07-26T08:34:05.169 INFO:teuthology.orchestra.run.smithi044.stderr:listed 15 entries 2024-07-26T08:34:05.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.168+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa3c80b3ca0 msgr2=0x7fa3c80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.168+0000 7fa3e6200640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa3c80b3ca0 0x7fa3c80b61b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc005fd0 tx=0x7fa3cc005850 comp rx=0 tx=0).stop 2024-07-26T08:34:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.168+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 msgr2=0x7fa3e019be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:05.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.168+0000 7fa3e6200640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e019be90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400e9f0 tx=0x7fa3d400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 shutdown_connections 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa3c80b3ca0 0x7fa3c80b61b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3e010d690 0x7fa3e01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e0104b80 0x7fa3e019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 --2- 172.21.15.44:0/1539131791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa3e0104230 0x7fa3e019be90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 >> 172.21.15.44:0/1539131791 conn(0x7fa3e00fdd50 msgr2=0x7fa3e0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 shutdown_connections 2024-07-26T08:34:05.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:05.169+0000 7fa3e6200640 1 -- 172.21.15.44:0/1539131791 wait complete. 2024-07-26T08:34:06.125 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-07-26T08:34:06.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:34:06.126 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-07-26T08:34:06.207 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:05 smithi044 ceph-mon[32404]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:06.207 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:05 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1539131791' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T08:34:06.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[83901]: fuse finished with error 0 and tester_r 0 2024-07-26T08:34:06.224 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T08:34:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:05 smithi116 ceph-mon[37097]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:05 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1539131791' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T08:34:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:05 smithi171 ceph-mon[37789]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:05 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1539131791' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T08:34:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:07 smithi044 ceph-mon[32404]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:07 smithi116 ceph-mon[37097]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:07 smithi171 ceph-mon[37789]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T08:34:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:09 smithi044 ceph-mon[32404]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T08:34:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:09 smithi116 ceph-mon[37097]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T08:34:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:09 smithi171 ceph-mon[37789]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T08:34:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:11 smithi044 ceph-mon[32404]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T08:34:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:11 smithi116 ceph-mon[37097]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T08:34:12.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:34:12.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:34:12.326 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-07-26T08:34:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:11 smithi171 ceph-mon[37789]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T08:34:12.399 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:34:12.876 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-07-26T08:34:13.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T08:34:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T08:34:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-07-26T08:34:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: pgmap v347: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]': finished 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-07-26T08:34:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:13 smithi044 ceph-mon[32404]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T08:34:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: pgmap v347: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T08:34:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-07-26T08:34:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-07-26T08:34:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]': finished 2024-07-26T08:34:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-07-26T08:34:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-07-26T08:34:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:13 smithi116 ceph-mon[37097]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T08:34:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: pgmap v347: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T08:34:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-07-26T08:34:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-07-26T08:34:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]': finished 2024-07-26T08:34:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-07-26T08:34:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-07-26T08:34:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:13 smithi171 ceph-mon[37789]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T08:34:15.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 -- 172.21.15.44:0/566648595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 msgr2=0x7fde94109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:15.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 --2- 172.21.15.44:0/566648595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 0x7fde94109110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fde80009900 tx=0x7fde8002a410 comp rx=0 tx=0).stop 2024-07-26T08:34:15.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 -- 172.21.15.44:0/566648595 shutdown_connections 2024-07-26T08:34:15.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 --2- 172.21.15.44:0/566648595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde94109650 0x7fde9410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 --2- 172.21.15.44:0/566648595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 0x7fde94109110 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 --2- 172.21.15.44:0/566648595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde941033c0 0x7fde941037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.233+0000 7fde9a800640 1 -- 172.21.15.44:0/566648595 >> 172.21.15.44:0/566648595 conn(0x7fde940fee10 msgr2=0x7fde94101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:15.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:14 smithi044 ceph-mon[32404]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.234+0000 7fde9a800640 1 -- 172.21.15.44:0/566648595 shutdown_connections 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.234+0000 7fde9a800640 1 -- 172.21.15.44:0/566648595 wait complete. 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.235+0000 7fde9a800640 1 Processor -- start 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.235+0000 7fde9a800640 1 -- start start 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde9a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 0x7fde9419bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde9a800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 0x7fde9419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde9a800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde94109650 0x7fde941a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde9a800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde941132c0 con 0x7fde94108d00 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde9a800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde94113140 con 0x7fde941033c0 2024-07-26T08:34:15.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde9a800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde94113440 con 0x7fde94109650 2024-07-26T08:34:15.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde99400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 0x7fde9419bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde99400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 0x7fde9419bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:56098/0 (socket says 172.21.15.44:56098) 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde99400640 1 -- 172.21.15.44:0/740954238 learned_addr learned my addr 172.21.15.44:0/740954238 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde98a00640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 0x7fde9419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.236+0000 7fde99e00640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde94109650 0x7fde941a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde99400640 1 -- 172.21.15.44:0/740954238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde94109650 msgr2=0x7fde941a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde99400640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde94109650 0x7fde941a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde99400640 1 -- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 msgr2=0x7fde9419c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde99400640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 0x7fde9419c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde99400640 1 -- 172.21.15.44:0/740954238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde941a1f40 con 0x7fde941033c0 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde99400640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 0x7fde9419bfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fde8800e9f0 tx=0x7fde8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8800cd80 con 0x7fde941033c0 2024-07-26T08:34:15.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.237+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde941a2230 con 0x7fde941033c0 2024-07-26T08:34:15.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.238+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde941a8990 con 0x7fde941033c0 2024-07-26T08:34:15.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.238+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde880108e0 con 0x7fde941033c0 2024-07-26T08:34:15.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.238+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88018980 con 0x7fde941033c0 2024-07-26T08:34:15.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.239+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde740052c0 con 0x7fde941033c0 2024-07-26T08:34:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.242+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fde88010430 con 0x7fde941033c0 2024-07-26T08:34:15.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.243+0000 7fde92000640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fde7c0b3ca0 0x7fde7c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.243+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fde880e44c0 con 0x7fde941033c0 2024-07-26T08:34:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.243+0000 7fde98a00640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fde7c0b3ca0 0x7fde7c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:15.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.244+0000 7fde98a00640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fde7c0b3ca0 0x7fde7c0b61b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fde80009230 tx=0x7fde80031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:15.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.245+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fde880aafb0 con 0x7fde941033c0 2024-07-26T08:34:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:14 smithi116 ceph-mon[37097]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T08:34:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:14 smithi171 ceph-mon[37789]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T08:34:15.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.447+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde74005880 con 0x7fde941033c0 2024-07-26T08:34:15.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.449+0000 7fde92000640 1 -- 172.21.15.44:0/740954238 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7fde880af500 con 0x7fde941033c0 2024-07-26T08:34:15.451 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:34:15.451 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:34:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fde7c0b3ca0 msgr2=0x7fde7c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fde7c0b3ca0 0x7fde7c0b61b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fde80009230 tx=0x7fde80031400 comp rx=0 tx=0).stop 2024-07-26T08:34:15.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 msgr2=0x7fde9419bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 0x7fde9419bfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fde8800e9f0 tx=0x7fde8800eeb0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 shutdown_connections 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fde7c0b3ca0 0x7fde7c0b61b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde94109650 0x7fde941a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde94108d00 0x7fde9419c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 --2- 172.21.15.44:0/740954238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde941033c0 0x7fde9419bfd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:15.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 >> 172.21.15.44:0/740954238 conn(0x7fde940fee10 msgr2=0x7fde94108430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:15.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.452+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 shutdown_connections 2024-07-26T08:34:15.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:15.453+0000 7fde9a800640 1 -- 172.21.15.44:0/740954238 wait complete. 2024-07-26T08:34:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:15 smithi044 ceph-mon[32404]: pgmap v350: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-07-26T08:34:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:15 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/740954238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:34:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:15 smithi116 ceph-mon[37097]: pgmap v350: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-07-26T08:34:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:15 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/740954238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:34:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:15 smithi171 ceph-mon[37789]: pgmap v350: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-07-26T08:34:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:15 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/740954238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:34:16.426 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:34:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:17 smithi044 ceph-mon[32404]: pgmap v351: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:17 smithi116 ceph-mon[37097]: pgmap v351: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:17 smithi171 ceph-mon[37789]: pgmap v351: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:19.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.328+0000 7f7546200640 1 -- 172.21.15.44:0/1606594680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 msgr2=0x7f7540109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:19.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.328+0000 7f7546200640 1 --2- 172.21.15.44:0/1606594680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f7540109110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7530009860 tx=0x7f753002a340 comp rx=0 tx=0).stop 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 -- 172.21.15.44:0/1606594680 shutdown_connections 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 --2- 172.21.15.44:0/1606594680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f7540109110 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 --2- 172.21.15.44:0/1606594680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540109650 0x7f754010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 --2- 172.21.15.44:0/1606594680 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75401033c0 0x7f75401037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 -- 172.21.15.44:0/1606594680 >> 172.21.15.44:0/1606594680 conn(0x7f75400fee10 msgr2=0x7f7540101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 -- 172.21.15.44:0/1606594680 shutdown_connections 2024-07-26T08:34:19.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.329+0000 7f7546200640 1 -- 172.21.15.44:0/1606594680 wait complete. 2024-07-26T08:34:19.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.330+0000 7f7546200640 1 Processor -- start 2024-07-26T08:34:19.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 -- start start 2024-07-26T08:34:19.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75401033c0 0x7f754019bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:19.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f754019c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:19.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7540109650 0x7f75401a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:19.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75401132c0 con 0x7f7540109650 2024-07-26T08:34:19.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7540113140 con 0x7f75401033c0 2024-07-26T08:34:19.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.331+0000 7f7546200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7540113440 con 0x7f7540108d00 2024-07-26T08:34:19.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f753fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f754019c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:19.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7544e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75401033c0 0x7f754019bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:19.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f753fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f754019c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:57850/0 (socket says 172.21.15.44:57850) 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f753fe00640 1 -- 172.21.15.44:0/2185662497 learned_addr learned my addr 172.21.15.44:0/2185662497 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7545800640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7540109650 0x7f75401a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7544e00640 1 -- 172.21.15.44:0/2185662497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 msgr2=0x7f754019c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7544e00640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f754019c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7544e00640 1 -- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7540109650 msgr2=0x7f75401a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7544e00640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7540109650 0x7f75401a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.332+0000 7f7544e00640 1 -- 172.21.15.44:0/2185662497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75401a1f60 con 0x7f75401033c0 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.333+0000 7f7544e00640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75401033c0 0x7f754019bff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f753400e9f0 tx=0x7f753400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.333+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753400ccc0 con 0x7f75401033c0 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.333+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75401a2250 con 0x7f75401033c0 2024-07-26T08:34:19.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.333+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75340108e0 con 0x7f75401033c0 2024-07-26T08:34:19.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.333+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534018880 con 0x7f75401033c0 2024-07-26T08:34:19.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.333+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75401a89b0 con 0x7f75401033c0 2024-07-26T08:34:19.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.334+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75401037d0 con 0x7f75401033c0 2024-07-26T08:34:19.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.337+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f7534010430 con 0x7f75401033c0 2024-07-26T08:34:19.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.338+0000 7f753d600640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f75280b3c20 0x7f75280b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:19.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.339+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f75340e4480 con 0x7f75401033c0 2024-07-26T08:34:19.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.339+0000 7f753fe00640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f75280b3c20 0x7f75280b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:19.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.340+0000 7f753fe00640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f75280b3c20 0x7f75280b6130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f75300057c0 tx=0x7f75300058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:19.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.340+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f75340aaf70 con 0x7f75401033c0 2024-07-26T08:34:19.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.546+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75401a8c90 con 0x7f75401033c0 2024-07-26T08:34:19.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.548+0000 7f753d600640 1 -- 172.21.15.44:0/2185662497 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7f75340af4c0 con 0x7f75401033c0 2024-07-26T08:34:19.550 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:34:19.550 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:34:19.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.551+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f75280b3c20 msgr2=0x7f75280b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:19.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.551+0000 7f7546200640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f75280b3c20 0x7f75280b6130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f75300057c0 tx=0x7f75300058d0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.551+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75401033c0 msgr2=0x7f754019bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.551+0000 7f7546200640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75401033c0 0x7f754019bff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f753400e9f0 tx=0x7f753400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 shutdown_connections 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f75280b3c20 0x7f75280b6130 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7540108d00 0x7f754019c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7540109650 0x7f75401a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 --2- 172.21.15.44:0/2185662497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75401033c0 0x7f754019bff0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 >> 172.21.15.44:0/2185662497 conn(0x7f75400fee10 msgr2=0x7f7540107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 shutdown_connections 2024-07-26T08:34:19.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:19.552+0000 7f7546200640 1 -- 172.21.15.44:0/2185662497 wait complete. 2024-07-26T08:34:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:19 smithi044 ceph-mon[32404]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:19 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2185662497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:34:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:19 smithi116 ceph-mon[37097]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:19 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2185662497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:34:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:19 smithi171 ceph-mon[37789]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:19 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2185662497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:34:20.532 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-26T08:34:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:21 smithi044 ceph-mon[32404]: pgmap v353: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:21 smithi116 ceph-mon[37097]: pgmap v353: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:21 smithi171 ceph-mon[37789]: pgmap v353: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:23.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 -- 172.21.15.44:0/1820229779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c1022a0 msgr2=0x7fdf6c1026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:23.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 --2- 172.21.15.44:0/1820229779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c1026b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009900 tx=0x7fdf5c02a410 comp rx=0 tx=0).stop 2024-07-26T08:34:23.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 -- 172.21.15.44:0/1820229779 shutdown_connections 2024-07-26T08:34:23.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 --2- 172.21.15.44:0/1820229779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c108530 0x7fdf6c10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:23.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 --2- 172.21.15.44:0/1820229779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:23.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 --2- 172.21.15.44:0/1820229779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c1026b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:23.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 -- 172.21.15.44:0/1820229779 >> 172.21.15.44:0/1820229779 conn(0x7fdf6c0fdcf0 msgr2=0x7fdf6c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:23.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.450+0000 7fdf6b200640 1 -- 172.21.15.44:0/1820229779 shutdown_connections 2024-07-26T08:34:23.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.451+0000 7fdf6b200640 1 -- 172.21.15.44:0/1820229779 wait complete. 2024-07-26T08:34:23.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 Processor -- start 2024-07-26T08:34:23.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 -- start start 2024-07-26T08:34:23.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:23.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:23.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c108530 0x7fdf6c1a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:23.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1121e0 con 0x7fdf6c107be0 2024-07-26T08:34:23.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c112060 con 0x7fdf6c1022a0 2024-07-26T08:34:23.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.452+0000 7fdf6b200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c112360 con 0x7fdf6c108530 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47824/0 (socket says 172.21.15.44:47824) 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c19c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47084/0 (socket says 172.21.15.44:47084) 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf6a800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c108530 0x7fdf6c1a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69e00640 1 -- 172.21.15.44:0/4025852822 learned_addr learned my addr 172.21.15.44:0/4025852822 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf6a800640 1 -- 172.21.15.44:0/4025852822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 msgr2=0x7fdf6c19bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:23.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf6a800640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c19bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:23.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf6a800640 1 -- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 msgr2=0x7fdf6c19c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:23.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf6a800640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c19c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:23.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf6a800640 1 -- 172.21.15.44:0/4025852822 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c1a1e70 con 0x7fdf6c108530 2024-07-26T08:34:23.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69e00640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:23.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.453+0000 7fdf69400640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c19c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:23.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.454+0000 7fdf6a800640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c108530 0x7fdf6c1a1740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009fd0 tx=0x7fdf6400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:23.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.454+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400ce20 con 0x7fdf6c108530 2024-07-26T08:34:23.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.454+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf64005020 con 0x7fdf6c108530 2024-07-26T08:34:23.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.454+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c1a2160 con 0x7fdf6c108530 2024-07-26T08:34:23.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.454+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64010690 con 0x7fdf6c108530 2024-07-26T08:34:23.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.454+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c1a88c0 con 0x7fdf6c108530 2024-07-26T08:34:23.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.456+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c1026b0 con 0x7fdf6c108530 2024-07-26T08:34:23.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.458+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fdf640054d0 con 0x7fdf6c108530 2024-07-26T08:34:23.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.459+0000 7fdf62a00640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdf500b3ca0 0x7fdf500b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:23.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.460+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fdf640e4790 con 0x7fdf6c108530 2024-07-26T08:34:23.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.460+0000 7fdf69e00640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdf500b3ca0 0x7fdf500b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:23.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.461+0000 7fdf69e00640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdf500b3ca0 0x7fdf500b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c031440 tx=0x7fdf5c0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:23.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.464+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdf640ab8d0 con 0x7fdf6c108530 2024-07-26T08:34:23.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:23.714+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdf6c02e0c0 con 0x7fdf6c108530 2024-07-26T08:34:24.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.000+0000 7fdf62a00640 1 -- 172.21.15.44:0/4025852822 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fdf640ab1d0 con 0x7fdf6c108530 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdf500b3ca0 msgr2=0x7fdf500b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdf500b3ca0 0x7fdf500b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c031440 tx=0x7fdf5c0058d0 comp rx=0 tx=0).stop 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c108530 msgr2=0x7fdf6c1a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c108530 0x7fdf6c1a1740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009fd0 tx=0x7fdf6400ef10 comp rx=0 tx=0).stop 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 shutdown_connections 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdf500b3ca0 0x7fdf500b61b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf6c108530 0x7fdf6c1a1740 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdf6c107be0 0x7fdf6c19c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:24.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 --2- 172.21.15.44:0/4025852822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf6c1022a0 0x7fdf6c19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:24.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.005+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 >> 172.21.15.44:0/4025852822 conn(0x7fdf6c0fdcf0 msgr2=0x7fdf6c1072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:24.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.006+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 shutdown_connections 2024-07-26T08:34:24.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:24.006+0000 7fdf6b200640 1 -- 172.21.15.44:0/4025852822 wait complete. 2024-07-26T08:34:24.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:24 smithi044 ceph-mon[32404]: pgmap v354: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:24.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:24 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4025852822' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T08:34:24.096 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:24 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T08:34:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:24 smithi116 ceph-mon[37097]: pgmap v354: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:24 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4025852822' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T08:34:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:24 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T08:34:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:24 smithi171 ceph-mon[37789]: pgmap v354: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:24 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4025852822' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T08:34:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:24 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T08:34:24.850 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-07-26T08:34:24.850 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-07-26T08:34:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:25 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T08:34:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:25 smithi044 ceph-mon[32404]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:34:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:25 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T08:34:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:25 smithi116 ceph-mon[37097]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:34:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:25 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T08:34:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:25 smithi171 ceph-mon[37789]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:34:25.340 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:26 smithi044 ceph-mon[32404]: pgmap v355: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:26 smithi171 ceph-mon[37789]: pgmap v355: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:26 smithi116 ceph-mon[37097]: pgmap v355: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.762+0000 7f2407e00640 1 -- 172.21.15.44:0/3293890227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 msgr2=0x7f24081164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.762+0000 7f2407e00640 1 --2- 172.21.15.44:0/3293890227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 0x7f24081164b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2400009ca0 tx=0x7f240002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:34:27.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 -- 172.21.15.44:0/3293890227 shutdown_connections 2024-07-26T08:34:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 --2- 172.21.15.44:0/3293890227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 0x7f24081164b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 --2- 172.21.15.44:0/3293890227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f240810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 --2- 172.21.15.44:0/3293890227 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 0x7f2408109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 -- 172.21.15.44:0/3293890227 >> 172.21.15.44:0/3293890227 conn(0x7f2408078ff0 msgr2=0x7f2408107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 -- 172.21.15.44:0/3293890227 shutdown_connections 2024-07-26T08:34:27.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.763+0000 7f2407e00640 1 -- 172.21.15.44:0/3293890227 wait complete. 2024-07-26T08:34:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.764+0000 7f2407e00640 1 Processor -- start 2024-07-26T08:34:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.764+0000 7f2407e00640 1 -- start start 2024-07-26T08:34:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.764+0000 7f2407e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 0x7f24081a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2407e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f24081a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2407e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 0x7f24081a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:27.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2407e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2408119940 con 0x7f240810fc80 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2407e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24081197c0 con 0x7f240810f330 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2407e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2408119ac0 con 0x7f24081099f0 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 0x7f24081a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f24081a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f24081a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47832/0 (socket says 172.21.15.44:47832) 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 -- 172.21.15.44:0/2057652002 learned_addr learned my addr 172.21.15.44:0/2057652002 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 0x7f24081a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:42914/0 (socket says 172.21.15.44:42914) 2024-07-26T08:34:27.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2407400640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 0x7f24081a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 -- 172.21.15.44:0/2057652002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 msgr2=0x7f24081a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 0x7f24081a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 -- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 msgr2=0x7f24081a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 0x7f24081a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:27.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.765+0000 7f2406000640 1 -- 172.21.15.44:0/2057652002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24081a6200 con 0x7f240810f330 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.766+0000 7f2406000640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f24081a0830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e9f0 tx=0x7f23f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.766+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f400ccc0 con 0x7f240810f330 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.766+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24081a64f0 con 0x7f240810f330 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.766+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23f40108e0 con 0x7f240810f330 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.766+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4018860 con 0x7f240810f330 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.766+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24081acc50 con 0x7f240810f330 2024-07-26T08:34:27.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.767+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2408109e00 con 0x7f240810f330 2024-07-26T08:34:27.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.770+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f23f4010430 con 0x7f240810f330 2024-07-26T08:34:27.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.772+0000 7f23ff400640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f23ec0b3c20 0x7f23ec0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:27.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.772+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f23f40e48d0 con 0x7f240810f330 2024-07-26T08:34:27.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.772+0000 7f2406a00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f23ec0b3c20 0x7f23ec0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:27.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.773+0000 7f2406a00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f23ec0b3c20 0x7f23ec0b6130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009bd0 tx=0x7f23f8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:27.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:27.773+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f23f40ab3c0 con 0x7f240810f330 2024-07-26T08:34:28.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.021+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f240802e0c0 con 0x7f240810f330 2024-07-26T08:34:28.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.039+0000 7f23ff400640 1 -- 172.21.15.44:0/2057652002 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f23f40af910 con 0x7f240810f330 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f23ec0b3c20 msgr2=0x7f23ec0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f23ec0b3c20 0x7f23ec0b6130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009bd0 tx=0x7f23f8005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 msgr2=0x7f24081a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f24081a0830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e9f0 tx=0x7f23f400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 shutdown_connections 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f23ec0b3c20 0x7f23ec0b6130 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240810fc80 0x7f24081a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240810f330 0x7f24081a0830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 --2- 172.21.15.44:0/2057652002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24081099f0 0x7f24081a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.043+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 >> 172.21.15.44:0/2057652002 conn(0x7f2408078ff0 msgr2=0x7f24080795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:28.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.044+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 shutdown_connections 2024-07-26T08:34:28.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:28.044+0000 7f2407e00640 1 -- 172.21.15.44:0/2057652002 wait complete. 2024-07-26T08:34:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:28 smithi171 ceph-mon[37789]: pgmap v356: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:28 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2057652002' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T08:34:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:28 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T08:34:28.423 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:28 smithi044 ceph-mon[32404]: pgmap v356: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:28.423 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:28 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2057652002' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T08:34:28.423 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:28 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T08:34:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:28 smithi116 ceph-mon[37097]: pgmap v356: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-07-26T08:34:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:28 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2057652002' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T08:34:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:28 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T08:34:28.876 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-26T08:34:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:29 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T08:34:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:29 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T08:34:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:29 smithi044 ceph-mon[32404]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T08:34:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:29 smithi044 ceph-mon[32404]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:34:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:29 smithi044 ceph-mon[32404]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-07-26T08:34:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:29 smithi044 ceph-mon[32404]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T08:34:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:29 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T08:34:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:29 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T08:34:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:29 smithi171 ceph-mon[37789]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T08:34:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:29 smithi171 ceph-mon[37789]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:34:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:29 smithi171 ceph-mon[37789]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-07-26T08:34:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:29 smithi171 ceph-mon[37789]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T08:34:29.368 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:29 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T08:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:29 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T08:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:29 smithi116 ceph-mon[37097]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T08:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:29 smithi116 ceph-mon[37097]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T08:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:29 smithi116 ceph-mon[37097]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-07-26T08:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:29 smithi116 ceph-mon[37097]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T08:34:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:30 smithi044 ceph-mon[32404]: pgmap v357: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-07-26T08:34:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:30 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:active 2024-07-26T08:34:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:30 smithi044 ceph-mon[32404]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T08:34:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:30 smithi044 ceph-mon[32404]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-07-26T08:34:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:30 smithi044 ceph-mon[32404]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-07-26T08:34:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:30 smithi044 ceph-mon[32404]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T08:34:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:30 smithi171 ceph-mon[37789]: pgmap v357: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-07-26T08:34:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:30 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:active 2024-07-26T08:34:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:30 smithi171 ceph-mon[37789]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T08:34:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:30 smithi171 ceph-mon[37789]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-07-26T08:34:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:30 smithi171 ceph-mon[37789]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-07-26T08:34:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:30 smithi171 ceph-mon[37789]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T08:34:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:30 smithi116 ceph-mon[37097]: pgmap v357: 129 pgs: 129 active+clean; 581 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-07-26T08:34:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:30 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:active 2024-07-26T08:34:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:30 smithi116 ceph-mon[37097]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T08:34:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:30 smithi116 ceph-mon[37097]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-07-26T08:34:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:30 smithi116 ceph-mon[37097]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-07-26T08:34:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:30 smithi116 ceph-mon[37097]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T08:34:31.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:31 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] up:active 2024-07-26T08:34:31.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:31 smithi044 ceph-mon[32404]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T08:34:31.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:31 smithi044 ceph-mon[32404]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-07-26T08:34:31.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:31 smithi044 ceph-mon[32404]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-07-26T08:34:31.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:31 smithi044 ceph-mon[32404]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-07-26T08:34:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:31 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] up:active 2024-07-26T08:34:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:31 smithi171 ceph-mon[37789]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T08:34:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:31 smithi171 ceph-mon[37789]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-07-26T08:34:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:31 smithi171 ceph-mon[37789]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-07-26T08:34:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:31 smithi171 ceph-mon[37789]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-07-26T08:34:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:31 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] up:active 2024-07-26T08:34:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:31 smithi116 ceph-mon[37097]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T08:34:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:31 smithi116 ceph-mon[37097]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-07-26T08:34:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:31 smithi116 ceph-mon[37097]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-07-26T08:34:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:31 smithi116 ceph-mon[37097]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-07-26T08:34:31.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 -- 172.21.15.44:0/1599801232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075ad0 msgr2=0x7fc24810f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 --2- 172.21.15.44:0/1599801232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075ad0 0x7fc24810f610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc230009900 tx=0x7fc23002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 -- 172.21.15.44:0/1599801232 shutdown_connections 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 --2- 172.21.15.44:0/1599801232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 0x7fc248111f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 --2- 172.21.15.44:0/1599801232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075ad0 0x7fc24810f610 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 --2- 172.21.15.44:0/1599801232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075180 0x7fc248075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 -- 172.21.15.44:0/1599801232 >> 172.21.15.44:0/1599801232 conn(0x7fc24807b280 msgr2=0x7fc24807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 -- 172.21.15.44:0/1599801232 shutdown_connections 2024-07-26T08:34:31.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.799+0000 7fc24e600640 1 -- 172.21.15.44:0/1599801232 wait complete. 2024-07-26T08:34:31.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 Processor -- start 2024-07-26T08:34:31.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 -- start start 2024-07-26T08:34:31.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075180 0x7fc24819c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:31.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 0x7fc24819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:31.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 0x7fc2481a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:31.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481156c0 con 0x7fc248075180 2024-07-26T08:34:31.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc248115540 con 0x7fc24810fb50 2024-07-26T08:34:31.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.801+0000 7fc24e600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc248115840 con 0x7fc248075ad0 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 0x7fc24819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc247400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075180 0x7fc24819c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 0x7fc24819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:34236/0 (socket says 172.21.15.44:34236) 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 -- 172.21.15.44:0/1664077762 learned_addr learned my addr 172.21.15.44:0/1664077762 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc247e00640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 0x7fc2481a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 -- 172.21.15.44:0/1664077762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 msgr2=0x7fc2481a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 0x7fc2481a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 -- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075180 msgr2=0x7fc24819c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075180 0x7fc24819c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:31.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc23fe00640 1 -- 172.21.15.44:0/1664077762 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2481a2040 con 0x7fc248075ad0 2024-07-26T08:34:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.802+0000 7fc247e00640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 0x7fc2481a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.803+0000 7fc23fe00640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 0x7fc24819c670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc230005ab0 tx=0x7fc230003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.803+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23003f070 con 0x7fc248075ad0 2024-07-26T08:34:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.803+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc230031b50 con 0x7fc248075ad0 2024-07-26T08:34:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.803+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230035030 con 0x7fc248075ad0 2024-07-26T08:34:31.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.803+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2481a22d0 con 0x7fc248075ad0 2024-07-26T08:34:31.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.803+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2481a8a90 con 0x7fc248075ad0 2024-07-26T08:34:31.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.804+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc248075590 con 0x7fc248075ad0 2024-07-26T08:34:31.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.807+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fc230042070 con 0x7fc248075ad0 2024-07-26T08:34:31.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.809+0000 7fc244c00640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc2280b3c20 0x7fc2280b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:31.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.809+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fc230104a80 con 0x7fc248075ad0 2024-07-26T08:34:31.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.809+0000 7fc247400640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc2280b3c20 0x7fc2280b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:31.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.810+0000 7fc247400640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc2280b3c20 0x7fc2280b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc238009bd0 tx=0x7fc238005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:31.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:31.810+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc2300cb570 con 0x7fc248075ad0 2024-07-26T08:34:32.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.057+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fc2481a8cf0 con 0x7fc248075ad0 2024-07-26T08:34:32.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.093+0000 7fc244c00640 1 -- 172.21.15.44:0/1664077762 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7fc2300cfac0 con 0x7fc248075ad0 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc2280b3c20 msgr2=0x7fc2280b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc2280b3c20 0x7fc2280b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc238009bd0 tx=0x7fc238005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 msgr2=0x7fc24819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 0x7fc24819c670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc230005ab0 tx=0x7fc230003b90 comp rx=0 tx=0).stop 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 shutdown_connections 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc2280b3c20 0x7fc2280b6130 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:32.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc248075ad0 0x7fc24819c670 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810fb50 0x7fc2481a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 --2- 172.21.15.44:0/1664077762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc248075180 0x7fc24819c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.096+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 >> 172.21.15.44:0/1664077762 conn(0x7fc24807b280 msgr2=0x7fc2481011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.097+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 shutdown_connections 2024-07-26T08:34:32.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:32.097+0000 7fc24e600640 1 -- 172.21.15.44:0/1664077762 wait complete. 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: pgmap v358: 129 pgs: 129 active+clean; 582 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] up:active 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1664077762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T08:34:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:32 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: pgmap v358: 129 pgs: 129 active+clean; 582 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] up:active 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-07-26T08:34:32.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1664077762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T08:34:32.514 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:32 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T08:34:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: pgmap v358: 129 pgs: 129 active+clean; 582 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-07-26T08:34:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] up:active 2024-07-26T08:34:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1664077762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T08:34:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:32 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T08:34:33.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-26T08:34:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:33 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T08:34:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:33 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] up:active 2024-07-26T08:34:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:33 smithi044 ceph-mon[32404]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T08:34:33.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:33 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:34:33.505 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:33 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T08:34:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:33 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] up:active 2024-07-26T08:34:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:33 smithi116 ceph-mon[37097]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T08:34:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:33 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:34:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:33 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T08:34:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:33 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] up:active 2024-07-26T08:34:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:33 smithi171 ceph-mon[37789]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T08:34:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:33 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:34:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:34 smithi044 ceph-mon[32404]: pgmap v359: 129 pgs: 129 active+clean; 585 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 3.3 KiB/s wr, 9 op/s 2024-07-26T08:34:34.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:34 smithi116 ceph-mon[37097]: pgmap v359: 129 pgs: 129 active+clean; 585 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 3.3 KiB/s wr, 9 op/s 2024-07-26T08:34:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:34 smithi171 ceph-mon[37789]: pgmap v359: 129 pgs: 129 active+clean; 585 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 3.3 KiB/s wr, 9 op/s 2024-07-26T08:34:36.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 -- 172.21.15.44:0/1694814381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810f330 msgr2=0x7fb7a810f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:36.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1694814381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810f330 0x7fb7a810f740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb794009900 tx=0x7fb79402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:34:36.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 -- 172.21.15.44:0/1694814381 shutdown_connections 2024-07-26T08:34:36.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1694814381 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7a810fc80 0x7fb7a81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:36.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1694814381 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810f330 0x7fb7a810f740 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:36.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1694814381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a81099f0 0x7fb7a8109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:36.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.035+0000 7fb7ae200640 1 -- 172.21.15.44:0/1694814381 >> 172.21.15.44:0/1694814381 conn(0x7fb7a8078ff0 msgr2=0x7fb7a8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:36.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.036+0000 7fb7ae200640 1 -- 172.21.15.44:0/1694814381 shutdown_connections 2024-07-26T08:34:36.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.036+0000 7fb7ae200640 1 -- 172.21.15.44:0/1694814381 wait complete. 2024-07-26T08:34:36.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.037+0000 7fb7ae200640 1 Processor -- start 2024-07-26T08:34:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.037+0000 7fb7ae200640 1 -- start start 2024-07-26T08:34:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.038+0000 7fb7ae200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7a81099f0 0x7fb7a81a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.038+0000 7fb7ae200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 0x7fb7a81a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.038+0000 7fb7ae200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810fc80 0x7fb7a81a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:36.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.038+0000 7fb7ae200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8119940 con 0x7fb7a810fc80 2024-07-26T08:34:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.038+0000 7fb7ae200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a81197c0 con 0x7fb7a810f330 2024-07-26T08:34:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.038+0000 7fb7ae200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8119ac0 con 0x7fb7a81099f0 2024-07-26T08:34:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 0x7fb7a81a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7ad800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810fc80 0x7fb7a81a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:36.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7ace00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7a81099f0 0x7fb7a81a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 0x7fb7a81a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:51576/0 (socket says 172.21.15.44:51576) 2024-07-26T08:34:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 -- 172.21.15.44:0/1799175308 learned_addr learned my addr 172.21.15.44:0/1799175308 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 -- 172.21.15.44:0/1799175308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7a81099f0 msgr2=0x7fb7a81a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7a81099f0 0x7fb7a81a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:36.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 -- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810fc80 msgr2=0x7fb7a81a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810fc80 0x7fb7a81a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7a7e00640 1 -- 172.21.15.44:0/1799175308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a81a62a0 con 0x7fb7a810f330 2024-07-26T08:34:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.039+0000 7fb7ad800640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810fc80 0x7fb7a81a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.040+0000 7fb7a7e00640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 0x7fb7a81a08d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb79402a910 tx=0x7fb794031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:36.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.040+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79403f030 con 0x7fb7a810f330 2024-07-26T08:34:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.040+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a81a6530 con 0x7fb7a810f330 2024-07-26T08:34:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.040+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb794004490 con 0x7fb7a810f330 2024-07-26T08:34:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.040+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794035070 con 0x7fb7a810f330 2024-07-26T08:34:36.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.040+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a81accf0 con 0x7fb7a810f330 2024-07-26T08:34:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.041+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a810ac70 con 0x7fb7a810f330 2024-07-26T08:34:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.044+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fb794042030 con 0x7fb7a810f330 2024-07-26T08:34:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.045+0000 7fb7a5600640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb78c0b3c20 0x7fb78c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:36.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.046+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb794104440 con 0x7fb7a810f330 2024-07-26T08:34:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.046+0000 7fb7ace00640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb78c0b3c20 0x7fb78c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:36.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.047+0000 7fb7ace00640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb78c0b3c20 0x7fb78c0b6130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb798009bd0 tx=0x7fb798005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:36.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.050+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7940caf30 con 0x7fb7a810f330 2024-07-26T08:34:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:36 smithi044 ceph-mon[32404]: pgmap v360: 129 pgs: 129 active+clean; 586 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-07-26T08:34:36.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:36.300+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb7a8111a70 con 0x7fb7a810f330 2024-07-26T08:34:36.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:36 smithi171 ceph-mon[37789]: pgmap v360: 129 pgs: 129 active+clean; 586 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-07-26T08:34:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:36 smithi116 ceph-mon[37097]: pgmap v360: 129 pgs: 129 active+clean; 586 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-07-26T08:34:37.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.135+0000 7fb7a5600640 1 -- 172.21.15.44:0/1799175308 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v29) ==== 115+0+0 (secure 0 0 0) 0x7fb7940cf480 con 0x7fb7a810f330 2024-07-26T08:34:37.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb78c0b3c20 msgr2=0x7fb78c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb78c0b3c20 0x7fb78c0b6130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb798009bd0 tx=0x7fb798005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 msgr2=0x7fb7a81a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 0x7fb7a81a08d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb79402a910 tx=0x7fb794031a20 comp rx=0 tx=0).stop 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 shutdown_connections 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb78c0b3c20 0x7fb78c0b6130 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7a810fc80 0x7fb7a81a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:37.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7a810f330 0x7fb7a81a08d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:37.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 --2- 172.21.15.44:0/1799175308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7a81099f0 0x7fb7a81a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:37.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 >> 172.21.15.44:0/1799175308 conn(0x7fb7a8078ff0 msgr2=0x7fb7a8107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:37.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.140+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 shutdown_connections 2024-07-26T08:34:37.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:37.141+0000 7fb7ae200640 1 -- 172.21.15.44:0/1799175308 wait complete. 2024-07-26T08:34:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:37 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1799175308' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T08:34:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:37 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T08:34:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:37 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1799175308' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T08:34:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:37 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T08:34:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:37 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1799175308' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T08:34:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:37 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T08:34:38.181 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-07-26T08:34:38.182 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-26T08:34:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:38 smithi044 ceph-mon[32404]: pgmap v361: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-26T08:34:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:38 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T08:34:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:38 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:34:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:38 smithi116 ceph-mon[37097]: pgmap v361: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-26T08:34:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:38 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T08:34:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:38 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:34:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:38 smithi171 ceph-mon[37789]: pgmap v361: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-26T08:34:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:38 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T08:34:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:38 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:34:38.679 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:40 smithi044 ceph-mon[32404]: pgmap v362: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.0 KiB/s wr, 20 op/s 2024-07-26T08:34:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:40 smithi116 ceph-mon[37097]: pgmap v362: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.0 KiB/s wr, 20 op/s 2024-07-26T08:34:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:40 smithi171 ceph-mon[37789]: pgmap v362: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.0 KiB/s wr, 20 op/s 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.219+0000 7fd759200640 1 -- 172.21.15.44:0/988690470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410f330 msgr2=0x7fd75410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.219+0000 7fd759200640 1 --2- 172.21.15.44:0/988690470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410f330 0x7fd75410f740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd740009900 tx=0x7fd74002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 -- 172.21.15.44:0/988690470 shutdown_connections 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 --2- 172.21.15.44:0/988690470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd75410fc80 0x7fd7541164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 --2- 172.21.15.44:0/988690470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410f330 0x7fd75410f740 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 --2- 172.21.15.44:0/988690470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7541099f0 0x7fd754109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 -- 172.21.15.44:0/988690470 >> 172.21.15.44:0/988690470 conn(0x7fd754078ff0 msgr2=0x7fd754107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 -- 172.21.15.44:0/988690470 shutdown_connections 2024-07-26T08:34:41.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.220+0000 7fd759200640 1 -- 172.21.15.44:0/988690470 wait complete. 2024-07-26T08:34:41.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.221+0000 7fd759200640 1 Processor -- start 2024-07-26T08:34:41.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 -- start start 2024-07-26T08:34:41.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7541099f0 0x7fd7541a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd75410f330 0x7fd7541a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 0x7fd7541a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd754119990 con 0x7fd7541099f0 2024-07-26T08:34:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd754119810 con 0x7fd75410fc80 2024-07-26T08:34:41.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.222+0000 7fd759200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd754119b10 con 0x7fd75410f330 2024-07-26T08:34:41.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7541099f0 0x7fd7541a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:41.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 0x7fd7541a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:41.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7541099f0 0x7fd7541a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50908/0 (socket says 172.21.15.44:50908) 2024-07-26T08:34:41.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 0x7fd7541a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:51600/0 (socket says 172.21.15.44:51600) 2024-07-26T08:34:41.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753400640 1 -- 172.21.15.44:0/345821644 learned_addr learned my addr 172.21.15.44:0/345821644 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:41.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd752a00640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd75410f330 0x7fd7541a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:41.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 -- 172.21.15.44:0/345821644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd75410f330 msgr2=0x7fd7541a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:41.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd75410f330 0x7fd7541a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 -- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7541099f0 msgr2=0x7fd7541a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:41.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7541099f0 0x7fd7541a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.223+0000 7fd753e00640 1 -- 172.21.15.44:0/345821644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7541a62d0 con 0x7fd75410fc80 2024-07-26T08:34:41.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.224+0000 7fd753e00640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 0x7fd7541a5ba0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00e940 tx=0x7fd74c00ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:41.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.224+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c004d20 con 0x7fd75410fc80 2024-07-26T08:34:41.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.224+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd74c005020 con 0x7fd75410fc80 2024-07-26T08:34:41.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.224+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0116d0 con 0x7fd75410fc80 2024-07-26T08:34:41.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.224+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7541a65c0 con 0x7fd75410fc80 2024-07-26T08:34:41.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.224+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7541acd20 con 0x7fd75410fc80 2024-07-26T08:34:41.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.225+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd754109e00 con 0x7fd75410fc80 2024-07-26T08:34:41.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.228+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fd74c005460 con 0x7fd75410fc80 2024-07-26T08:34:41.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.230+0000 7fd74be00640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd7380b3ca0 0x7fd7380b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:41.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.230+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fd74c0e4500 con 0x7fd75410fc80 2024-07-26T08:34:41.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.230+0000 7fd753400640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd7380b3ca0 0x7fd7380b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:41.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.231+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd74c016030 con 0x7fd75410fc80 2024-07-26T08:34:41.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.231+0000 7fd753400640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd7380b3ca0 0x7fd7380b61b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd744009c90 tx=0x7fd744005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:41.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.465+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd754063230 con 0x7fd7380b3ca0 2024-07-26T08:34:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.507+0000 7fd74be00640 1 -- 172.21.15.44:0/345821644 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd754063230 con 0x7fd7380b3ca0 2024-07-26T08:34:41.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.510+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd7380b3ca0 msgr2=0x7fd7380b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:41.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.510+0000 7fd759200640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd7380b3ca0 0x7fd7380b61b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd744009c90 tx=0x7fd744005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:41.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.510+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 msgr2=0x7fd7541a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:41.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.510+0000 7fd759200640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 0x7fd7541a5ba0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00e940 tx=0x7fd74c00ee00 comp rx=0 tx=0).stop 2024-07-26T08:34:41.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 shutdown_connections 2024-07-26T08:34:41.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd7380b3ca0 0x7fd7380b61b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd75410f330 0x7fd7541a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75410fc80 0x7fd7541a5ba0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 --2- 172.21.15.44:0/345821644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7541099f0 0x7fd7541a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:41.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 >> 172.21.15.44:0/345821644 conn(0x7fd754078ff0 msgr2=0x7fd754107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:41.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 shutdown_connections 2024-07-26T08:34:41.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:41.511+0000 7fd759200640 1 -- 172.21.15.44:0/345821644 wait complete. 2024-07-26T08:34:41.766 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:34:41 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:34:41.475+0000 7f4562e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:34:41.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:34:41 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:34:41.475+0000 7f4562e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:34:41.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:34:41 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:34:41.475+0000 7f4562e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:34:41.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:34:41 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:34:41.475+0000 7f4562e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:34:41.767 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:34:41 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:34:41.475+0000 7f4562e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:34:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:42 smithi044 ceph-mon[32404]: pgmap v363: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.5 KiB/s wr, 20 op/s 2024-07-26T08:34:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:42 smithi044 ceph-mon[32404]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:42.391 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-26T08:34:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:42 smithi116 ceph-mon[37097]: pgmap v363: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.5 KiB/s wr, 20 op/s 2024-07-26T08:34:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:42 smithi116 ceph-mon[37097]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:42 smithi171 ceph-mon[37789]: pgmap v363: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.5 KiB/s wr, 20 op/s 2024-07-26T08:34:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:42 smithi171 ceph-mon[37789]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:42.867 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:44 smithi044 ceph-mon[32404]: pgmap v364: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.7 KiB/s wr, 20 op/s 2024-07-26T08:34:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:44 smithi044 ceph-mon[32404]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T08:34:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:44 smithi116 ceph-mon[37097]: pgmap v364: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.7 KiB/s wr, 20 op/s 2024-07-26T08:34:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:44 smithi116 ceph-mon[37097]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T08:34:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:44 smithi171 ceph-mon[37789]: pgmap v364: 129 pgs: 129 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.7 KiB/s wr, 20 op/s 2024-07-26T08:34:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:44 smithi171 ceph-mon[37789]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T08:34:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 -- 172.21.15.44:0/3643535400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc0751e0 msgr2=0x7fc0dc0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3643535400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc0755f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009860 tx=0x7fc0cc02a340 comp rx=0 tx=0).stop 2024-07-26T08:34:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 -- 172.21.15.44:0/3643535400 shutdown_connections 2024-07-26T08:34:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3643535400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc075b30 0x7fc0dc10f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3643535400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc10f920 0x7fc0dc111d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3643535400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc0755f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 -- 172.21.15.44:0/3643535400 >> 172.21.15.44:0/3643535400 conn(0x7fc0dc0fee10 msgr2=0x7fc0dc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:45.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.207+0000 7fc0e1600640 1 -- 172.21.15.44:0/3643535400 shutdown_connections 2024-07-26T08:34:45.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.208+0000 7fc0e1600640 1 -- 172.21.15.44:0/3643535400 wait complete. 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 Processor -- start 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 -- start start 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc075b30 0x7fc0dc19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc10f920 0x7fc0dc1a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1154b0 con 0x7fc0dc10f920 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc115330 con 0x7fc0dc075b30 2024-07-26T08:34:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.209+0000 7fc0e1600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc115630 con 0x7fc0dc0751e0 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0e0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc10f920 0x7fc0dc1a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0dbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0d3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc075b30 0x7fc0dc19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0e0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc10f920 0x7fc0dc1a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56776/0 (socket says 172.21.15.44:56776) 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0dbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46322/0 (socket says 172.21.15.44:46322) 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0e0c00640 1 -- 172.21.15.44:0/3246843747 learned_addr learned my addr 172.21.15.44:0/3246843747 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0d3400640 1 -- 172.21.15.44:0/3246843747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc0751e0 msgr2=0x7fc0dc19bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0d3400640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc19bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0d3400640 1 -- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc10f920 msgr2=0x7fc0dc1a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0d3400640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc10f920 0x7fc0dc1a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.210+0000 7fc0d3400640 1 -- 172.21.15.44:0/3246843747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc1a1e70 con 0x7fc0dc075b30 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.211+0000 7fc0d3400640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc075b30 0x7fc0dc19c4a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc06bfb0 tx=0x7fc0c400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.211+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4004440 con 0x7fc0dc075b30 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.211+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0c400cd70 con 0x7fc0dc075b30 2024-07-26T08:34:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.211+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc1a2160 con 0x7fc0dc075b30 2024-07-26T08:34:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.211+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4019480 con 0x7fc0dc075b30 2024-07-26T08:34:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.211+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc1a88c0 con 0x7fc0dc075b30 2024-07-26T08:34:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.212+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc0755f0 con 0x7fc0dc075b30 2024-07-26T08:34:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.215+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fc0c4003680 con 0x7fc0dc075b30 2024-07-26T08:34:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.216+0000 7fc0d9600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc0c00b3d30 0x7fc0c00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.217+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fc0c40e4740 con 0x7fc0dc075b30 2024-07-26T08:34:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.217+0000 7fc0dbe00640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc0c00b3d30 0x7fc0c00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.218+0000 7fc0dbe00640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc0c00b3d30 0x7fc0c00b6240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc02a930 tx=0x7fc0cc0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.218+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc0c4009510 con 0x7fc0dc075b30 2024-07-26T08:34:45.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.451+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc0dc102130 con 0x7fc0c00b3d30 2024-07-26T08:34:45.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.497+0000 7fc0d9600640 1 -- 172.21.15.44:0/3246843747 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc0dc102130 con 0x7fc0c00b3d30 2024-07-26T08:34:45.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.500+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc0c00b3d30 msgr2=0x7fc0c00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:45.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.500+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc0c00b3d30 0x7fc0c00b6240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc02a930 tx=0x7fc0cc0058d0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.500+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc075b30 msgr2=0x7fc0dc19c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:45.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.500+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc075b30 0x7fc0dc19c4a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc06bfb0 tx=0x7fc0c400ee50 comp rx=0 tx=0).stop 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 shutdown_connections 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc0c00b3d30 0x7fc0c00b6240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0dc10f920 0x7fc0dc1a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc075b30 0x7fc0dc19c4a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 --2- 172.21.15.44:0/3246843747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc0dc0751e0 0x7fc0dc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 >> 172.21.15.44:0/3246843747 conn(0x7fc0dc0fee10 msgr2=0x7fc0dc101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 shutdown_connections 2024-07-26T08:34:45.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:45.501+0000 7fc0e1600640 1 -- 172.21.15.44:0/3246843747 wait complete. 2024-07-26T08:34:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:46 smithi044 ceph-mon[32404]: pgmap v365: 129 pgs: 129 active+clean; 593 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-07-26T08:34:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:46 smithi044 ceph-mon[32404]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:46.346 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-26T08:34:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:46 smithi116 ceph-mon[37097]: pgmap v365: 129 pgs: 129 active+clean; 593 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-07-26T08:34:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:46 smithi116 ceph-mon[37097]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:46 smithi171 ceph-mon[37789]: pgmap v365: 129 pgs: 129 active+clean; 593 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-07-26T08:34:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:46 smithi171 ceph-mon[37789]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:46.835 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:48 smithi044 ceph-mon[32404]: pgmap v366: 129 pgs: 129 active+clean; 594 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-07-26T08:34:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:48 smithi116 ceph-mon[37097]: pgmap v366: 129 pgs: 129 active+clean; 594 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-07-26T08:34:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:48 smithi171 ceph-mon[37789]: pgmap v366: 129 pgs: 129 active+clean; 594 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 -- 172.21.15.44:0/1210655327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 msgr2=0x7fbe5c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 --2- 172.21.15.44:0/1210655327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c10f740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009900 tx=0x7fbe4802a410 comp rx=0 tx=0).stop 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 -- 172.21.15.44:0/1210655327 shutdown_connections 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 --2- 172.21.15.44:0/1210655327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c10fc80 0x7fbe5c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 --2- 172.21.15.44:0/1210655327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c10f740 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 --2- 172.21.15.44:0/1210655327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c1099f0 0x7fbe5c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 -- 172.21.15.44:0/1210655327 >> 172.21.15.44:0/1210655327 conn(0x7fbe5c078ff0 msgr2=0x7fbe5c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:49.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 -- 172.21.15.44:0/1210655327 shutdown_connections 2024-07-26T08:34:49.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.249+0000 7fbe62200640 1 -- 172.21.15.44:0/1210655327 wait complete. 2024-07-26T08:34:49.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.251+0000 7fbe62200640 1 Processor -- start 2024-07-26T08:34:49.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.251+0000 7fbe62200640 1 -- start start 2024-07-26T08:34:49.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe62200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c1099f0 0x7fbe5c1a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:49.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe62200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:49.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe62200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c10fc80 0x7fbe5c1a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:49.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe62200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c119990 con 0x7fbe5c1099f0 2024-07-26T08:34:49.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe62200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c119810 con 0x7fbe5c10f330 2024-07-26T08:34:49.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe62200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c119b10 con 0x7fbe5c10fc80 2024-07-26T08:34:49.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe60e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c1099f0 0x7fbe5c1a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:49.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe5be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:49.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe5be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c1a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:51980/0 (socket says 172.21.15.44:51980) 2024-07-26T08:34:49.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe5be00640 1 -- 172.21.15.44:0/4109763297 learned_addr learned my addr 172.21.15.44:0/4109763297 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:49.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.252+0000 7fbe61800640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c10fc80 0x7fbe5c1a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe5be00640 1 -- 172.21.15.44:0/4109763297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c10fc80 msgr2=0x7fbe5c1a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe5be00640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c10fc80 0x7fbe5c1a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe5be00640 1 -- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c1099f0 msgr2=0x7fbe5c1a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe5be00640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c1099f0 0x7fbe5c1a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe5be00640 1 -- 172.21.15.44:0/4109763297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c1a62b0 con 0x7fbe5c10f330 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe61800640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c10fc80 0x7fbe5c1a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe60e00640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c1099f0 0x7fbe5c1a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.253+0000 7fbe5be00640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c1a08e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbe48005ab0 tx=0x7fbe48003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.254+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4803f070 con 0x7fbe5c10f330 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.254+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5c1a6540 con 0x7fbe5c10f330 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.254+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe48031b50 con 0x7fbe5c10f330 2024-07-26T08:34:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.254+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe48035030 con 0x7fbe5c10f330 2024-07-26T08:34:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.254+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5c1acd00 con 0x7fbe5c10f330 2024-07-26T08:34:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.255+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe5c109e00 con 0x7fbe5c10f330 2024-07-26T08:34:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.258+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fbe48042070 con 0x7fbe5c10f330 2024-07-26T08:34:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.259+0000 7fbe59600640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbe440b3d30 0x7fbe440b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.260+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fbe48104a10 con 0x7fbe5c10f330 2024-07-26T08:34:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.260+0000 7fbe60e00640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbe440b3d30 0x7fbe440b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:49.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.261+0000 7fbe60e00640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbe440b3d30 0x7fbe440b6240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe50009fc0 tx=0x7fbe50005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:49.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.261+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbe480cb3a0 con 0x7fbe5c10f330 2024-07-26T08:34:49.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.494+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fbe5c063230 con 0x7fbe440b3d30 2024-07-26T08:34:49.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.507+0000 7fbe59600640 1 -- 172.21.15.44:0/4109763297 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fbe5c063230 con 0x7fbe440b3d30 2024-07-26T08:34:49.508 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/qa/sv_0/399b83c0-a0e3-42e2-9910-2b99311038e3 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbe440b3d30 msgr2=0x7fbe440b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbe440b3d30 0x7fbe440b6240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe50009fc0 tx=0x7fbe50005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 msgr2=0x7fbe5c1a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c1a08e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbe48005ab0 tx=0x7fbe48003b90 comp rx=0 tx=0).stop 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 shutdown_connections 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbe440b3d30 0x7fbe440b6240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbe5c10fc80 0x7fbe5c1a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe5c10f330 0x7fbe5c1a08e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 --2- 172.21.15.44:0/4109763297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbe5c1099f0 0x7fbe5c1a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 >> 172.21.15.44:0/4109763297 conn(0x7fbe5c078ff0 msgr2=0x7fbe5c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:49.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 shutdown_connections 2024-07-26T08:34:49.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:49.510+0000 7fbe62200640 1 -- 172.21.15.44:0/4109763297 wait complete. 2024-07-26T08:34:50.365 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-26T08:34:50.448 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:50 smithi044 ceph-mon[32404]: pgmap v367: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-07-26T08:34:50.449 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:50 smithi044 ceph-mon[32404]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:50 smithi116 ceph-mon[37097]: pgmap v367: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-07-26T08:34:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:50 smithi116 ceph-mon[37097]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:50 smithi171 ceph-mon[37789]: pgmap v367: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-07-26T08:34:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:50 smithi171 ceph-mon[37789]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:50.871 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:52 smithi044 ceph-mon[32404]: pgmap v368: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-07-26T08:34:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:52 smithi116 ceph-mon[37097]: pgmap v368: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-07-26T08:34:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:52 smithi171 ceph-mon[37789]: pgmap v368: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 -- 172.21.15.44:0/4099107394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c104ce0 msgr2=0x7ff32c10d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 --2- 172.21.15.44:0/4099107394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c104ce0 0x7ff32c10d320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff318009900 tx=0x7ff31802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 -- 172.21.15.44:0/4099107394 shutdown_connections 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 --2- 172.21.15.44:0/4099107394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c10d860 0x7ff32c10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 --2- 172.21.15.44:0/4099107394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c104ce0 0x7ff32c10d320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 --2- 172.21.15.44:0/4099107394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff32c1042b0 0x7ff32c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 -- 172.21.15.44:0/4099107394 >> 172.21.15.44:0/4099107394 conn(0x7ff32c079160 msgr2=0x7ff32c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 -- 172.21.15.44:0/4099107394 shutdown_connections 2024-07-26T08:34:53.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.205+0000 7ff331200640 1 -- 172.21.15.44:0/4099107394 wait complete. 2024-07-26T08:34:53.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.206+0000 7ff331200640 1 Processor -- start 2024-07-26T08:34:53.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.206+0000 7ff331200640 1 -- start start 2024-07-26T08:34:53.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff331200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 0x7ff32c1021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff331200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff32c104ce0 0x7ff32c102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff331200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c10d860 0x7ff32c102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff331200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c1134b0 con 0x7ff32c10d860 2024-07-26T08:34:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff331200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c113330 con 0x7ff32c1042b0 2024-07-26T08:34:53.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff331200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c113630 con 0x7ff32c104ce0 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff32a800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c10d860 0x7ff32c102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff329e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 0x7ff32c1021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff329e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 0x7ff32c1021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:41878/0 (socket says 172.21.15.44:41878) 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff32a800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c10d860 0x7ff32c102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41568/0 (socket says 172.21.15.44:41568) 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff329e00640 1 -- 172.21.15.44:0/189298850 learned_addr learned my addr 172.21.15.44:0/189298850 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.207+0000 7ff329400640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff32c104ce0 0x7ff32c102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:53.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.208+0000 7ff329e00640 1 -- 172.21.15.44:0/189298850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff32c104ce0 msgr2=0x7ff32c102700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.208+0000 7ff329e00640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff32c104ce0 0x7ff32c102700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.208+0000 7ff329e00640 1 -- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c10d860 msgr2=0x7ff32c102c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.208+0000 7ff329e00640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c10d860 0x7ff32c102c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.208+0000 7ff329e00640 1 -- 172.21.15.44:0/189298850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c1a8160 con 0x7ff32c1042b0 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.208+0000 7ff329e00640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 0x7ff32c1021c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009d20 tx=0x7ff31c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.209+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c010630 con 0x7ff32c1042b0 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.209+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff31c010c80 con 0x7ff32c1042b0 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.209+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c1a8450 con 0x7ff32c1042b0 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.209+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c018d50 con 0x7ff32c1042b0 2024-07-26T08:34:53.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.209+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c1a8960 con 0x7ff32c1042b0 2024-07-26T08:34:53.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.210+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff32c1046c0 con 0x7ff32c1042b0 2024-07-26T08:34:53.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.212+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7ff31c01f020 con 0x7ff32c1042b0 2024-07-26T08:34:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.214+0000 7ff322a00640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3100b3cb0 0x7ff3100b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.214+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff31c0e47c0 con 0x7ff32c1042b0 2024-07-26T08:34:53.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.214+0000 7ff329400640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3100b3cb0 0x7ff3100b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:53.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.215+0000 7ff329400640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3100b3cb0 0x7ff3100b61c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff318031440 tx=0x7ff318005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:53.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.218+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff31c0ab7a0 con 0x7ff32c1042b0 2024-07-26T08:34:53.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.449+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff32c1069b0 con 0x7ff3100b3cb0 2024-07-26T08:34:53.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.487+0000 7ff322a00640 1 -- 172.21.15.44:0/189298850 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff32c1069b0 con 0x7ff3100b3cb0 2024-07-26T08:34:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3100b3cb0 msgr2=0x7ff3100b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3100b3cb0 0x7ff3100b61c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff318031440 tx=0x7ff318005a20 comp rx=0 tx=0).stop 2024-07-26T08:34:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 msgr2=0x7ff32c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:53.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 0x7ff32c1021c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009d20 tx=0x7ff31c00c6a0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 shutdown_connections 2024-07-26T08:34:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3100b3cb0 0x7ff3100b61c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff32c104ce0 0x7ff32c102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff32c10d860 0x7ff32c102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 --2- 172.21.15.44:0/189298850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff32c1042b0 0x7ff32c1021c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:53.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.490+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 >> 172.21.15.44:0/189298850 conn(0x7ff32c079160 msgr2=0x7ff32c1062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:53.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.491+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 shutdown_connections 2024-07-26T08:34:53.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:53.491+0000 7ff331200640 1 -- 172.21.15.44:0/189298850 wait complete. 2024-07-26T08:34:54.369 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-26T08:34:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:54 smithi044 ceph-mon[32404]: pgmap v369: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.5 KiB/s wr, 11 op/s 2024-07-26T08:34:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:54 smithi044 ceph-mon[32404]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:54 smithi116 ceph-mon[37097]: pgmap v369: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.5 KiB/s wr, 11 op/s 2024-07-26T08:34:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:54 smithi116 ceph-mon[37097]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:54 smithi171 ceph-mon[37789]: pgmap v369: 129 pgs: 129 active+clean; 608 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.5 KiB/s wr, 11 op/s 2024-07-26T08:34:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:54 smithi171 ceph-mon[37789]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:54.837 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:34:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:56 smithi044 ceph-mon[32404]: pgmap v370: 129 pgs: 129 active+clean; 613 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-07-26T08:34:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:56 smithi116 ceph-mon[37097]: pgmap v370: 129 pgs: 129 active+clean; 613 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-07-26T08:34:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:56 smithi171 ceph-mon[37789]: pgmap v370: 129 pgs: 129 active+clean; 613 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-07-26T08:34:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 -- 172.21.15.44:0/418821518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd40751e0 msgr2=0x7f3fd40755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 --2- 172.21.15.44:0/418821518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd40751e0 0x7f3fd40755f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009900 tx=0x7f3fc802a410 comp rx=0 tx=0).stop 2024-07-26T08:34:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 -- 172.21.15.44:0/418821518 shutdown_connections 2024-07-26T08:34:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 --2- 172.21.15.44:0/418821518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd4075b30 0x7f3fd410f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 --2- 172.21.15.44:0/418821518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd410f940 0x7f3fd4111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 --2- 172.21.15.44:0/418821518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd40751e0 0x7f3fd40755f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 -- 172.21.15.44:0/418821518 >> 172.21.15.44:0/418821518 conn(0x7f3fd40fee70 msgr2=0x7f3fd41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:57.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 -- 172.21.15.44:0/418821518 shutdown_connections 2024-07-26T08:34:57.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.137+0000 7f3fdac00640 1 -- 172.21.15.44:0/418821518 wait complete. 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.138+0000 7f3fdac00640 1 Processor -- start 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 -- start start 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 0x7f3fd419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd4075b30 0x7f3fd419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd410f940 0x7f3fd41a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd41154f0 con 0x7f3fd4075b30 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd4115370 con 0x7f3fd410f940 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fdac00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd4115670 con 0x7f3fd40751e0 2024-07-26T08:34:57.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.139+0000 7f3fd9800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 0x7f3fd419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:57.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd8e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd4075b30 0x7f3fd419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:57.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 0x7f3fd419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:55062/0 (socket says 172.21.15.44:55062) 2024-07-26T08:34:57.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 -- 172.21.15.44:0/103286740 learned_addr learned my addr 172.21.15.44:0/103286740 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:34:57.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fda200640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd410f940 0x7f3fd41a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 -- 172.21.15.44:0/103286740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd410f940 msgr2=0x7f3fd41a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd410f940 0x7f3fd41a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 -- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd4075b30 msgr2=0x7f3fd419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd4075b30 0x7f3fd419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd9800640 1 -- 172.21.15.44:0/103286740 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd41a1e90 con 0x7f3fd40751e0 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.140+0000 7f3fd8e00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd4075b30 0x7f3fd419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.141+0000 7f3fd9800640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 0x7f3fd419bf20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8005940 tx=0x7f3fc80040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.141+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc803f030 con 0x7f3fd40751e0 2024-07-26T08:34:57.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.141+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd41a2120 con 0x7f3fd40751e0 2024-07-26T08:34:57.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.141+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fc80044c0 con 0x7f3fd40751e0 2024-07-26T08:34:57.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.141+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8035070 con 0x7f3fd40751e0 2024-07-26T08:34:57.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.141+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd41a88e0 con 0x7f3fd40751e0 2024-07-26T08:34:57.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.142+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd40755f0 con 0x7f3fd40751e0 2024-07-26T08:34:57.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.147+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f3fc8042030 con 0x7f3fd40751e0 2024-07-26T08:34:57.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.148+0000 7f3fd2000640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3fbc0b3d30 0x7f3fbc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:34:57.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.149+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f3fc8104850 con 0x7f3fd40751e0 2024-07-26T08:34:57.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.149+0000 7f3fd8e00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3fbc0b3d30 0x7f3fbc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:34:57.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.149+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3fc81346a0 con 0x7f3fd40751e0 2024-07-26T08:34:57.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.150+0000 7f3fd8e00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3fbc0b3d30 0x7f3fbc0b6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4101680 tx=0x7f3fc0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:34:57.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.385+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f3fd4109bf0 con 0x7f3fbc0b3d30 2024-07-26T08:34:57.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.393+0000 7f3fd2000640 1 -- 172.21.15.44:0/103286740 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3fd4109bf0 con 0x7f3fbc0b3d30 2024-07-26T08:34:57.394 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/qa/sv_1/b30a7374-b969-4a02-9f96-a8a2cec18bd0 2024-07-26T08:34:57.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.395+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3fbc0b3d30 msgr2=0x7f3fbc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:57.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.395+0000 7f3fdac00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3fbc0b3d30 0x7f3fbc0b6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4101680 tx=0x7f3fc0005f90 comp rx=0 tx=0).stop 2024-07-26T08:34:57.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.395+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 msgr2=0x7f3fd419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:34:57.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.395+0000 7f3fdac00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 0x7f3fd419bf20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8005940 tx=0x7f3fc80040c0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 shutdown_connections 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3fbc0b3d30 0x7f3fbc0b6240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fd410f940 0x7f3fd41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3fd4075b30 0x7f3fd419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 --2- 172.21.15.44:0/103286740 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3fd40751e0 0x7f3fd419bf20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 >> 172.21.15.44:0/103286740 conn(0x7f3fd40fee70 msgr2=0x7f3fd41094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:34:57.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 shutdown_connections 2024-07-26T08:34:57.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:34:57.396+0000 7f3fdac00640 1 -- 172.21.15.44:0/103286740 wait complete. 2024-07-26T08:34:57.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:58 smithi044 ceph-mon[32404]: pgmap v371: 129 pgs: 129 active+clean; 613 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-07-26T08:34:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:58 smithi044 ceph-mon[32404]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:34:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:34:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:58 smithi116 ceph-mon[37097]: pgmap v371: 129 pgs: 129 active+clean; 613 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-07-26T08:34:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:58 smithi116 ceph-mon[37097]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:34:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:58 smithi171 ceph-mon[37789]: pgmap v371: 129 pgs: 129 active+clean; 613 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-07-26T08:34:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:58 smithi171 ceph-mon[37789]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:34:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:34:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:34:59.148 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:00 smithi044 ceph-mon[32404]: pgmap v372: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-07-26T08:35:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:00 smithi116 ceph-mon[37097]: pgmap v372: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-07-26T08:35:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:00 smithi171 ceph-mon[37789]: pgmap v372: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-07-26T08:35:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:01.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 -- 172.21.15.44:0/2703210090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0108d40 msgr2=0x7f9ab0109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 --2- 172.21.15.44:0/2703210090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0108d40 0x7f9ab0109150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009900 tx=0x7f9a9c02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 -- 172.21.15.44:0/2703210090 shutdown_connections 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 --2- 172.21.15.44:0/2703210090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 0x7f9ab010fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 --2- 172.21.15.44:0/2703210090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0108d40 0x7f9ab0109150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 --2- 172.21.15.44:0/2703210090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab0103400 0x7f9ab0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 -- 172.21.15.44:0/2703210090 >> 172.21.15.44:0/2703210090 conn(0x7f9ab00fee70 msgr2=0x7f9ab01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 -- 172.21.15.44:0/2703210090 shutdown_connections 2024-07-26T08:35:01.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.461+0000 7f9ab6600640 1 -- 172.21.15.44:0/2703210090 wait complete. 2024-07-26T08:35:01.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.462+0000 7f9ab6600640 1 Processor -- start 2024-07-26T08:35:01.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.462+0000 7f9ab6600640 1 -- start start 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 0x7f9ab0075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab0108d40 0x7f9ab0075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 0x7f9ab0075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab6600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab0113370 con 0x7f9ab0103400 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab6600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01131f0 con 0x7f9ab0108d40 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab6600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01134f0 con 0x7f9ab0109690 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab5200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 0x7f9ab0075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab5200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 0x7f9ab0075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41620/0 (socket says 172.21.15.44:41620) 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab5200640 1 -- 172.21.15.44:0/1034406017 learned_addr learned my addr 172.21.15.44:0/1034406017 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9ab5c00640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 0x7f9ab0075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:01.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.463+0000 7f9aafe00640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab0108d40 0x7f9ab0075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:01.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5200640 1 -- 172.21.15.44:0/1034406017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 msgr2=0x7f9ab0075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:01.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5200640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 0x7f9ab0075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5200640 1 -- 172.21.15.44:0/1034406017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab0108d40 msgr2=0x7f9ab0075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:01.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5200640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab0108d40 0x7f9ab0075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5200640 1 -- 172.21.15.44:0/1034406017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab01a7fc0 con 0x7f9ab0103400 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5c00640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 0x7f9ab0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.464+0000 7f9ab5200640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 0x7f9ab0075140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400ea60 tx=0x7f9aa400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.465+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa400cde0 con 0x7f9ab0103400 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.465+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa40035f0 con 0x7f9ab0103400 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.465+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab01a82b0 con 0x7f9ab0103400 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.465+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4010690 con 0x7f9ab0103400 2024-07-26T08:35:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.465+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab01a87c0 con 0x7f9ab0103400 2024-07-26T08:35:01.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.466+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab0103810 con 0x7f9ab0103400 2024-07-26T08:35:01.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.469+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f9aa40037a0 con 0x7f9ab0103400 2024-07-26T08:35:01.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.471+0000 7f9aad600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9a980b3cb0 0x7f9a980b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:01.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.471+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9aa40e47c0 con 0x7f9ab0103400 2024-07-26T08:35:01.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.471+0000 7f9aafe00640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9a980b3cb0 0x7f9a980b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:01.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.472+0000 7f9aafe00640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9a980b3cb0 0x7f9a980b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c02a9e0 tx=0x7f9a9c005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:01.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.473+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9aa4055600 con 0x7f9ab0103400 2024-07-26T08:35:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:01.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.718+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ab0076440 con 0x7f9ab0103400 2024-07-26T08:35:01.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.720+0000 7f9aad600640 1 -- 172.21.15.44:0/1034406017 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f9aa40ab7a0 con 0x7f9ab0103400 2024-07-26T08:35:01.723 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:01.724 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"btime":"2024-07-26T08:34:37:118441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:34:36.302968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24661,24668,24679,24686,24701]},"id":1}]} 2024-07-26T08:35:01.724 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:35:01.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.723+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9a980b3cb0 msgr2=0x7f9a980b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:01.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.723+0000 7f9ab6600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9a980b3cb0 0x7f9a980b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c02a9e0 tx=0x7f9a9c005a20 comp rx=0 tx=0).stop 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.723+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 msgr2=0x7f9ab0075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.723+0000 7f9ab6600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 0x7f9ab0075140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400ea60 tx=0x7f9aa400ef20 comp rx=0 tx=0).stop 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 shutdown_connections 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9a980b3cb0 0x7f9a980b61c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ab0109690 0x7f9ab0075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab0108d40 0x7f9ab0075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 --2- 172.21.15.44:0/1034406017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab0103400 0x7f9ab0075140 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 >> 172.21.15.44:0/1034406017 conn(0x7f9ab00fee70 msgr2=0x7f9ab0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 shutdown_connections 2024-07-26T08:35:01.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:01.724+0000 7f9ab6600640 1 -- 172.21.15.44:0/1034406017 wait complete. 2024-07-26T08:35:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:35:02.603 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:35:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:02 smithi044 ceph-mon[32404]: pgmap v373: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-07-26T08:35:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:02 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1034406017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:35:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:02 smithi116 ceph-mon[37097]: pgmap v373: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-07-26T08:35:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:02 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1034406017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:35:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:02 smithi171 ceph-mon[37789]: pgmap v373: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-07-26T08:35:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:02 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1034406017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:35:03.098 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:04 smithi044 ceph-mon[32404]: pgmap v374: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-07-26T08:35:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:04 smithi116 ceph-mon[37097]: pgmap v374: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-07-26T08:35:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:04 smithi171 ceph-mon[37789]: pgmap v374: 129 pgs: 129 active+clean; 630 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 -- 172.21.15.44:0/2755204678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae010d840 msgr2=0x7fdae010fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 --2- 172.21.15.44:0/2755204678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae010d840 0x7fdae010fd30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009ca0 tx=0x7fdad802a8b0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 -- 172.21.15.44:0/2755204678 shutdown_connections 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 --2- 172.21.15.44:0/2755204678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae010d840 0x7fdae010fd30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 --2- 172.21.15.44:0/2755204678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae0104d30 0x7fdae010d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 --2- 172.21.15.44:0/2755204678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdae01043e0 0x7fdae01047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 -- 172.21.15.44:0/2755204678 >> 172.21.15.44:0/2755204678 conn(0x7fdae0079160 msgr2=0x7fdae0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:05.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 -- 172.21.15.44:0/2755204678 shutdown_connections 2024-07-26T08:35:05.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.428+0000 7fdae6e00640 1 -- 172.21.15.44:0/2755204678 wait complete. 2024-07-26T08:35:05.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 Processor -- start 2024-07-26T08:35:05.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 -- start start 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdae01043e0 0x7fdae0102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 0x7fdae0102660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae010d840 0x7fdae0102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdae01134f0 con 0x7fdae01043e0 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdae0113370 con 0x7fdae010d840 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.430+0000 7fdae6e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdae0113670 con 0x7fdae0104d30 2024-07-26T08:35:05.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdadfe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdae01043e0 0x7fdae0102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 0x7fdae0102660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 0x7fdae0102660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:36178/0 (socket says 172.21.15.44:36178) 2024-07-26T08:35:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 -- 172.21.15.44:0/4171928602 learned_addr learned my addr 172.21.15.44:0/4171928602 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:05.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdae4e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae010d840 0x7fdae0102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 -- 172.21.15.44:0/4171928602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae010d840 msgr2=0x7fdae0102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae010d840 0x7fdae0102ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 -- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdae01043e0 msgr2=0x7fdae0102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdae01043e0 0x7fdae0102120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.431+0000 7fdad7e00640 1 -- 172.21.15.44:0/4171928602 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae01a81e0 con 0x7fdae0104d30 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.432+0000 7fdad7e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 0x7fdae0102660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdac800e970 tx=0x7fdac800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.432+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac800cd80 con 0x7fdae0104d30 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.432+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac80035f0 con 0x7fdae0104d30 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.432+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae01a84d0 con 0x7fdae0104d30 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.432+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8010690 con 0x7fdae0104d30 2024-07-26T08:35:05.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.432+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae01a8a60 con 0x7fdae0104d30 2024-07-26T08:35:05.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.433+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae01047f0 con 0x7fdae0104d30 2024-07-26T08:35:05.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.436+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fdac8003aa0 con 0x7fdae0104d30 2024-07-26T08:35:05.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.438+0000 7fdadd600640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdac40b3d30 0x7fdac40b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.438+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fdac80e4450 con 0x7fdae0104d30 2024-07-26T08:35:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.439+0000 7fdadfe00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdac40b3d30 0x7fdac40b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.439+0000 7fdadfe00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdac40b3d30 0x7fdac40b6240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdae0103b70 tx=0x7fdad0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:05.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.439+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdac80ab530 con 0x7fdae0104d30 2024-07-26T08:35:05.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.648+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdae01a8ed0 con 0x7fdae0104d30 2024-07-26T08:35:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.650+0000 7fdadd600640 1 -- 172.21.15.44:0/4171928602 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7fdac80aae30 con 0x7fdae0104d30 2024-07-26T08:35:05.653 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:05.653 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:35:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdac40b3d30 msgr2=0x7fdac40b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdac40b3d30 0x7fdac40b6240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdae0103b70 tx=0x7fdad0005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 msgr2=0x7fdae0102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 0x7fdae0102660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdac800e970 tx=0x7fdac800ee30 comp rx=0 tx=0).stop 2024-07-26T08:35:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 shutdown_connections 2024-07-26T08:35:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdac40b3d30 0x7fdac40b6240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdae0104d30 0x7fdae0102660 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae010d840 0x7fdae0102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 --2- 172.21.15.44:0/4171928602 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdae01043e0 0x7fdae0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 >> 172.21.15.44:0/4171928602 conn(0x7fdae0079160 msgr2=0x7fdae00fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:05.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 shutdown_connections 2024-07-26T08:35:05.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:05.653+0000 7fdae6e00640 1 -- 172.21.15.44:0/4171928602 wait complete. 2024-07-26T08:35:06.532 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T08:35:06.532 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T08:35:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:06 smithi044 ceph-mon[32404]: pgmap v375: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-07-26T08:35:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:06 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4171928602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:06 smithi116 ceph-mon[37097]: pgmap v375: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-07-26T08:35:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:06 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4171928602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:06 smithi171 ceph-mon[37789]: pgmap v375: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-07-26T08:35:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:06 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4171928602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:07.046 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:08 smithi044 ceph-mon[32404]: pgmap v376: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-26T08:35:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:08 smithi116 ceph-mon[37097]: pgmap v376: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-26T08:35:08.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:08 smithi171 ceph-mon[37789]: pgmap v376: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-26T08:35:09.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.399+0000 7f4ba0a00640 1 -- 172.21.15.44:0/2869200235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c108d20 msgr2=0x7f4b9c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:09.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.399+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/2869200235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c108d20 0x7f4b9c109130 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009860 tx=0x7f4b8402a340 comp rx=0 tx=0).stop 2024-07-26T08:35:09.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.399+0000 7f4ba0a00640 1 -- 172.21.15.44:0/2869200235 shutdown_connections 2024-07-26T08:35:09.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.399+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/2869200235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b9c109670 0x7f4b9c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.399+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/2869200235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c108d20 0x7f4b9c109130 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.399+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/2869200235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c1033e0 0x7f4b9c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.400+0000 7f4ba0a00640 1 -- 172.21.15.44:0/2869200235 >> 172.21.15.44:0/2869200235 conn(0x7f4b9c0fee30 msgr2=0x7f4b9c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:09.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.400+0000 7f4ba0a00640 1 -- 172.21.15.44:0/2869200235 shutdown_connections 2024-07-26T08:35:09.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.400+0000 7f4ba0a00640 1 -- 172.21.15.44:0/2869200235 wait complete. 2024-07-26T08:35:09.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.401+0000 7f4ba0a00640 1 Processor -- start 2024-07-26T08:35:09.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 -- start start 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c1033e0 0x7f4b9c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 0x7f4b9c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b9c109670 0x7f4b9c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c113300 con 0x7f4b9c108d20 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c113180 con 0x7f4b9c1033e0 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.402+0000 7f4ba0a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c113480 con 0x7f4b9c109670 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 0x7f4b9c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:09.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b9b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c1033e0 0x7f4b9c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:09.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 0x7f4b9c075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42374/0 (socket says 172.21.15.44:42374) 2024-07-26T08:35:09.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 -- 172.21.15.44:0/1400648472 learned_addr learned my addr 172.21.15.44:0/1400648472 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:09.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b9be00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b9c109670 0x7f4b9c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 -- 172.21.15.44:0/1400648472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b9c109670 msgr2=0x7f4b9c075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b9c109670 0x7f4b9c075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 -- 172.21.15.44:0/1400648472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c1033e0 msgr2=0x7f4b9c075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c1033e0 0x7f4b9c075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b93e00640 1 -- 172.21.15.44:0/1400648472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9c1a7fd0 con 0x7f4b9c108d20 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.403+0000 7f4b9b400640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c1033e0 0x7f4b9c075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:35:09.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.404+0000 7f4b93e00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 0x7f4b9c075680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4b84035040 tx=0x7f4b84032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:09.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.404+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84037040 con 0x7f4b9c108d20 2024-07-26T08:35:09.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.404+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9c1a8260 con 0x7f4b9c108d20 2024-07-26T08:35:09.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.404+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b84030c10 con 0x7f4b9c108d20 2024-07-26T08:35:09.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.404+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840035f0 con 0x7f4b9c108d20 2024-07-26T08:35:09.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.404+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9c1a86e0 con 0x7f4b9c108d20 2024-07-26T08:35:09.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.405+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9c1037f0 con 0x7f4b9c108d20 2024-07-26T08:35:09.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.408+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f4b84043020 con 0x7f4b9c108d20 2024-07-26T08:35:09.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.409+0000 7f4b98c00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b800b3cb0 0x7f4b800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:09.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.410+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4b84103920 con 0x7f4b9c108d20 2024-07-26T08:35:09.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.410+0000 7f4b9b400640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b800b3cb0 0x7f4b800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:09.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.411+0000 7f4b9b400640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b800b3cb0 0x7f4b800b61c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0096c0 tx=0x7f4b8c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:09.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.412+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b8404a060 con 0x7f4b9c108d20 2024-07-26T08:35:09.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.613+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4b9c063230 con 0x7f4b800b3cb0 2024-07-26T08:35:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.640+0000 7f4b98c00640 1 -- 172.21.15.44:0/1400648472 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340684 (secure 0 0 0) 0x7f4b9c063230 con 0x7f4b800b3cb0 2024-07-26T08:35:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:09.650 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:35:09.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b800b3cb0 msgr2=0x7f4b800b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b800b3cb0 0x7f4b800b61c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0096c0 tx=0x7f4b8c009260 comp rx=0 tx=0).stop 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 msgr2=0x7f4b9c075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 0x7f4b9c075680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4b84035040 tx=0x7f4b84032d50 comp rx=0 tx=0).stop 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 shutdown_connections 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4b800b3cb0 0x7f4b800b61c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b9c109670 0x7f4b9c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4b9c108d20 0x7f4b9c075680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 --2- 172.21.15.44:0/1400648472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b9c1033e0 0x7f4b9c075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 >> 172.21.15.44:0/1400648472 conn(0x7f4b9c0fee30 msgr2=0x7f4b9c101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:09.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.648+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 shutdown_connections 2024-07-26T08:35:09.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:09.649+0000 7f4ba0a00640 1 -- 172.21.15.44:0/1400648472 wait complete. 2024-07-26T08:35:10.526 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-07-26T08:35:08.800418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663188,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":476,"num_read_kb":347,"num_write":376,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365716,"kb_used_data":12404,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576108,"statfs":{"total":1151940427776,"available":1151565934592,"internally_reserved":0,"allocated":12701696,"data_stored":8434214,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35037,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":101,"num_write":18,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005666"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:06.191997+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:35:06.191997+0000","last_peered":"2024-07-26T08:35:06.191997+0000","last_clean":"2024-07-26T08:35:06.191997+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:35:06.191997+0000","last_undegraded":"2024-07-26T08:35:06.191997+0000","last_fullsized":"2024-07-26T08:35:06.191997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591297+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:34:14.591297+0000","last_peered":"2024-07-26T08:34:14.591297+0000","last_clean":"2024-07-26T08:34:14.591297+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:34:14.591297+0000","last_undegraded":"2024-07-26T08:34:14.591297+0000","last_fullsized":"2024-07-26T08:34:14.591297+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965646+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:34:13.965646+0000","last_peered":"2024-07-26T08:34:13.965646+0000","last_clean":"2024-07-26T08:34:13.965646+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:34:13.965646+0000","last_undegraded":"2024-07-26T08:34:13.965646+0000","last_fullsized":"2024-07-26T08:34:13.965646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:29.071793+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:29.071793+0000","last_peered":"2024-07-26T08:34:29.071793+0000","last_clean":"2024-07-26T08:34:29.071793+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:29.071793+0000","last_undegraded":"2024-07-26T08:34:29.071793+0000","last_fullsized":"2024-07-26T08:34:29.071793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.968858+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:34:13.968858+0000","last_peered":"2024-07-26T08:34:13.968858+0000","last_clean":"2024-07-26T08:34:13.968858+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:34:13.968858+0000","last_undegraded":"2024-07-26T08:34:13.968858+0000","last_fullsized":"2024-07-26T08:34:13.968858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.593163+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:34:14.593163+0000","last_peered":"2024-07-26T08:34:14.593163+0000","last_clean":"2024-07-26T08:34:14.593163+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:34:14.593163+0000","last_undegraded":"2024-07-26T08:34:14.593163+0000","last_fullsized":"2024-07-26T08:34:14.593163+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108513+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:34:32.108513+0000","last_peered":"2024-07-26T08:34:32.108513+0000","last_clean":"2024-07-26T08:34:32.108513+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:34:32.108513+0000","last_undegraded":"2024-07-26T08:34:32.108513+0000","last_fullsized":"2024-07-26T08:34:32.108513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952853+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:34:13.952853+0000","last_peered":"2024-07-26T08:34:13.952853+0000","last_clean":"2024-07-26T08:34:13.952853+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:34:13.952853+0000","last_undegraded":"2024-07-26T08:34:13.952853+0000","last_fullsized":"2024-07-26T08:34:13.952853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955297+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:34:13.955297+0000","last_peered":"2024-07-26T08:34:13.955297+0000","last_clean":"2024-07-26T08:34:13.955297+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:34:13.955297+0000","last_undegraded":"2024-07-26T08:34:13.955297+0000","last_fullsized":"2024-07-26T08:34:13.955297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096019+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:34:31.096019+0000","last_peered":"2024-07-26T08:34:31.096019+0000","last_clean":"2024-07-26T08:34:31.096019+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:34:31.096019+0000","last_undegraded":"2024-07-26T08:34:31.096019+0000","last_fullsized":"2024-07-26T08:34:31.096019+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953629+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:34:13.953629+0000","last_peered":"2024-07-26T08:34:13.953629+0000","last_clean":"2024-07-26T08:34:13.953629+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:34:13.953629+0000","last_undegraded":"2024-07-26T08:34:13.953629+0000","last_fullsized":"2024-07-26T08:34:13.953629+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954807+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:34:13.954807+0000","last_peered":"2024-07-26T08:34:13.954807+0000","last_clean":"2024-07-26T08:34:13.954807+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:34:13.954807+0000","last_undegraded":"2024-07-26T08:34:13.954807+0000","last_fullsized":"2024-07-26T08:34:13.954807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252879+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:34:14.252879+0000","last_peered":"2024-07-26T08:34:14.252879+0000","last_clean":"2024-07-26T08:34:14.252879+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:34:14.252879+0000","last_undegraded":"2024-07-26T08:34:14.252879+0000","last_fullsized":"2024-07-26T08:34:14.252879+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:28.059522+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:34:28.059522+0000","last_peered":"2024-07-26T08:34:28.059522+0000","last_clean":"2024-07-26T08:34:28.059522+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:34:28.059522+0000","last_undegraded":"2024-07-26T08:34:28.059522+0000","last_fullsized":"2024-07-26T08:34:28.059522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954915+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.954915+0000","last_peered":"2024-07-26T08:34:13.954915+0000","last_clean":"2024-07-26T08:34:13.954915+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:34:13.954915+0000","last_undegraded":"2024-07-26T08:34:13.954915+0000","last_fullsized":"2024-07-26T08:34:13.954915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.589885+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:34:14.589885+0000","last_peered":"2024-07-26T08:34:14.589885+0000","last_clean":"2024-07-26T08:34:14.589885+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:34:14.589885+0000","last_undegraded":"2024-07-26T08:34:14.589885+0000","last_fullsized":"2024-07-26T08:34:14.589885+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.092098+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:34:30.092098+0000","last_peered":"2024-07-26T08:34:30.092098+0000","last_clean":"2024-07-26T08:34:30.092098+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:34:30.092098+0000","last_undegraded":"2024-07-26T08:34:30.092098+0000","last_fullsized":"2024-07-26T08:34:30.092098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.417763+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:34:14.417763+0000","last_peered":"2024-07-26T08:34:14.417763+0000","last_clean":"2024-07-26T08:34:14.417763+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:34:14.417763+0000","last_undegraded":"2024-07-26T08:34:14.417763+0000","last_fullsized":"2024-07-26T08:34:14.417763+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962008+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:34:13.962008+0000","last_peered":"2024-07-26T08:34:13.962008+0000","last_clean":"2024-07-26T08:34:13.962008+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:34:13.962008+0000","last_undegraded":"2024-07-26T08:34:13.962008+0000","last_fullsized":"2024-07-26T08:34:13.962008+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097156+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:34:31.097156+0000","last_peered":"2024-07-26T08:34:31.097156+0000","last_clean":"2024-07-26T08:34:31.097156+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:34:31.097156+0000","last_undegraded":"2024-07-26T08:34:31.097156+0000","last_fullsized":"2024-07-26T08:34:31.097156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.594897+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:34:14.594897+0000","last_peered":"2024-07-26T08:34:14.594897+0000","last_clean":"2024-07-26T08:34:14.594897+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:34:14.594897+0000","last_undegraded":"2024-07-26T08:34:14.594897+0000","last_fullsized":"2024-07-26T08:34:14.594897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965086+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:34:13.965086+0000","last_peered":"2024-07-26T08:34:13.965086+0000","last_clean":"2024-07-26T08:34:13.965086+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:34:13.965086+0000","last_undegraded":"2024-07-26T08:34:13.965086+0000","last_fullsized":"2024-07-26T08:34:13.965086+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:28.060952+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:34:28.060952+0000","last_peered":"2024-07-26T08:34:28.060952+0000","last_clean":"2024-07-26T08:34:28.060952+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:34:28.060952+0000","last_undegraded":"2024-07-26T08:34:28.060952+0000","last_fullsized":"2024-07-26T08:34:28.060952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955179+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:34:13.955179+0000","last_peered":"2024-07-26T08:34:13.955179+0000","last_clean":"2024-07-26T08:34:13.955179+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:34:13.955179+0000","last_undegraded":"2024-07-26T08:34:13.955179+0000","last_fullsized":"2024-07-26T08:34:13.955179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967891+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:34:13.967891+0000","last_peered":"2024-07-26T08:34:13.967891+0000","last_clean":"2024-07-26T08:34:13.967891+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:34:13.967891+0000","last_undegraded":"2024-07-26T08:34:13.967891+0000","last_fullsized":"2024-07-26T08:34:13.967891+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108189+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:34:32.108189+0000","last_peered":"2024-07-26T08:34:32.108189+0000","last_clean":"2024-07-26T08:34:32.108189+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:34:32.108189+0000","last_undegraded":"2024-07-26T08:34:32.108189+0000","last_fullsized":"2024-07-26T08:34:32.108189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965977+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:34:13.965977+0000","last_peered":"2024-07-26T08:34:13.965977+0000","last_clean":"2024-07-26T08:34:13.965977+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:34:13.965977+0000","last_undegraded":"2024-07-26T08:34:13.965977+0000","last_fullsized":"2024-07-26T08:34:13.965977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.956980+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:34:13.956980+0000","last_peered":"2024-07-26T08:34:13.956980+0000","last_clean":"2024-07-26T08:34:13.956980+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:34:13.956980+0000","last_undegraded":"2024-07-26T08:34:13.956980+0000","last_fullsized":"2024-07-26T08:34:13.956980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'16","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:00.188886+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:35:00.188886+0000","last_peered":"2024-07-26T08:35:00.188886+0000","last_clean":"2024-07-26T08:35:00.188886+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:35:00.188886+0000","last_undegraded":"2024-07-26T08:35:00.188886+0000","last_fullsized":"2024-07-26T08:35:00.188886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590502+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:34:14.590502+0000","last_peered":"2024-07-26T08:34:14.590502+0000","last_clean":"2024-07-26T08:34:14.590502+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:34:14.590502+0000","last_undegraded":"2024-07-26T08:34:14.590502+0000","last_fullsized":"2024-07-26T08:34:14.590502+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329369+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:34:14.329369+0000","last_peered":"2024-07-26T08:34:14.329369+0000","last_clean":"2024-07-26T08:34:14.329369+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:34:14.329369+0000","last_undegraded":"2024-07-26T08:34:14.329369+0000","last_fullsized":"2024-07-26T08:34:14.329369+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108436+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:34:32.108436+0000","last_peered":"2024-07-26T08:34:32.108436+0000","last_clean":"2024-07-26T08:34:32.108436+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:34:32.108436+0000","last_undegraded":"2024-07-26T08:34:32.108436+0000","last_fullsized":"2024-07-26T08:34:32.108436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961897+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:34:13.961897+0000","last_peered":"2024-07-26T08:34:13.961897+0000","last_clean":"2024-07-26T08:34:13.961897+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:34:13.961897+0000","last_undegraded":"2024-07-26T08:34:13.961897+0000","last_fullsized":"2024-07-26T08:34:13.961897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415652+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:34:14.415652+0000","last_peered":"2024-07-26T08:34:14.415652+0000","last_clean":"2024-07-26T08:34:14.415652+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:34:14.415652+0000","last_undegraded":"2024-07-26T08:34:14.415652+0000","last_fullsized":"2024-07-26T08:34:14.415652+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.095298+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:34:30.095298+0000","last_peered":"2024-07-26T08:34:30.095298+0000","last_clean":"2024-07-26T08:34:30.095298+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:34:30.095298+0000","last_undegraded":"2024-07-26T08:34:30.095298+0000","last_fullsized":"2024-07-26T08:34:30.095298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965873+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:34:13.965873+0000","last_peered":"2024-07-26T08:34:13.965873+0000","last_clean":"2024-07-26T08:34:13.965873+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:34:13.965873+0000","last_undegraded":"2024-07-26T08:34:13.965873+0000","last_fullsized":"2024-07-26T08:34:13.965873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415523+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:34:14.415523+0000","last_peered":"2024-07-26T08:34:14.415523+0000","last_clean":"2024-07-26T08:34:14.415523+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:34:14.415523+0000","last_undegraded":"2024-07-26T08:34:14.415523+0000","last_fullsized":"2024-07-26T08:34:14.415523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:06.192930+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:35:06.192930+0000","last_peered":"2024-07-26T08:35:06.192930+0000","last_clean":"2024-07-26T08:35:06.192930+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:35:06.192930+0000","last_undegraded":"2024-07-26T08:35:06.192930+0000","last_fullsized":"2024-07-26T08:35:06.192930+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328787+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:34:14.328787+0000","last_peered":"2024-07-26T08:34:14.328787+0000","last_clean":"2024-07-26T08:34:14.328787+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:34:14.328787+0000","last_undegraded":"2024-07-26T08:34:14.328787+0000","last_fullsized":"2024-07-26T08:34:14.328787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954090+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:34:13.954090+0000","last_peered":"2024-07-26T08:34:13.954090+0000","last_clean":"2024-07-26T08:34:13.954090+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:34:13.954090+0000","last_undegraded":"2024-07-26T08:34:13.954090+0000","last_fullsized":"2024-07-26T08:34:13.954090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:06.204784+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:35:06.204784+0000","last_peered":"2024-07-26T08:35:06.204784+0000","last_clean":"2024-07-26T08:35:06.204784+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:35:06.204784+0000","last_undegraded":"2024-07-26T08:35:06.204784+0000","last_fullsized":"2024-07-26T08:35:06.204784+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328920+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:34:14.328920+0000","last_peered":"2024-07-26T08:34:14.328920+0000","last_clean":"2024-07-26T08:34:14.328920+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:34:14.328920+0000","last_undegraded":"2024-07-26T08:34:14.328920+0000","last_fullsized":"2024-07-26T08:34:14.328920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328077+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:34:14.328077+0000","last_peered":"2024-07-26T08:34:14.328077+0000","last_clean":"2024-07-26T08:34:14.328077+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:34:14.328077+0000","last_undegraded":"2024-07-26T08:34:14.328077+0000","last_fullsized":"2024-07-26T08:34:14.328077+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096346+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:31.096346+0000","last_peered":"2024-07-26T08:34:31.096346+0000","last_clean":"2024-07-26T08:34:31.096346+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:31.096346+0000","last_undegraded":"2024-07-26T08:34:31.096346+0000","last_fullsized":"2024-07-26T08:34:31.096346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953664+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.953664+0000","last_peered":"2024-07-26T08:34:13.953664+0000","last_clean":"2024-07-26T08:34:13.953664+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:34:13.953664+0000","last_undegraded":"2024-07-26T08:34:13.953664+0000","last_fullsized":"2024-07-26T08:34:13.953664+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590008+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:34:14.590008+0000","last_peered":"2024-07-26T08:34:14.590008+0000","last_clean":"2024-07-26T08:34:14.590008+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:34:14.590008+0000","last_undegraded":"2024-07-26T08:34:14.590008+0000","last_fullsized":"2024-07-26T08:34:14.590008+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.109130+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:34:32.109130+0000","last_peered":"2024-07-26T08:34:32.109130+0000","last_clean":"2024-07-26T08:34:32.109130+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:34:32.109130+0000","last_undegraded":"2024-07-26T08:34:32.109130+0000","last_fullsized":"2024-07-26T08:34:32.109130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":122,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252720+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:34:14.252720+0000","last_peered":"2024-07-26T08:34:14.252720+0000","last_clean":"2024-07-26T08:34:14.252720+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:34:14.252720+0000","last_undegraded":"2024-07-26T08:34:14.252720+0000","last_fullsized":"2024-07-26T08:34:14.252720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253123+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:34:14.253123+0000","last_peered":"2024-07-26T08:34:14.253123+0000","last_clean":"2024-07-26T08:34:14.253123+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:34:14.253123+0000","last_undegraded":"2024-07-26T08:34:14.253123+0000","last_fullsized":"2024-07-26T08:34:14.253123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253314+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:34:14.253314+0000","last_peered":"2024-07-26T08:34:14.253314+0000","last_clean":"2024-07-26T08:34:14.253314+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:34:14.253314+0000","last_undegraded":"2024-07-26T08:34:14.253314+0000","last_fullsized":"2024-07-26T08:34:14.253314+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:03.160875+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:35:03.160875+0000","last_peered":"2024-07-26T08:35:03.160875+0000","last_clean":"2024-07-26T08:35:03.160875+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:35:03.160875+0000","last_undegraded":"2024-07-26T08:35:03.160875+0000","last_fullsized":"2024-07-26T08:35:03.160875+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952616+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:34:13.952616+0000","last_peered":"2024-07-26T08:34:13.952616+0000","last_clean":"2024-07-26T08:34:13.952616+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:34:13.952616+0000","last_undegraded":"2024-07-26T08:34:13.952616+0000","last_fullsized":"2024-07-26T08:34:13.952616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253091+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:34:14.253091+0000","last_peered":"2024-07-26T08:34:14.253091+0000","last_clean":"2024-07-26T08:34:14.253091+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:34:14.253091+0000","last_undegraded":"2024-07-26T08:34:14.253091+0000","last_fullsized":"2024-07-26T08:34:14.253091+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097156+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:34:31.097156+0000","last_peered":"2024-07-26T08:34:31.097156+0000","last_clean":"2024-07-26T08:34:31.097156+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:34:31.097156+0000","last_undegraded":"2024-07-26T08:34:31.097156+0000","last_fullsized":"2024-07-26T08:34:31.097156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966970+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:34:13.966970+0000","last_peered":"2024-07-26T08:34:13.966970+0000","last_clean":"2024-07-26T08:34:13.966970+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:34:13.966970+0000","last_undegraded":"2024-07-26T08:34:13.966970+0000","last_fullsized":"2024-07-26T08:34:13.966970+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253049+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:34:14.253049+0000","last_peered":"2024-07-26T08:34:14.253049+0000","last_clean":"2024-07-26T08:34:14.253049+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:34:14.253049+0000","last_undegraded":"2024-07-26T08:34:14.253049+0000","last_fullsized":"2024-07-26T08:34:14.253049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.095902+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:34:31.095902+0000","last_peered":"2024-07-26T08:34:31.095902+0000","last_clean":"2024-07-26T08:34:31.095902+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:34:31.095902+0000","last_undegraded":"2024-07-26T08:34:31.095902+0000","last_fullsized":"2024-07-26T08:34:31.095902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953086+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:34:13.953086+0000","last_peered":"2024-07-26T08:34:13.953086+0000","last_clean":"2024-07-26T08:34:13.953086+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:34:13.953086+0000","last_undegraded":"2024-07-26T08:34:13.953086+0000","last_fullsized":"2024-07-26T08:34:13.953086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962246+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:34:13.962246+0000","last_peered":"2024-07-26T08:34:13.962246+0000","last_clean":"2024-07-26T08:34:13.962246+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:34:13.962246+0000","last_undegraded":"2024-07-26T08:34:13.962246+0000","last_fullsized":"2024-07-26T08:34:13.962246+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.061878+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:34:31.061878+0000","last_peered":"2024-07-26T08:34:31.061878+0000","last_clean":"2024-07-26T08:34:31.061878+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:34:31.061878+0000","last_undegraded":"2024-07-26T08:34:31.061878+0000","last_fullsized":"2024-07-26T08:34:31.061878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953623+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:34:13.953623+0000","last_peered":"2024-07-26T08:34:13.953623+0000","last_clean":"2024-07-26T08:34:13.953623+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:34:13.953623+0000","last_undegraded":"2024-07-26T08:34:13.953623+0000","last_fullsized":"2024-07-26T08:34:13.953623+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953319+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:34:13.953319+0000","last_peered":"2024-07-26T08:34:13.953319+0000","last_clean":"2024-07-26T08:34:13.953319+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:34:13.953319+0000","last_undegraded":"2024-07-26T08:34:13.953319+0000","last_fullsized":"2024-07-26T08:34:13.953319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096284+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:31.096284+0000","last_peered":"2024-07-26T08:34:31.096284+0000","last_clean":"2024-07-26T08:34:31.096284+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:31.096284+0000","last_undegraded":"2024-07-26T08:34:31.096284+0000","last_fullsized":"2024-07-26T08:34:31.096284+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328983+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:34:14.328983+0000","last_peered":"2024-07-26T08:34:14.328983+0000","last_clean":"2024-07-26T08:34:14.328983+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:34:14.328983+0000","last_undegraded":"2024-07-26T08:34:14.328983+0000","last_fullsized":"2024-07-26T08:34:14.328983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252467+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:34:14.252467+0000","last_peered":"2024-07-26T08:34:14.252467+0000","last_clean":"2024-07-26T08:34:14.252467+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:34:14.252467+0000","last_undegraded":"2024-07-26T08:34:14.252467+0000","last_fullsized":"2024-07-26T08:34:14.252467+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'6","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.208823+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:35:07.208823+0000","last_peered":"2024-07-26T08:35:07.208823+0000","last_clean":"2024-07-26T08:35:07.208823+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:35:07.208823+0000","last_undegraded":"2024-07-26T08:35:07.208823+0000","last_fullsized":"2024-07-26T08:35:07.208823+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.957292+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:34:13.957292+0000","last_peered":"2024-07-26T08:34:13.957292+0000","last_clean":"2024-07-26T08:34:13.957292+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:34:13.957292+0000","last_undegraded":"2024-07-26T08:34:13.957292+0000","last_fullsized":"2024-07-26T08:34:13.957292+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253417+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:34:14.253417+0000","last_peered":"2024-07-26T08:34:14.253417+0000","last_clean":"2024-07-26T08:34:14.253417+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:34:14.253417+0000","last_undegraded":"2024-07-26T08:34:14.253417+0000","last_fullsized":"2024-07-26T08:34:14.253417+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.094654+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:34:31.094654+0000","last_peered":"2024-07-26T08:34:31.094654+0000","last_clean":"2024-07-26T08:34:31.094654+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:34:31.094654+0000","last_undegraded":"2024-07-26T08:34:31.094654+0000","last_fullsized":"2024-07-26T08:34:31.094654+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961912+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:34:13.961912+0000","last_peered":"2024-07-26T08:34:13.961912+0000","last_clean":"2024-07-26T08:34:13.961912+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:34:13.961912+0000","last_undegraded":"2024-07-26T08:34:13.961912+0000","last_fullsized":"2024-07-26T08:34:13.961912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328698+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:34:14.328698+0000","last_peered":"2024-07-26T08:34:14.328698+0000","last_clean":"2024-07-26T08:34:14.328698+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:34:14.328698+0000","last_undegraded":"2024-07-26T08:34:14.328698+0000","last_fullsized":"2024-07-26T08:34:14.328698+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106688+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:34:32.106688+0000","last_peered":"2024-07-26T08:34:32.106688+0000","last_clean":"2024-07-26T08:34:32.106688+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:34:32.106688+0000","last_undegraded":"2024-07-26T08:34:32.106688+0000","last_fullsized":"2024-07-26T08:34:32.106688+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953494+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:34:13.953494+0000","last_peered":"2024-07-26T08:34:13.953494+0000","last_clean":"2024-07-26T08:34:13.953494+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:34:13.953494+0000","last_undegraded":"2024-07-26T08:34:13.953494+0000","last_fullsized":"2024-07-26T08:34:13.953494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965416+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:34:13.965416+0000","last_peered":"2024-07-26T08:34:13.965416+0000","last_clean":"2024-07-26T08:34:13.965416+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:34:13.965416+0000","last_undegraded":"2024-07-26T08:34:13.965416+0000","last_fullsized":"2024-07-26T08:34:13.965416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":161,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.167037+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:35:07.167037+0000","last_peered":"2024-07-26T08:35:07.167037+0000","last_clean":"2024-07-26T08:35:07.167037+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:35:07.167037+0000","last_undegraded":"2024-07-26T08:35:07.167037+0000","last_fullsized":"2024-07-26T08:35:07.167037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65382,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":80,"num_read_kb":104,"num_write":38,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955249+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:34:13.955249+0000","last_peered":"2024-07-26T08:34:13.955249+0000","last_clean":"2024-07-26T08:34:13.955249+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:34:13.955249+0000","last_undegraded":"2024-07-26T08:34:13.955249+0000","last_fullsized":"2024-07-26T08:34:13.955249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952954+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:34:13.952954+0000","last_peered":"2024-07-26T08:34:13.952954+0000","last_clean":"2024-07-26T08:34:13.952954+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:34:13.952954+0000","last_undegraded":"2024-07-26T08:34:13.952954+0000","last_fullsized":"2024-07-26T08:34:13.952954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.090949+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:34:30.090949+0000","last_peered":"2024-07-26T08:34:30.090949+0000","last_clean":"2024-07-26T08:34:30.090949+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:34:30.090949+0000","last_undegraded":"2024-07-26T08:34:30.090949+0000","last_fullsized":"2024-07-26T08:34:30.090949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.969464+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:34:13.969464+0000","last_peered":"2024-07-26T08:34:13.969464+0000","last_clean":"2024-07-26T08:34:13.969464+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:34:13.969464+0000","last_undegraded":"2024-07-26T08:34:13.969464+0000","last_fullsized":"2024-07-26T08:34:13.969464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.968720+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:34:13.968720+0000","last_peered":"2024-07-26T08:34:13.968720+0000","last_clean":"2024-07-26T08:34:13.968720+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:34:13.968720+0000","last_undegraded":"2024-07-26T08:34:13.968720+0000","last_fullsized":"2024-07-26T08:34:13.968720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096430+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:34:31.096430+0000","last_peered":"2024-07-26T08:34:31.096430+0000","last_clean":"2024-07-26T08:34:31.096430+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:34:31.096430+0000","last_undegraded":"2024-07-26T08:34:31.096430+0000","last_fullsized":"2024-07-26T08:34:31.096430+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954079+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:34:13.954079+0000","last_peered":"2024-07-26T08:34:13.954079+0000","last_clean":"2024-07-26T08:34:13.954079+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:34:13.954079+0000","last_undegraded":"2024-07-26T08:34:13.954079+0000","last_fullsized":"2024-07-26T08:34:13.954079+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953258+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:34:13.953258+0000","last_peered":"2024-07-26T08:34:13.953258+0000","last_clean":"2024-07-26T08:34:13.953258+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:34:13.953258+0000","last_undegraded":"2024-07-26T08:34:13.953258+0000","last_fullsized":"2024-07-26T08:34:13.953258+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.166560+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:35:07.166560+0000","last_peered":"2024-07-26T08:35:07.166560+0000","last_clean":"2024-07-26T08:35:07.166560+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:35:07.166560+0000","last_undegraded":"2024-07-26T08:35:07.166560+0000","last_fullsized":"2024-07-26T08:35:07.166560+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.589873+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:34:14.589873+0000","last_peered":"2024-07-26T08:34:14.589873+0000","last_clean":"2024-07-26T08:34:14.589873+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:34:14.589873+0000","last_undegraded":"2024-07-26T08:34:14.589873+0000","last_fullsized":"2024-07-26T08:34:14.589873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967456+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:34:13.967456+0000","last_peered":"2024-07-26T08:34:13.967456+0000","last_clean":"2024-07-26T08:34:13.967456+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:34:13.967456+0000","last_undegraded":"2024-07-26T08:34:13.967456+0000","last_fullsized":"2024-07-26T08:34:13.967456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954424+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:34:13.954424+0000","last_peered":"2024-07-26T08:34:13.954424+0000","last_clean":"2024-07-26T08:34:13.954424+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:34:13.954424+0000","last_undegraded":"2024-07-26T08:34:13.954424+0000","last_fullsized":"2024-07-26T08:34:13.954424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096426+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:34:31.096426+0000","last_peered":"2024-07-26T08:34:31.096426+0000","last_clean":"2024-07-26T08:34:31.096426+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:34:31.096426+0000","last_undegraded":"2024-07-26T08:34:31.096426+0000","last_fullsized":"2024-07-26T08:34:31.096426+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962146+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:34:13.962146+0000","last_peered":"2024-07-26T08:34:13.962146+0000","last_clean":"2024-07-26T08:34:13.962146+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:34:13.962146+0000","last_undegraded":"2024-07-26T08:34:13.962146+0000","last_fullsized":"2024-07-26T08:34:13.962146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591240+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:34:14.591240+0000","last_peered":"2024-07-26T08:34:14.591240+0000","last_clean":"2024-07-26T08:34:14.591240+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:34:14.591240+0000","last_undegraded":"2024-07-26T08:34:14.591240+0000","last_fullsized":"2024-07-26T08:34:14.591240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:29.072713+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:34:29.072713+0000","last_peered":"2024-07-26T08:34:29.072713+0000","last_clean":"2024-07-26T08:34:29.072713+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:34:29.072713+0000","last_undegraded":"2024-07-26T08:34:29.072713+0000","last_fullsized":"2024-07-26T08:34:29.072713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.090404+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:34:30.090404+0000","last_peered":"2024-07-26T08:34:30.090404+0000","last_clean":"2024-07-26T08:34:30.090404+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:34:30.090404+0000","last_undegraded":"2024-07-26T08:34:30.090404+0000","last_fullsized":"2024-07-26T08:34:30.090404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955741+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:34:13.955741+0000","last_peered":"2024-07-26T08:34:13.955741+0000","last_clean":"2024-07-26T08:34:13.955741+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:34:13.955741+0000","last_undegraded":"2024-07-26T08:34:13.955741+0000","last_fullsized":"2024-07-26T08:34:13.955741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.964550+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:34:13.964550+0000","last_peered":"2024-07-26T08:34:13.964550+0000","last_clean":"2024-07-26T08:34:13.964550+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:34:13.964550+0000","last_undegraded":"2024-07-26T08:34:13.964550+0000","last_fullsized":"2024-07-26T08:34:13.964550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:05.164218+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:35:05.164218+0000","last_peered":"2024-07-26T08:35:05.164218+0000","last_clean":"2024-07-26T08:35:05.164218+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:35:05.164218+0000","last_undegraded":"2024-07-26T08:35:05.164218+0000","last_fullsized":"2024-07-26T08:35:05.164218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965968+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:34:13.965968+0000","last_peered":"2024-07-26T08:34:13.965968+0000","last_clean":"2024-07-26T08:34:13.965968+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:34:13.965968+0000","last_undegraded":"2024-07-26T08:34:13.965968+0000","last_fullsized":"2024-07-26T08:34:13.965968+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965484+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:34:13.965484+0000","last_peered":"2024-07-26T08:34:13.965484+0000","last_clean":"2024-07-26T08:34:13.965484+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:34:13.965484+0000","last_undegraded":"2024-07-26T08:34:13.965484+0000","last_fullsized":"2024-07-26T08:34:13.965484+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961849+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:34:13.961849+0000","last_peered":"2024-07-26T08:34:13.961849+0000","last_clean":"2024-07-26T08:34:13.961849+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:34:13.961849+0000","last_undegraded":"2024-07-26T08:34:13.961849+0000","last_fullsized":"2024-07-26T08:34:13.961849+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966223+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:34:13.966223+0000","last_peered":"2024-07-26T08:34:13.966223+0000","last_clean":"2024-07-26T08:34:13.966223+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:34:13.966223+0000","last_undegraded":"2024-07-26T08:34:13.966223+0000","last_fullsized":"2024-07-26T08:34:13.966223+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.964377+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:34:13.964377+0000","last_peered":"2024-07-26T08:34:13.964377+0000","last_clean":"2024-07-26T08:34:13.964377+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:34:13.964377+0000","last_undegraded":"2024-07-26T08:34:13.964377+0000","last_fullsized":"2024-07-26T08:34:13.964377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.969270+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:34:13.969270+0000","last_peered":"2024-07-26T08:34:13.969270+0000","last_clean":"2024-07-26T08:34:13.969270+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:34:13.969270+0000","last_undegraded":"2024-07-26T08:34:13.969270+0000","last_fullsized":"2024-07-26T08:34:13.969270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329138+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:34:14.329138+0000","last_peered":"2024-07-26T08:34:14.329138+0000","last_clean":"2024-07-26T08:34:14.329138+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:34:14.329138+0000","last_undegraded":"2024-07-26T08:34:14.329138+0000","last_fullsized":"2024-07-26T08:34:14.329138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962264+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:34:13.962264+0000","last_peered":"2024-07-26T08:34:13.962264+0000","last_clean":"2024-07-26T08:34:13.962264+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:34:13.962264+0000","last_undegraded":"2024-07-26T08:34:13.962264+0000","last_fullsized":"2024-07-26T08:34:13.962264+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966257+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:34:13.966257+0000","last_peered":"2024-07-26T08:34:13.966257+0000","last_clean":"2024-07-26T08:34:13.966257+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:34:13.966257+0000","last_undegraded":"2024-07-26T08:34:13.966257+0000","last_fullsized":"2024-07-26T08:34:13.966257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591228+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:34:14.591228+0000","last_peered":"2024-07-26T08:34:14.591228+0000","last_clean":"2024-07-26T08:34:14.591228+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:34:14.591228+0000","last_undegraded":"2024-07-26T08:34:14.591228+0000","last_fullsized":"2024-07-26T08:34:14.591228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953257+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:34:13.953257+0000","last_peered":"2024-07-26T08:34:13.953257+0000","last_clean":"2024-07-26T08:34:13.953257+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:34:13.953257+0000","last_undegraded":"2024-07-26T08:34:13.953257+0000","last_fullsized":"2024-07-26T08:34:13.953257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590501+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:34:14.590501+0000","last_peered":"2024-07-26T08:34:14.590501+0000","last_clean":"2024-07-26T08:34:14.590501+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:34:14.590501+0000","last_undegraded":"2024-07-26T08:34:14.590501+0000","last_fullsized":"2024-07-26T08:34:14.590501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952802+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:34:13.952802+0000","last_peered":"2024-07-26T08:34:13.952802+0000","last_clean":"2024-07-26T08:34:13.952802+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:34:13.952802+0000","last_undegraded":"2024-07-26T08:34:13.952802+0000","last_fullsized":"2024-07-26T08:34:13.952802+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328854+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:34:14.328854+0000","last_peered":"2024-07-26T08:34:14.328854+0000","last_clean":"2024-07-26T08:34:14.328854+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:34:14.328854+0000","last_undegraded":"2024-07-26T08:34:14.328854+0000","last_fullsized":"2024-07-26T08:34:14.328854+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961972+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:34:13.961972+0000","last_peered":"2024-07-26T08:34:13.961972+0000","last_clean":"2024-07-26T08:34:13.961972+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:34:13.961972+0000","last_undegraded":"2024-07-26T08:34:13.961972+0000","last_fullsized":"2024-07-26T08:34:13.961972+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965961+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:34:13.965961+0000","last_peered":"2024-07-26T08:34:13.965961+0000","last_clean":"2024-07-26T08:34:13.965961+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:34:13.965961+0000","last_undegraded":"2024-07-26T08:34:13.965961+0000","last_fullsized":"2024-07-26T08:34:13.965961+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.957489+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:34:13.957489+0000","last_peered":"2024-07-26T08:34:13.957489+0000","last_clean":"2024-07-26T08:34:13.957489+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:34:13.957489+0000","last_undegraded":"2024-07-26T08:34:13.957489+0000","last_fullsized":"2024-07-26T08:34:13.957489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962419+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:34:13.962419+0000","last_peered":"2024-07-26T08:34:13.962419+0000","last_clean":"2024-07-26T08:34:13.962419+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:34:13.962419+0000","last_undegraded":"2024-07-26T08:34:13.962419+0000","last_fullsized":"2024-07-26T08:34:13.962419+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952479+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:34:13.952479+0000","last_peered":"2024-07-26T08:34:13.952479+0000","last_clean":"2024-07-26T08:34:13.952479+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:34:13.952479+0000","last_undegraded":"2024-07-26T08:34:13.952479+0000","last_fullsized":"2024-07-26T08:34:13.952479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966639+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:34:13.966639+0000","last_peered":"2024-07-26T08:34:13.966639+0000","last_clean":"2024-07-26T08:34:13.966639+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:34:13.966639+0000","last_undegraded":"2024-07-26T08:34:13.966639+0000","last_fullsized":"2024-07-26T08:34:13.966639+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966925+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:34:13.966925+0000","last_peered":"2024-07-26T08:34:13.966925+0000","last_clean":"2024-07-26T08:34:13.966925+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:34:13.966925+0000","last_undegraded":"2024-07-26T08:34:13.966925+0000","last_fullsized":"2024-07-26T08:34:13.966925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328677+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:34:14.328677+0000","last_peered":"2024-07-26T08:34:14.328677+0000","last_clean":"2024-07-26T08:34:14.328677+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:34:14.328677+0000","last_undegraded":"2024-07-26T08:34:14.328677+0000","last_fullsized":"2024-07-26T08:34:14.328677+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955562+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:34:13.955562+0000","last_peered":"2024-07-26T08:34:13.955562+0000","last_clean":"2024-07-26T08:34:13.955562+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:34:13.955562+0000","last_undegraded":"2024-07-26T08:34:13.955562+0000","last_fullsized":"2024-07-26T08:34:13.955562+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253255+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:34:14.253255+0000","last_peered":"2024-07-26T08:34:14.253255+0000","last_clean":"2024-07-26T08:34:14.253255+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:34:14.253255+0000","last_undegraded":"2024-07-26T08:34:14.253255+0000","last_fullsized":"2024-07-26T08:34:14.253255+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415930+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:34:14.415930+0000","last_peered":"2024-07-26T08:34:14.415930+0000","last_clean":"2024-07-26T08:34:14.415930+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:34:14.415930+0000","last_undegraded":"2024-07-26T08:34:14.415930+0000","last_fullsized":"2024-07-26T08:34:14.415930+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953212+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:34:13.953212+0000","last_peered":"2024-07-26T08:34:13.953212+0000","last_clean":"2024-07-26T08:34:13.953212+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:34:13.953212+0000","last_undegraded":"2024-07-26T08:34:13.953212+0000","last_fullsized":"2024-07-26T08:34:13.953212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590926+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:34:14.590926+0000","last_peered":"2024-07-26T08:34:14.590926+0000","last_clean":"2024-07-26T08:34:14.590926+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:34:14.590926+0000","last_undegraded":"2024-07-26T08:34:14.590926+0000","last_fullsized":"2024-07-26T08:34:14.590926+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252550+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:34:14.252550+0000","last_peered":"2024-07-26T08:34:14.252550+0000","last_clean":"2024-07-26T08:34:14.252550+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:34:14.252550+0000","last_undegraded":"2024-07-26T08:34:14.252550+0000","last_fullsized":"2024-07-26T08:34:14.252550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415406+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:34:14.415406+0000","last_peered":"2024-07-26T08:34:14.415406+0000","last_clean":"2024-07-26T08:34:14.415406+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:34:14.415406+0000","last_undegraded":"2024-07-26T08:34:14.415406+0000","last_fullsized":"2024-07-26T08:34:14.415406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965397+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:34:13.965397+0000","last_peered":"2024-07-26T08:34:13.965397+0000","last_clean":"2024-07-26T08:34:13.965397+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:34:13.965397+0000","last_undegraded":"2024-07-26T08:34:13.965397+0000","last_fullsized":"2024-07-26T08:34:13.965397+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955486+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:34:13.955486+0000","last_peered":"2024-07-26T08:34:13.955486+0000","last_clean":"2024-07-26T08:34:13.955486+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:34:13.955486+0000","last_undegraded":"2024-07-26T08:34:13.955486+0000","last_fullsized":"2024-07-26T08:34:13.955486+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967332+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:34:13.967332+0000","last_peered":"2024-07-26T08:34:13.967332+0000","last_clean":"2024-07-26T08:34:13.967332+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:34:13.967332+0000","last_undegraded":"2024-07-26T08:34:13.967332+0000","last_fullsized":"2024-07-26T08:34:13.967332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967579+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:34:13.967579+0000","last_peered":"2024-07-26T08:34:13.967579+0000","last_clean":"2024-07-26T08:34:13.967579+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:34:13.967579+0000","last_undegraded":"2024-07-26T08:34:13.967579+0000","last_fullsized":"2024-07-26T08:34:13.967579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962404+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:34:13.962404+0000","last_peered":"2024-07-26T08:34:13.962404+0000","last_clean":"2024-07-26T08:34:13.962404+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:34:13.962404+0000","last_undegraded":"2024-07-26T08:34:13.962404+0000","last_fullsized":"2024-07-26T08:34:13.962404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72586,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":395,"num_read_kb":278,"num_write":251,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":217758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645354,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540173,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841582,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1196355,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005107,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397568,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136475,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561091,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539569,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117076,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378716,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":602127,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542240,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":601516,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":536951,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:35:10.532 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T08:35:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:10 smithi044 ceph-mon[32404]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-07-26T08:35:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:10 smithi116 ceph-mon[37097]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-07-26T08:35:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:10 smithi171 ceph-mon[37789]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-07-26T08:35:11.018 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:11 smithi044 ceph-mon[32404]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:35:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:11 smithi116 ceph-mon[37097]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:35:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:11 smithi171 ceph-mon[37789]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:35:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:12 smithi044 ceph-mon[32404]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-07-26T08:35:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:12 smithi116 ceph-mon[37097]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-07-26T08:35:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:12 smithi171 ceph-mon[37789]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-07-26T08:35:13.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 -- 172.21.15.44:0/757744908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 msgr2=0x7f0928107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:13.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 --2- 172.21.15.44:0/757744908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f0928107ff0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0910009900 tx=0x7f091002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 -- 172.21.15.44:0/757744908 shutdown_connections 2024-07-26T08:35:13.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 --2- 172.21.15.44:0/757744908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0928108530 0x7f092810ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 --2- 172.21.15.44:0/757744908 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f0928107ff0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 --2- 172.21.15.44:0/757744908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09281022a0 0x7f09281026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 -- 172.21.15.44:0/757744908 >> 172.21.15.44:0/757744908 conn(0x7f09280fdcf0 msgr2=0x7f0928100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:13.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 -- 172.21.15.44:0/757744908 shutdown_connections 2024-07-26T08:35:13.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.463+0000 7f0927200640 1 -- 172.21.15.44:0/757744908 wait complete. 2024-07-26T08:35:13.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.465+0000 7f0927200640 1 Processor -- start 2024-07-26T08:35:13.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.465+0000 7f0927200640 1 -- start start 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0927200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09281022a0 0x7f0928199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0927200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f092819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0927200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0928108530 0x7f092819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0927200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0928112210 con 0x7f0928107be0 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0927200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0928112090 con 0x7f0928108530 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0927200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0928112390 con 0x7f09281022a0 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0925400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f092819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0925e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09281022a0 0x7f0928199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0925e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09281022a0 0x7f0928199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:38542/0 (socket says 172.21.15.44:38542) 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0925e00640 1 -- 172.21.15.44:0/1278569872 learned_addr learned my addr 172.21.15.44:0/1278569872 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0925400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f092819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53740/0 (socket says 172.21.15.44:53740) 2024-07-26T08:35:13.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.466+0000 7f0926800640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0928108530 0x7f092819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:13.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0926800640 1 -- 172.21.15.44:0/1278569872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09281022a0 msgr2=0x7f0928199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:13.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0926800640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09281022a0 0x7f0928199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0926800640 1 -- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 msgr2=0x7f092819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:13.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0926800640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f092819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0926800640 1 -- 172.21.15.44:0/1278569872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092819fc80 con 0x7f0928108530 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0926800640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0928108530 0x7f092819f550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0920009fd0 tx=0x7f092000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092000ce20 con 0x7f0928108530 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0920005020 con 0x7f0928108530 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.467+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092819ff70 con 0x7f0928108530 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.468+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920010690 con 0x7f0928108530 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.468+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09281a66d0 con 0x7f0928108530 2024-07-26T08:35:13.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.469+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09281026b0 con 0x7f0928108530 2024-07-26T08:35:13.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.471+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f09200054d0 con 0x7f0928108530 2024-07-26T08:35:13.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.473+0000 7f091ea00640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f090c0b3cb0 0x7f090c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:13.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.473+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f09200e4970 con 0x7f0928108530 2024-07-26T08:35:13.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.473+0000 7f0925e00640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f090c0b3cb0 0x7f090c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:13.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.474+0000 7f0925e00640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f090c0b3cb0 0x7f090c0b61c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0918009c90 tx=0x7f0918005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:13.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.474+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f09200ab950 con 0x7f0928108530 2024-07-26T08:35:13.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.676+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0928105a90 con 0x7f090c0b3cb0 2024-07-26T08:35:13.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.693+0000 7f091ea00640 1 -- 172.21.15.44:0/1278569872 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340684 (secure 0 0 0) 0x7f0928105a90 con 0x7f090c0b3cb0 2024-07-26T08:35:13.696 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:35:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.701+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f090c0b3cb0 msgr2=0x7f090c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.701+0000 7f0927200640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f090c0b3cb0 0x7f090c0b61c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0918009c90 tx=0x7f0918005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.701+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0928108530 msgr2=0x7f092819f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:13.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.701+0000 7f0927200640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0928108530 0x7f092819f550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0920009fd0 tx=0x7f092000ef10 comp rx=0 tx=0).stop 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 shutdown_connections 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f090c0b3cb0 0x7f090c0b61c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0928108530 0x7f092819f550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0928107be0 0x7f092819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 --2- 172.21.15.44:0/1278569872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09281022a0 0x7f0928199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 >> 172.21.15.44:0/1278569872 conn(0x7f09280fdcf0 msgr2=0x7f0928100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 shutdown_connections 2024-07-26T08:35:13.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:13.702+0000 7f0927200640 1 -- 172.21.15.44:0/1278569872 wait complete. 2024-07-26T08:35:14.510 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-07-26T08:35:12.802630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663188,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":501,"num_read_kb":357,"num_write":376,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365716,"kb_used_data":12404,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576108,"statfs":{"total":1151940427776,"available":1151565934592,"internally_reserved":0,"allocated":12701696,"data_stored":8434214,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17593,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":78,"num_write":12,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006232"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:10.201488+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:35:10.201488+0000","last_peered":"2024-07-26T08:35:10.201488+0000","last_clean":"2024-07-26T08:35:10.201488+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:35:10.201488+0000","last_undegraded":"2024-07-26T08:35:10.201488+0000","last_fullsized":"2024-07-26T08:35:10.201488+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591297+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:34:14.591297+0000","last_peered":"2024-07-26T08:34:14.591297+0000","last_clean":"2024-07-26T08:34:14.591297+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:34:14.591297+0000","last_undegraded":"2024-07-26T08:34:14.591297+0000","last_fullsized":"2024-07-26T08:34:14.591297+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965646+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:34:13.965646+0000","last_peered":"2024-07-26T08:34:13.965646+0000","last_clean":"2024-07-26T08:34:13.965646+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:34:13.965646+0000","last_undegraded":"2024-07-26T08:34:13.965646+0000","last_fullsized":"2024-07-26T08:34:13.965646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:29.071793+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:29.071793+0000","last_peered":"2024-07-26T08:34:29.071793+0000","last_clean":"2024-07-26T08:34:29.071793+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:29.071793+0000","last_undegraded":"2024-07-26T08:34:29.071793+0000","last_fullsized":"2024-07-26T08:34:29.071793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.968858+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:34:13.968858+0000","last_peered":"2024-07-26T08:34:13.968858+0000","last_clean":"2024-07-26T08:34:13.968858+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:34:13.968858+0000","last_undegraded":"2024-07-26T08:34:13.968858+0000","last_fullsized":"2024-07-26T08:34:13.968858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.593163+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:34:14.593163+0000","last_peered":"2024-07-26T08:34:14.593163+0000","last_clean":"2024-07-26T08:34:14.593163+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:34:14.593163+0000","last_undegraded":"2024-07-26T08:34:14.593163+0000","last_fullsized":"2024-07-26T08:34:14.593163+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108513+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:34:32.108513+0000","last_peered":"2024-07-26T08:34:32.108513+0000","last_clean":"2024-07-26T08:34:32.108513+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:34:32.108513+0000","last_undegraded":"2024-07-26T08:34:32.108513+0000","last_fullsized":"2024-07-26T08:34:32.108513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952853+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:34:13.952853+0000","last_peered":"2024-07-26T08:34:13.952853+0000","last_clean":"2024-07-26T08:34:13.952853+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:34:13.952853+0000","last_undegraded":"2024-07-26T08:34:13.952853+0000","last_fullsized":"2024-07-26T08:34:13.952853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955297+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:34:13.955297+0000","last_peered":"2024-07-26T08:34:13.955297+0000","last_clean":"2024-07-26T08:34:13.955297+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:34:13.955297+0000","last_undegraded":"2024-07-26T08:34:13.955297+0000","last_fullsized":"2024-07-26T08:34:13.955297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096019+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:34:31.096019+0000","last_peered":"2024-07-26T08:34:31.096019+0000","last_clean":"2024-07-26T08:34:31.096019+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:34:31.096019+0000","last_undegraded":"2024-07-26T08:34:31.096019+0000","last_fullsized":"2024-07-26T08:34:31.096019+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953629+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:34:13.953629+0000","last_peered":"2024-07-26T08:34:13.953629+0000","last_clean":"2024-07-26T08:34:13.953629+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:34:13.953629+0000","last_undegraded":"2024-07-26T08:34:13.953629+0000","last_fullsized":"2024-07-26T08:34:13.953629+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954807+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:34:13.954807+0000","last_peered":"2024-07-26T08:34:13.954807+0000","last_clean":"2024-07-26T08:34:13.954807+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:34:13.954807+0000","last_undegraded":"2024-07-26T08:34:13.954807+0000","last_fullsized":"2024-07-26T08:34:13.954807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253469+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:34:14.253469+0000","last_peered":"2024-07-26T08:34:14.253469+0000","last_clean":"2024-07-26T08:34:14.253469+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:34:14.253469+0000","last_undegraded":"2024-07-26T08:34:14.253469+0000","last_fullsized":"2024-07-26T08:34:14.253469+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:28.059522+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:34:28.059522+0000","last_peered":"2024-07-26T08:34:28.059522+0000","last_clean":"2024-07-26T08:34:28.059522+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:34:28.059522+0000","last_undegraded":"2024-07-26T08:34:28.059522+0000","last_fullsized":"2024-07-26T08:34:28.059522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954915+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.954915+0000","last_peered":"2024-07-26T08:34:13.954915+0000","last_clean":"2024-07-26T08:34:13.954915+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:34:13.954915+0000","last_undegraded":"2024-07-26T08:34:13.954915+0000","last_fullsized":"2024-07-26T08:34:13.954915+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.589885+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:34:14.589885+0000","last_peered":"2024-07-26T08:34:14.589885+0000","last_clean":"2024-07-26T08:34:14.589885+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:34:14.589885+0000","last_undegraded":"2024-07-26T08:34:14.589885+0000","last_fullsized":"2024-07-26T08:34:14.589885+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.092098+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:34:30.092098+0000","last_peered":"2024-07-26T08:34:30.092098+0000","last_clean":"2024-07-26T08:34:30.092098+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:34:30.092098+0000","last_undegraded":"2024-07-26T08:34:30.092098+0000","last_fullsized":"2024-07-26T08:34:30.092098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.417763+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:34:14.417763+0000","last_peered":"2024-07-26T08:34:14.417763+0000","last_clean":"2024-07-26T08:34:14.417763+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:34:14.417763+0000","last_undegraded":"2024-07-26T08:34:14.417763+0000","last_fullsized":"2024-07-26T08:34:14.417763+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962008+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:34:13.962008+0000","last_peered":"2024-07-26T08:34:13.962008+0000","last_clean":"2024-07-26T08:34:13.962008+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:34:13.962008+0000","last_undegraded":"2024-07-26T08:34:13.962008+0000","last_fullsized":"2024-07-26T08:34:13.962008+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097156+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:34:31.097156+0000","last_peered":"2024-07-26T08:34:31.097156+0000","last_clean":"2024-07-26T08:34:31.097156+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:34:31.097156+0000","last_undegraded":"2024-07-26T08:34:31.097156+0000","last_fullsized":"2024-07-26T08:34:31.097156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.594897+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:34:14.594897+0000","last_peered":"2024-07-26T08:34:14.594897+0000","last_clean":"2024-07-26T08:34:14.594897+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:34:14.594897+0000","last_undegraded":"2024-07-26T08:34:14.594897+0000","last_fullsized":"2024-07-26T08:34:14.594897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965086+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:34:13.965086+0000","last_peered":"2024-07-26T08:34:13.965086+0000","last_clean":"2024-07-26T08:34:13.965086+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:34:13.965086+0000","last_undegraded":"2024-07-26T08:34:13.965086+0000","last_fullsized":"2024-07-26T08:34:13.965086+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:28.060952+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:34:28.060952+0000","last_peered":"2024-07-26T08:34:28.060952+0000","last_clean":"2024-07-26T08:34:28.060952+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:34:28.060952+0000","last_undegraded":"2024-07-26T08:34:28.060952+0000","last_fullsized":"2024-07-26T08:34:28.060952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955179+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:34:13.955179+0000","last_peered":"2024-07-26T08:34:13.955179+0000","last_clean":"2024-07-26T08:34:13.955179+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:34:13.955179+0000","last_undegraded":"2024-07-26T08:34:13.955179+0000","last_fullsized":"2024-07-26T08:34:13.955179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967891+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:34:13.967891+0000","last_peered":"2024-07-26T08:34:13.967891+0000","last_clean":"2024-07-26T08:34:13.967891+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:34:13.967891+0000","last_undegraded":"2024-07-26T08:34:13.967891+0000","last_fullsized":"2024-07-26T08:34:13.967891+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108189+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:34:32.108189+0000","last_peered":"2024-07-26T08:34:32.108189+0000","last_clean":"2024-07-26T08:34:32.108189+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:34:32.108189+0000","last_undegraded":"2024-07-26T08:34:32.108189+0000","last_fullsized":"2024-07-26T08:34:32.108189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965977+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:34:13.965977+0000","last_peered":"2024-07-26T08:34:13.965977+0000","last_clean":"2024-07-26T08:34:13.965977+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:34:13.965977+0000","last_undegraded":"2024-07-26T08:34:13.965977+0000","last_fullsized":"2024-07-26T08:34:13.965977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.956980+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:34:13.956980+0000","last_peered":"2024-07-26T08:34:13.956980+0000","last_clean":"2024-07-26T08:34:13.956980+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:34:13.956980+0000","last_undegraded":"2024-07-26T08:34:13.956980+0000","last_fullsized":"2024-07-26T08:34:13.956980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'16","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:03.189366+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:35:03.189366+0000","last_peered":"2024-07-26T08:35:03.189366+0000","last_clean":"2024-07-26T08:35:03.189366+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:35:03.189366+0000","last_undegraded":"2024-07-26T08:35:03.189366+0000","last_fullsized":"2024-07-26T08:35:03.189366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590502+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:34:14.590502+0000","last_peered":"2024-07-26T08:34:14.590502+0000","last_clean":"2024-07-26T08:34:14.590502+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:34:14.590502+0000","last_undegraded":"2024-07-26T08:34:14.590502+0000","last_fullsized":"2024-07-26T08:34:14.590502+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329461+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:34:14.329461+0000","last_peered":"2024-07-26T08:34:14.329461+0000","last_clean":"2024-07-26T08:34:14.329461+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:34:14.329461+0000","last_undegraded":"2024-07-26T08:34:14.329461+0000","last_fullsized":"2024-07-26T08:34:14.329461+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108436+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:34:32.108436+0000","last_peered":"2024-07-26T08:34:32.108436+0000","last_clean":"2024-07-26T08:34:32.108436+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:34:32.108436+0000","last_undegraded":"2024-07-26T08:34:32.108436+0000","last_fullsized":"2024-07-26T08:34:32.108436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961897+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:34:13.961897+0000","last_peered":"2024-07-26T08:34:13.961897+0000","last_clean":"2024-07-26T08:34:13.961897+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:34:13.961897+0000","last_undegraded":"2024-07-26T08:34:13.961897+0000","last_fullsized":"2024-07-26T08:34:13.961897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415652+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:34:14.415652+0000","last_peered":"2024-07-26T08:34:14.415652+0000","last_clean":"2024-07-26T08:34:14.415652+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:34:14.415652+0000","last_undegraded":"2024-07-26T08:34:14.415652+0000","last_fullsized":"2024-07-26T08:34:14.415652+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.095298+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:34:30.095298+0000","last_peered":"2024-07-26T08:34:30.095298+0000","last_clean":"2024-07-26T08:34:30.095298+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:34:30.095298+0000","last_undegraded":"2024-07-26T08:34:30.095298+0000","last_fullsized":"2024-07-26T08:34:30.095298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965873+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:34:13.965873+0000","last_peered":"2024-07-26T08:34:13.965873+0000","last_clean":"2024-07-26T08:34:13.965873+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:34:13.965873+0000","last_undegraded":"2024-07-26T08:34:13.965873+0000","last_fullsized":"2024-07-26T08:34:13.965873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415523+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:34:14.415523+0000","last_peered":"2024-07-26T08:34:14.415523+0000","last_clean":"2024-07-26T08:34:14.415523+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:34:14.415523+0000","last_undegraded":"2024-07-26T08:34:14.415523+0000","last_fullsized":"2024-07-26T08:34:14.415523+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:11.204977+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:35:11.204977+0000","last_peered":"2024-07-26T08:35:11.204977+0000","last_clean":"2024-07-26T08:35:11.204977+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:35:11.204977+0000","last_undegraded":"2024-07-26T08:35:11.204977+0000","last_fullsized":"2024-07-26T08:35:11.204977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328787+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:34:14.328787+0000","last_peered":"2024-07-26T08:34:14.328787+0000","last_clean":"2024-07-26T08:34:14.328787+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:34:14.328787+0000","last_undegraded":"2024-07-26T08:34:14.328787+0000","last_fullsized":"2024-07-26T08:34:14.328787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954090+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:34:13.954090+0000","last_peered":"2024-07-26T08:34:13.954090+0000","last_clean":"2024-07-26T08:34:13.954090+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:34:13.954090+0000","last_undegraded":"2024-07-26T08:34:13.954090+0000","last_fullsized":"2024-07-26T08:34:13.954090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:10.215731+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:35:10.215731+0000","last_peered":"2024-07-26T08:35:10.215731+0000","last_clean":"2024-07-26T08:35:10.215731+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:35:10.215731+0000","last_undegraded":"2024-07-26T08:35:10.215731+0000","last_fullsized":"2024-07-26T08:35:10.215731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328920+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:34:14.328920+0000","last_peered":"2024-07-26T08:34:14.328920+0000","last_clean":"2024-07-26T08:34:14.328920+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:34:14.328920+0000","last_undegraded":"2024-07-26T08:34:14.328920+0000","last_fullsized":"2024-07-26T08:34:14.328920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328166+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:34:14.328166+0000","last_peered":"2024-07-26T08:34:14.328166+0000","last_clean":"2024-07-26T08:34:14.328166+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:34:14.328166+0000","last_undegraded":"2024-07-26T08:34:14.328166+0000","last_fullsized":"2024-07-26T08:34:14.328166+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096346+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:31.096346+0000","last_peered":"2024-07-26T08:34:31.096346+0000","last_clean":"2024-07-26T08:34:31.096346+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:31.096346+0000","last_undegraded":"2024-07-26T08:34:31.096346+0000","last_fullsized":"2024-07-26T08:34:31.096346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953664+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.953664+0000","last_peered":"2024-07-26T08:34:13.953664+0000","last_clean":"2024-07-26T08:34:13.953664+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:34:13.953664+0000","last_undegraded":"2024-07-26T08:34:13.953664+0000","last_fullsized":"2024-07-26T08:34:13.953664+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590008+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:34:14.590008+0000","last_peered":"2024-07-26T08:34:14.590008+0000","last_clean":"2024-07-26T08:34:14.590008+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:34:14.590008+0000","last_undegraded":"2024-07-26T08:34:14.590008+0000","last_fullsized":"2024-07-26T08:34:14.590008+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.109130+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:34:32.109130+0000","last_peered":"2024-07-26T08:34:32.109130+0000","last_clean":"2024-07-26T08:34:32.109130+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:34:32.109130+0000","last_undegraded":"2024-07-26T08:34:32.109130+0000","last_fullsized":"2024-07-26T08:34:32.109130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":122,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252720+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:34:14.252720+0000","last_peered":"2024-07-26T08:34:14.252720+0000","last_clean":"2024-07-26T08:34:14.252720+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:34:14.252720+0000","last_undegraded":"2024-07-26T08:34:14.252720+0000","last_fullsized":"2024-07-26T08:34:14.252720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253123+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:34:14.253123+0000","last_peered":"2024-07-26T08:34:14.253123+0000","last_clean":"2024-07-26T08:34:14.253123+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:34:14.253123+0000","last_undegraded":"2024-07-26T08:34:14.253123+0000","last_fullsized":"2024-07-26T08:34:14.253123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253314+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:34:14.253314+0000","last_peered":"2024-07-26T08:34:14.253314+0000","last_clean":"2024-07-26T08:34:14.253314+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:34:14.253314+0000","last_undegraded":"2024-07-26T08:34:14.253314+0000","last_fullsized":"2024-07-26T08:34:14.253314+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:08.169032+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:35:08.169032+0000","last_peered":"2024-07-26T08:35:08.169032+0000","last_clean":"2024-07-26T08:35:08.169032+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:35:08.169032+0000","last_undegraded":"2024-07-26T08:35:08.169032+0000","last_fullsized":"2024-07-26T08:35:08.169032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952616+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:34:13.952616+0000","last_peered":"2024-07-26T08:34:13.952616+0000","last_clean":"2024-07-26T08:34:13.952616+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:34:13.952616+0000","last_undegraded":"2024-07-26T08:34:13.952616+0000","last_fullsized":"2024-07-26T08:34:13.952616+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253091+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:34:14.253091+0000","last_peered":"2024-07-26T08:34:14.253091+0000","last_clean":"2024-07-26T08:34:14.253091+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:34:14.253091+0000","last_undegraded":"2024-07-26T08:34:14.253091+0000","last_fullsized":"2024-07-26T08:34:14.253091+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097156+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:34:31.097156+0000","last_peered":"2024-07-26T08:34:31.097156+0000","last_clean":"2024-07-26T08:34:31.097156+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:34:31.097156+0000","last_undegraded":"2024-07-26T08:34:31.097156+0000","last_fullsized":"2024-07-26T08:34:31.097156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966970+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:34:13.966970+0000","last_peered":"2024-07-26T08:34:13.966970+0000","last_clean":"2024-07-26T08:34:13.966970+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:34:13.966970+0000","last_undegraded":"2024-07-26T08:34:13.966970+0000","last_fullsized":"2024-07-26T08:34:13.966970+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253049+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:34:14.253049+0000","last_peered":"2024-07-26T08:34:14.253049+0000","last_clean":"2024-07-26T08:34:14.253049+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:34:14.253049+0000","last_undegraded":"2024-07-26T08:34:14.253049+0000","last_fullsized":"2024-07-26T08:34:14.253049+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.095902+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:34:31.095902+0000","last_peered":"2024-07-26T08:34:31.095902+0000","last_clean":"2024-07-26T08:34:31.095902+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:34:31.095902+0000","last_undegraded":"2024-07-26T08:34:31.095902+0000","last_fullsized":"2024-07-26T08:34:31.095902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953086+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:34:13.953086+0000","last_peered":"2024-07-26T08:34:13.953086+0000","last_clean":"2024-07-26T08:34:13.953086+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:34:13.953086+0000","last_undegraded":"2024-07-26T08:34:13.953086+0000","last_fullsized":"2024-07-26T08:34:13.953086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962246+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:34:13.962246+0000","last_peered":"2024-07-26T08:34:13.962246+0000","last_clean":"2024-07-26T08:34:13.962246+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:34:13.962246+0000","last_undegraded":"2024-07-26T08:34:13.962246+0000","last_fullsized":"2024-07-26T08:34:13.962246+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.061878+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:34:31.061878+0000","last_peered":"2024-07-26T08:34:31.061878+0000","last_clean":"2024-07-26T08:34:31.061878+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:34:31.061878+0000","last_undegraded":"2024-07-26T08:34:31.061878+0000","last_fullsized":"2024-07-26T08:34:31.061878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953623+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:34:13.953623+0000","last_peered":"2024-07-26T08:34:13.953623+0000","last_clean":"2024-07-26T08:34:13.953623+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:34:13.953623+0000","last_undegraded":"2024-07-26T08:34:13.953623+0000","last_fullsized":"2024-07-26T08:34:13.953623+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953319+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:34:13.953319+0000","last_peered":"2024-07-26T08:34:13.953319+0000","last_clean":"2024-07-26T08:34:13.953319+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:34:13.953319+0000","last_undegraded":"2024-07-26T08:34:13.953319+0000","last_fullsized":"2024-07-26T08:34:13.953319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096284+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:31.096284+0000","last_peered":"2024-07-26T08:34:31.096284+0000","last_clean":"2024-07-26T08:34:31.096284+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:31.096284+0000","last_undegraded":"2024-07-26T08:34:31.096284+0000","last_fullsized":"2024-07-26T08:34:31.096284+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328983+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:34:14.328983+0000","last_peered":"2024-07-26T08:34:14.328983+0000","last_clean":"2024-07-26T08:34:14.328983+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:34:14.328983+0000","last_undegraded":"2024-07-26T08:34:14.328983+0000","last_fullsized":"2024-07-26T08:34:14.328983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252467+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:34:14.252467+0000","last_peered":"2024-07-26T08:34:14.252467+0000","last_clean":"2024-07-26T08:34:14.252467+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:34:14.252467+0000","last_undegraded":"2024-07-26T08:34:14.252467+0000","last_fullsized":"2024-07-26T08:34:14.252467+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'6","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.208823+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:35:07.208823+0000","last_peered":"2024-07-26T08:35:07.208823+0000","last_clean":"2024-07-26T08:35:07.208823+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:35:07.208823+0000","last_undegraded":"2024-07-26T08:35:07.208823+0000","last_fullsized":"2024-07-26T08:35:07.208823+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.957292+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:34:13.957292+0000","last_peered":"2024-07-26T08:34:13.957292+0000","last_clean":"2024-07-26T08:34:13.957292+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:34:13.957292+0000","last_undegraded":"2024-07-26T08:34:13.957292+0000","last_fullsized":"2024-07-26T08:34:13.957292+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253417+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:34:14.253417+0000","last_peered":"2024-07-26T08:34:14.253417+0000","last_clean":"2024-07-26T08:34:14.253417+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:34:14.253417+0000","last_undegraded":"2024-07-26T08:34:14.253417+0000","last_fullsized":"2024-07-26T08:34:14.253417+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.094654+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:34:31.094654+0000","last_peered":"2024-07-26T08:34:31.094654+0000","last_clean":"2024-07-26T08:34:31.094654+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:34:31.094654+0000","last_undegraded":"2024-07-26T08:34:31.094654+0000","last_fullsized":"2024-07-26T08:34:31.094654+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961912+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:34:13.961912+0000","last_peered":"2024-07-26T08:34:13.961912+0000","last_clean":"2024-07-26T08:34:13.961912+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:34:13.961912+0000","last_undegraded":"2024-07-26T08:34:13.961912+0000","last_fullsized":"2024-07-26T08:34:13.961912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328698+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:34:14.328698+0000","last_peered":"2024-07-26T08:34:14.328698+0000","last_clean":"2024-07-26T08:34:14.328698+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:34:14.328698+0000","last_undegraded":"2024-07-26T08:34:14.328698+0000","last_fullsized":"2024-07-26T08:34:14.328698+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106688+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:34:32.106688+0000","last_peered":"2024-07-26T08:34:32.106688+0000","last_clean":"2024-07-26T08:34:32.106688+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:34:32.106688+0000","last_undegraded":"2024-07-26T08:34:32.106688+0000","last_fullsized":"2024-07-26T08:34:32.106688+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953494+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:34:13.953494+0000","last_peered":"2024-07-26T08:34:13.953494+0000","last_clean":"2024-07-26T08:34:13.953494+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:34:13.953494+0000","last_undegraded":"2024-07-26T08:34:13.953494+0000","last_fullsized":"2024-07-26T08:34:13.953494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965416+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:34:13.965416+0000","last_peered":"2024-07-26T08:34:13.965416+0000","last_clean":"2024-07-26T08:34:13.965416+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:34:13.965416+0000","last_undegraded":"2024-07-26T08:34:13.965416+0000","last_fullsized":"2024-07-26T08:34:13.965416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":161,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.167037+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:35:07.167037+0000","last_peered":"2024-07-26T08:35:07.167037+0000","last_clean":"2024-07-26T08:35:07.167037+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:35:07.167037+0000","last_undegraded":"2024-07-26T08:35:07.167037+0000","last_fullsized":"2024-07-26T08:35:07.167037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65382,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":80,"num_read_kb":104,"num_write":38,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955249+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:34:13.955249+0000","last_peered":"2024-07-26T08:34:13.955249+0000","last_clean":"2024-07-26T08:34:13.955249+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:34:13.955249+0000","last_undegraded":"2024-07-26T08:34:13.955249+0000","last_fullsized":"2024-07-26T08:34:13.955249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952954+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:34:13.952954+0000","last_peered":"2024-07-26T08:34:13.952954+0000","last_clean":"2024-07-26T08:34:13.952954+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:34:13.952954+0000","last_undegraded":"2024-07-26T08:34:13.952954+0000","last_fullsized":"2024-07-26T08:34:13.952954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.090949+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:34:30.090949+0000","last_peered":"2024-07-26T08:34:30.090949+0000","last_clean":"2024-07-26T08:34:30.090949+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:34:30.090949+0000","last_undegraded":"2024-07-26T08:34:30.090949+0000","last_fullsized":"2024-07-26T08:34:30.090949+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.969464+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:34:13.969464+0000","last_peered":"2024-07-26T08:34:13.969464+0000","last_clean":"2024-07-26T08:34:13.969464+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:34:13.969464+0000","last_undegraded":"2024-07-26T08:34:13.969464+0000","last_fullsized":"2024-07-26T08:34:13.969464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.968720+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:34:13.968720+0000","last_peered":"2024-07-26T08:34:13.968720+0000","last_clean":"2024-07-26T08:34:13.968720+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:34:13.968720+0000","last_undegraded":"2024-07-26T08:34:13.968720+0000","last_fullsized":"2024-07-26T08:34:13.968720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096430+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:34:31.096430+0000","last_peered":"2024-07-26T08:34:31.096430+0000","last_clean":"2024-07-26T08:34:31.096430+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:34:31.096430+0000","last_undegraded":"2024-07-26T08:34:31.096430+0000","last_fullsized":"2024-07-26T08:34:31.096430+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954079+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:34:13.954079+0000","last_peered":"2024-07-26T08:34:13.954079+0000","last_clean":"2024-07-26T08:34:13.954079+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:34:13.954079+0000","last_undegraded":"2024-07-26T08:34:13.954079+0000","last_fullsized":"2024-07-26T08:34:13.954079+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953258+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:34:13.953258+0000","last_peered":"2024-07-26T08:34:13.953258+0000","last_clean":"2024-07-26T08:34:13.953258+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:34:13.953258+0000","last_undegraded":"2024-07-26T08:34:13.953258+0000","last_fullsized":"2024-07-26T08:34:13.953258+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.166560+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:35:07.166560+0000","last_peered":"2024-07-26T08:35:07.166560+0000","last_clean":"2024-07-26T08:35:07.166560+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:35:07.166560+0000","last_undegraded":"2024-07-26T08:35:07.166560+0000","last_fullsized":"2024-07-26T08:35:07.166560+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.589873+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:34:14.589873+0000","last_peered":"2024-07-26T08:34:14.589873+0000","last_clean":"2024-07-26T08:34:14.589873+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:34:14.589873+0000","last_undegraded":"2024-07-26T08:34:14.589873+0000","last_fullsized":"2024-07-26T08:34:14.589873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967456+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:34:13.967456+0000","last_peered":"2024-07-26T08:34:13.967456+0000","last_clean":"2024-07-26T08:34:13.967456+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:34:13.967456+0000","last_undegraded":"2024-07-26T08:34:13.967456+0000","last_fullsized":"2024-07-26T08:34:13.967456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954424+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:34:13.954424+0000","last_peered":"2024-07-26T08:34:13.954424+0000","last_clean":"2024-07-26T08:34:13.954424+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:34:13.954424+0000","last_undegraded":"2024-07-26T08:34:13.954424+0000","last_fullsized":"2024-07-26T08:34:13.954424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096426+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:34:31.096426+0000","last_peered":"2024-07-26T08:34:31.096426+0000","last_clean":"2024-07-26T08:34:31.096426+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:34:31.096426+0000","last_undegraded":"2024-07-26T08:34:31.096426+0000","last_fullsized":"2024-07-26T08:34:31.096426+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962146+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:34:13.962146+0000","last_peered":"2024-07-26T08:34:13.962146+0000","last_clean":"2024-07-26T08:34:13.962146+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:34:13.962146+0000","last_undegraded":"2024-07-26T08:34:13.962146+0000","last_fullsized":"2024-07-26T08:34:13.962146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591240+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:34:14.591240+0000","last_peered":"2024-07-26T08:34:14.591240+0000","last_clean":"2024-07-26T08:34:14.591240+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:34:14.591240+0000","last_undegraded":"2024-07-26T08:34:14.591240+0000","last_fullsized":"2024-07-26T08:34:14.591240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:29.072713+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:34:29.072713+0000","last_peered":"2024-07-26T08:34:29.072713+0000","last_clean":"2024-07-26T08:34:29.072713+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:34:29.072713+0000","last_undegraded":"2024-07-26T08:34:29.072713+0000","last_fullsized":"2024-07-26T08:34:29.072713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.090404+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:34:30.090404+0000","last_peered":"2024-07-26T08:34:30.090404+0000","last_clean":"2024-07-26T08:34:30.090404+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:34:30.090404+0000","last_undegraded":"2024-07-26T08:34:30.090404+0000","last_fullsized":"2024-07-26T08:34:30.090404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955741+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:34:13.955741+0000","last_peered":"2024-07-26T08:34:13.955741+0000","last_clean":"2024-07-26T08:34:13.955741+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:34:13.955741+0000","last_undegraded":"2024-07-26T08:34:13.955741+0000","last_fullsized":"2024-07-26T08:34:13.955741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.964550+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:34:13.964550+0000","last_peered":"2024-07-26T08:34:13.964550+0000","last_clean":"2024-07-26T08:34:13.964550+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:34:13.964550+0000","last_undegraded":"2024-07-26T08:34:13.964550+0000","last_fullsized":"2024-07-26T08:34:13.964550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:10.172225+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:35:10.172225+0000","last_peered":"2024-07-26T08:35:10.172225+0000","last_clean":"2024-07-26T08:35:10.172225+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:35:10.172225+0000","last_undegraded":"2024-07-26T08:35:10.172225+0000","last_fullsized":"2024-07-26T08:35:10.172225+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965968+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:34:13.965968+0000","last_peered":"2024-07-26T08:34:13.965968+0000","last_clean":"2024-07-26T08:34:13.965968+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:34:13.965968+0000","last_undegraded":"2024-07-26T08:34:13.965968+0000","last_fullsized":"2024-07-26T08:34:13.965968+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965484+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:34:13.965484+0000","last_peered":"2024-07-26T08:34:13.965484+0000","last_clean":"2024-07-26T08:34:13.965484+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:34:13.965484+0000","last_undegraded":"2024-07-26T08:34:13.965484+0000","last_fullsized":"2024-07-26T08:34:13.965484+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961913+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:34:13.961913+0000","last_peered":"2024-07-26T08:34:13.961913+0000","last_clean":"2024-07-26T08:34:13.961913+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:34:13.961913+0000","last_undegraded":"2024-07-26T08:34:13.961913+0000","last_fullsized":"2024-07-26T08:34:13.961913+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966316+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:34:13.966316+0000","last_peered":"2024-07-26T08:34:13.966316+0000","last_clean":"2024-07-26T08:34:13.966316+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:34:13.966316+0000","last_undegraded":"2024-07-26T08:34:13.966316+0000","last_fullsized":"2024-07-26T08:34:13.966316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.964448+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:34:13.964448+0000","last_peered":"2024-07-26T08:34:13.964448+0000","last_clean":"2024-07-26T08:34:13.964448+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:34:13.964448+0000","last_undegraded":"2024-07-26T08:34:13.964448+0000","last_fullsized":"2024-07-26T08:34:13.964448+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.969270+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:34:13.969270+0000","last_peered":"2024-07-26T08:34:13.969270+0000","last_clean":"2024-07-26T08:34:13.969270+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:34:13.969270+0000","last_undegraded":"2024-07-26T08:34:13.969270+0000","last_fullsized":"2024-07-26T08:34:13.969270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329138+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:34:14.329138+0000","last_peered":"2024-07-26T08:34:14.329138+0000","last_clean":"2024-07-26T08:34:14.329138+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:34:14.329138+0000","last_undegraded":"2024-07-26T08:34:14.329138+0000","last_fullsized":"2024-07-26T08:34:14.329138+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962264+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:34:13.962264+0000","last_peered":"2024-07-26T08:34:13.962264+0000","last_clean":"2024-07-26T08:34:13.962264+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:34:13.962264+0000","last_undegraded":"2024-07-26T08:34:13.962264+0000","last_fullsized":"2024-07-26T08:34:13.962264+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966257+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:34:13.966257+0000","last_peered":"2024-07-26T08:34:13.966257+0000","last_clean":"2024-07-26T08:34:13.966257+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:34:13.966257+0000","last_undegraded":"2024-07-26T08:34:13.966257+0000","last_fullsized":"2024-07-26T08:34:13.966257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591375+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:34:14.591375+0000","last_peered":"2024-07-26T08:34:14.591375+0000","last_clean":"2024-07-26T08:34:14.591375+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:34:14.591375+0000","last_undegraded":"2024-07-26T08:34:14.591375+0000","last_fullsized":"2024-07-26T08:34:14.591375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953257+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:34:13.953257+0000","last_peered":"2024-07-26T08:34:13.953257+0000","last_clean":"2024-07-26T08:34:13.953257+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:34:13.953257+0000","last_undegraded":"2024-07-26T08:34:13.953257+0000","last_fullsized":"2024-07-26T08:34:13.953257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590501+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:34:14.590501+0000","last_peered":"2024-07-26T08:34:14.590501+0000","last_clean":"2024-07-26T08:34:14.590501+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:34:14.590501+0000","last_undegraded":"2024-07-26T08:34:14.590501+0000","last_fullsized":"2024-07-26T08:34:14.590501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952802+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:34:13.952802+0000","last_peered":"2024-07-26T08:34:13.952802+0000","last_clean":"2024-07-26T08:34:13.952802+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:34:13.952802+0000","last_undegraded":"2024-07-26T08:34:13.952802+0000","last_fullsized":"2024-07-26T08:34:13.952802+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328854+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:34:14.328854+0000","last_peered":"2024-07-26T08:34:14.328854+0000","last_clean":"2024-07-26T08:34:14.328854+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:34:14.328854+0000","last_undegraded":"2024-07-26T08:34:14.328854+0000","last_fullsized":"2024-07-26T08:34:14.328854+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961972+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:34:13.961972+0000","last_peered":"2024-07-26T08:34:13.961972+0000","last_clean":"2024-07-26T08:34:13.961972+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:34:13.961972+0000","last_undegraded":"2024-07-26T08:34:13.961972+0000","last_fullsized":"2024-07-26T08:34:13.961972+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965961+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:34:13.965961+0000","last_peered":"2024-07-26T08:34:13.965961+0000","last_clean":"2024-07-26T08:34:13.965961+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:34:13.965961+0000","last_undegraded":"2024-07-26T08:34:13.965961+0000","last_fullsized":"2024-07-26T08:34:13.965961+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.957489+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:34:13.957489+0000","last_peered":"2024-07-26T08:34:13.957489+0000","last_clean":"2024-07-26T08:34:13.957489+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:34:13.957489+0000","last_undegraded":"2024-07-26T08:34:13.957489+0000","last_fullsized":"2024-07-26T08:34:13.957489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962419+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:34:13.962419+0000","last_peered":"2024-07-26T08:34:13.962419+0000","last_clean":"2024-07-26T08:34:13.962419+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:34:13.962419+0000","last_undegraded":"2024-07-26T08:34:13.962419+0000","last_fullsized":"2024-07-26T08:34:13.962419+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952479+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:34:13.952479+0000","last_peered":"2024-07-26T08:34:13.952479+0000","last_clean":"2024-07-26T08:34:13.952479+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:34:13.952479+0000","last_undegraded":"2024-07-26T08:34:13.952479+0000","last_fullsized":"2024-07-26T08:34:13.952479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966639+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:34:13.966639+0000","last_peered":"2024-07-26T08:34:13.966639+0000","last_clean":"2024-07-26T08:34:13.966639+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:34:13.966639+0000","last_undegraded":"2024-07-26T08:34:13.966639+0000","last_fullsized":"2024-07-26T08:34:13.966639+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966925+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:34:13.966925+0000","last_peered":"2024-07-26T08:34:13.966925+0000","last_clean":"2024-07-26T08:34:13.966925+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:34:13.966925+0000","last_undegraded":"2024-07-26T08:34:13.966925+0000","last_fullsized":"2024-07-26T08:34:13.966925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328677+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:34:14.328677+0000","last_peered":"2024-07-26T08:34:14.328677+0000","last_clean":"2024-07-26T08:34:14.328677+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:34:14.328677+0000","last_undegraded":"2024-07-26T08:34:14.328677+0000","last_fullsized":"2024-07-26T08:34:14.328677+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955562+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:34:13.955562+0000","last_peered":"2024-07-26T08:34:13.955562+0000","last_clean":"2024-07-26T08:34:13.955562+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:34:13.955562+0000","last_undegraded":"2024-07-26T08:34:13.955562+0000","last_fullsized":"2024-07-26T08:34:13.955562+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.253255+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:34:14.253255+0000","last_peered":"2024-07-26T08:34:14.253255+0000","last_clean":"2024-07-26T08:34:14.253255+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:34:14.253255+0000","last_undegraded":"2024-07-26T08:34:14.253255+0000","last_fullsized":"2024-07-26T08:34:14.253255+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415930+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:34:14.415930+0000","last_peered":"2024-07-26T08:34:14.415930+0000","last_clean":"2024-07-26T08:34:14.415930+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:34:14.415930+0000","last_undegraded":"2024-07-26T08:34:14.415930+0000","last_fullsized":"2024-07-26T08:34:14.415930+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953212+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:34:13.953212+0000","last_peered":"2024-07-26T08:34:13.953212+0000","last_clean":"2024-07-26T08:34:13.953212+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:34:13.953212+0000","last_undegraded":"2024-07-26T08:34:13.953212+0000","last_fullsized":"2024-07-26T08:34:13.953212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.590926+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:34:14.590926+0000","last_peered":"2024-07-26T08:34:14.590926+0000","last_clean":"2024-07-26T08:34:14.590926+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:34:14.590926+0000","last_undegraded":"2024-07-26T08:34:14.590926+0000","last_fullsized":"2024-07-26T08:34:14.590926+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252550+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:34:14.252550+0000","last_peered":"2024-07-26T08:34:14.252550+0000","last_clean":"2024-07-26T08:34:14.252550+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:34:14.252550+0000","last_undegraded":"2024-07-26T08:34:14.252550+0000","last_fullsized":"2024-07-26T08:34:14.252550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.415406+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:34:14.415406+0000","last_peered":"2024-07-26T08:34:14.415406+0000","last_clean":"2024-07-26T08:34:14.415406+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:34:14.415406+0000","last_undegraded":"2024-07-26T08:34:14.415406+0000","last_fullsized":"2024-07-26T08:34:14.415406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965397+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:34:13.965397+0000","last_peered":"2024-07-26T08:34:13.965397+0000","last_clean":"2024-07-26T08:34:13.965397+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:34:13.965397+0000","last_undegraded":"2024-07-26T08:34:13.965397+0000","last_fullsized":"2024-07-26T08:34:13.965397+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955486+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:34:13.955486+0000","last_peered":"2024-07-26T08:34:13.955486+0000","last_clean":"2024-07-26T08:34:13.955486+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:34:13.955486+0000","last_undegraded":"2024-07-26T08:34:13.955486+0000","last_fullsized":"2024-07-26T08:34:13.955486+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967332+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:34:13.967332+0000","last_peered":"2024-07-26T08:34:13.967332+0000","last_clean":"2024-07-26T08:34:13.967332+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:34:13.967332+0000","last_undegraded":"2024-07-26T08:34:13.967332+0000","last_fullsized":"2024-07-26T08:34:13.967332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967579+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:34:13.967579+0000","last_peered":"2024-07-26T08:34:13.967579+0000","last_clean":"2024-07-26T08:34:13.967579+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:34:13.967579+0000","last_undegraded":"2024-07-26T08:34:13.967579+0000","last_fullsized":"2024-07-26T08:34:13.967579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962458+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:34:13.962458+0000","last_peered":"2024-07-26T08:34:13.962458+0000","last_clean":"2024-07-26T08:34:13.962458+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:34:13.962458+0000","last_undegraded":"2024-07-26T08:34:13.962458+0000","last_fullsized":"2024-07-26T08:34:13.962458+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72586,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":420,"num_read_kb":288,"num_write":251,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":217758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645355,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540173,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841582,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1196355,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005108,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201339,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397569,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136475,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561092,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539569,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117077,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378717,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":602127,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542241,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":601516,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":536951,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:35:14.516 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T08:35:14.516 INFO:tasks.cephadm:Setting up client nodes... 2024-07-26T08:35:14.517 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-26T08:35:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:14 smithi044 ceph-mon[32404]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-26T08:35:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:14 smithi116 ceph-mon[37097]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-26T08:35:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:14 smithi171 ceph-mon[37789]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-26T08:35:15.062 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:15 smithi044 ceph-mon[32404]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:35:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:15 smithi116 ceph-mon[37097]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:35:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:15 smithi171 ceph-mon[37789]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:35:16.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:16 smithi044 ceph-mon[32404]: pgmap v380: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T08:35:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:16 smithi116 ceph-mon[37097]: pgmap v380: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T08:35:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:16 smithi171 ceph-mon[37789]: pgmap v380: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 -- 172.21.15.44:0/2830163114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca8077370 msgr2=0x7f9ca8077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 --2- 172.21.15.44:0/2830163114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca8077370 0x7f9ca8077780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009860 tx=0x7f9c9802a340 comp rx=0 tx=0).stop 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 -- 172.21.15.44:0/2830163114 shutdown_connections 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 --2- 172.21.15.44:0/2830163114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ca81141b0 0x7f9ca81165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 --2- 172.21.15.44:0/2830163114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077cc0 0x7f9ca8113c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 --2- 172.21.15.44:0/2830163114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca8077370 0x7f9ca8077780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 -- 172.21.15.44:0/2830163114 >> 172.21.15.44:0/2830163114 conn(0x7f9ca8101200 msgr2=0x7f9ca8103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 -- 172.21.15.44:0/2830163114 shutdown_connections 2024-07-26T08:35:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.293+0000 7f9cae200640 1 -- 172.21.15.44:0/2830163114 wait complete. 2024-07-26T08:35:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.295+0000 7f9cae200640 1 Processor -- start 2024-07-26T08:35:17.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.295+0000 7f9cae200640 1 -- start start 2024-07-26T08:35:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9cae200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 0x7f9ca81a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9cae200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ca8077cc0 0x7f9ca81a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9cae200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca81141b0 0x7f9ca81a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9cae200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8119b10 con 0x7f9ca8077370 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9cae200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8119990 con 0x7f9ca81141b0 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9cae200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8119c90 con 0x7f9ca8077cc0 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9ca7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 0x7f9ca81a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9ca7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 0x7f9ca81a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53754/0 (socket says 172.21.15.44:53754) 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9ca7400640 1 -- 172.21.15.44:0/658869491 learned_addr learned my addr 172.21.15.44:0/658869491 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9ca7e00640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca81141b0 0x7f9ca81a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.296+0000 7f9ca6a00640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ca8077cc0 0x7f9ca81a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.297+0000 7f9ca7400640 1 -- 172.21.15.44:0/658869491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ca8077cc0 msgr2=0x7f9ca81a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.297+0000 7f9ca7400640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ca8077cc0 0x7f9ca81a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.297+0000 7f9ca7400640 1 -- 172.21.15.44:0/658869491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca81141b0 msgr2=0x7f9ca81a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.297+0000 7f9ca7400640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca81141b0 0x7f9ca81a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.297+0000 7f9ca7400640 1 -- 172.21.15.44:0/658869491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca81a6430 con 0x7f9ca8077370 2024-07-26T08:35:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.297+0000 7f9ca7400640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 0x7f9ca81a04c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9c980303f0 tx=0x7f9c98032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.298+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98040020 con 0x7f9ca8077370 2024-07-26T08:35:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.298+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca81a66c0 con 0x7f9ca8077370 2024-07-26T08:35:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.298+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c98030450 con 0x7f9ca8077370 2024-07-26T08:35:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.298+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98037040 con 0x7f9ca8077370 2024-07-26T08:35:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.298+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca81ace80 con 0x7f9ca8077370 2024-07-26T08:35:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.299+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca8077780 con 0x7f9ca8077370 2024-07-26T08:35:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.302+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f9c98043020 con 0x7f9ca8077370 2024-07-26T08:35:17.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.303+0000 7f9c9fe00640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c8c0b3d30 0x7f9c8c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:17.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.304+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9c981043e0 con 0x7f9ca8077370 2024-07-26T08:35:17.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.304+0000 7f9ca6a00640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c8c0b3d30 0x7f9c8c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:17.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.305+0000 7f9ca6a00640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c8c0b3d30 0x7f9c8c0b6240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8079ac0 tx=0x7f9c90009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:17.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.306+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9c9804a020 con 0x7f9ca8077370 2024-07-26T08:35:17.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.586+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9ca81a15f0 con 0x7f9ca8077370 2024-07-26T08:35:17.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.591+0000 7f9c9fe00640 1 -- 172.21.15.44:0/658869491 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v33) ==== 170+0+59 (secure 0 0 0) 0x7f9c980cb550 con 0x7f9ca8077370 2024-07-26T08:35:17.593 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-07-26T08:35:17.593 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDFX6Nm0A4PIxAAnqZNvUOgmrAwzz1qG0fS+w== 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.594+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c8c0b3d30 msgr2=0x7f9c8c0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.594+0000 7f9cae200640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c8c0b3d30 0x7f9c8c0b6240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8079ac0 tx=0x7f9c90009260 comp rx=0 tx=0).stop 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.594+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 msgr2=0x7f9ca81a04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.594+0000 7f9cae200640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 0x7f9ca81a04c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9c980303f0 tx=0x7f9c98032eb0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 shutdown_connections 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9c8c0b3d30 0x7f9c8c0b6240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ca8077cc0 0x7f9ca81a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca81141b0 0x7f9ca81a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 --2- 172.21.15.44:0/658869491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ca8077370 0x7f9ca81a04c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:17.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 >> 172.21.15.44:0/658869491 conn(0x7f9ca8101200 msgr2=0x7f9ca8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:17.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 shutdown_connections 2024-07-26T08:35:17.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:17.595+0000 7f9cae200640 1 -- 172.21.15.44:0/658869491 wait complete. 2024-07-26T08:35:18.457 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:35:18.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-26T08:35:18.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T08:35:18.606 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph config log 1 --format=json 2024-07-26T08:35:18.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:18 smithi044 ceph-mon[32404]: pgmap v381: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T08:35:18.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:18 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/658869491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T08:35:18.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:18 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/658869491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T08:35:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:18 smithi116 ceph-mon[37097]: pgmap v381: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T08:35:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:18 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/658869491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T08:35:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:18 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/658869491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T08:35:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:18 smithi171 ceph-mon[37789]: pgmap v381: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T08:35:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:18 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/658869491' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T08:35:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:18 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/658869491' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T08:35:19.161 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:20 smithi044 ceph-mon[32404]: pgmap v382: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T08:35:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:20 smithi116 ceph-mon[37097]: pgmap v382: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T08:35:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:20 smithi171 ceph-mon[37789]: pgmap v382: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T08:35:21.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 -- 172.21.15.44:0/133417603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec10ba70 msgr2=0x7f81ec1122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:21.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 --2- 172.21.15.44:0/133417603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec10ba70 0x7f81ec1122a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009ca0 tx=0x7f81e402a8c0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 -- 172.21.15.44:0/133417603 shutdown_connections 2024-07-26T08:35:21.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 --2- 172.21.15.44:0/133417603 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec10ba70 0x7f81ec1122a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 --2- 172.21.15.44:0/133417603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 0x7f81ec10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 --2- 172.21.15.44:0/133417603 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81ec1057e0 0x7f81ec105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.385+0000 7f81f3600640 1 -- 172.21.15.44:0/133417603 >> 172.21.15.44:0/133417603 conn(0x7f81ec1011f0 msgr2=0x7f81ec103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:21.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.386+0000 7f81f3600640 1 -- 172.21.15.44:0/133417603 shutdown_connections 2024-07-26T08:35:21.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.386+0000 7f81f3600640 1 -- 172.21.15.44:0/133417603 wait complete. 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 Processor -- start 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 -- start start 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec1057e0 0x7f81ec19c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 0x7f81ec19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81ec10ba70 0x7f81ec1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec115790 con 0x7f81ec1057e0 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec115610 con 0x7f81ec10b120 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.387+0000 7f81f3600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec115910 con 0x7f81ec10ba70 2024-07-26T08:35:21.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81f0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec1057e0 0x7f81ec19c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:21.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 0x7f81ec19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:21.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81f0c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec1057e0 0x7f81ec19c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53780/0 (socket says 172.21.15.44:53780) 2024-07-26T08:35:21.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81f0c00640 1 -- 172.21.15.44:0/741420474 learned_addr learned my addr 172.21.15.44:0/741420474 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81f1600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81ec10ba70 0x7f81ec1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81ebe00640 1 -- 172.21.15.44:0/741420474 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81ec10ba70 msgr2=0x7f81ec1a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81ebe00640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81ec10ba70 0x7f81ec1a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81ebe00640 1 -- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec1057e0 msgr2=0x7f81ec19c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81ebe00640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec1057e0 0x7f81ec19c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.388+0000 7f81ebe00640 1 -- 172.21.15.44:0/741420474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec1a1ff0 con 0x7f81ec10b120 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.389+0000 7f81ebe00640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 0x7f81ec19c620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f81ec06c3a0 tx=0x7f81d800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:21.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.389+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d800cc40 con 0x7f81ec10b120 2024-07-26T08:35:21.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.389+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d800cde0 con 0x7f81ec10b120 2024-07-26T08:35:21.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.389+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ec1a22e0 con 0x7f81ec10b120 2024-07-26T08:35:21.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.389+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8018780 con 0x7f81ec10b120 2024-07-26T08:35:21.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.389+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ec1a8a40 con 0x7f81ec10b120 2024-07-26T08:35:21.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.390+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81ec105bf0 con 0x7f81ec10b120 2024-07-26T08:35:21.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.394+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f81d8010a90 con 0x7f81ec10b120 2024-07-26T08:35:21.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.395+0000 7f81e9600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81d40b3cb0 0x7f81d40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:21.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.396+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f81d80e4ac0 con 0x7f81ec10b120 2024-07-26T08:35:21.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.396+0000 7f81f0c00640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81d40b3cb0 0x7f81d40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:21.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.396+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f81d80abaa0 con 0x7f81ec10b120 2024-07-26T08:35:21.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.397+0000 7f81f0c00640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81d40b3cb0 0x7f81d40b61c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009cd0 tx=0x7f81e0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:21.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.602+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f81ec10d2d0 con 0x7f81ec10b120 2024-07-26T08:35:21.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.604+0000 7f81e9600640 1 -- 172.21.15.44:0/741420474 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v54) ==== 86+0+133 (secure 0 0 0) 0x7f81d80ab3a0 con 0x7f81ec10b120 2024-07-26T08:35:21.605 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.606+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81d40b3cb0 msgr2=0x7f81d40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.606+0000 7f81f3600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81d40b3cb0 0x7f81d40b61c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009cd0 tx=0x7f81e0005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.606+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 msgr2=0x7f81ec19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.606+0000 7f81f3600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 0x7f81ec19c620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f81ec06c3a0 tx=0x7f81d800ee50 comp rx=0 tx=0).stop 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 shutdown_connections 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f81d40b3cb0 0x7f81d40b61c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81ec10ba70 0x7f81ec1a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81ec10b120 0x7f81ec19c620 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 --2- 172.21.15.44:0/741420474 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81ec1057e0 0x7f81ec19c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:21.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 >> 172.21.15.44:0/741420474 conn(0x7f81ec1011f0 msgr2=0x7f81ec1085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:21.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 shutdown_connections 2024-07-26T08:35:21.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:21.607+0000 7f81f3600640 1 -- 172.21.15.44:0/741420474 wait complete. 2024-07-26T08:35:22.415 INFO:teuthology.orchestra.run.smithi044.stdout:[{"version":54,"timestamp":"2024-07-26T08:32:43.393885+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-26T08:35:22.416 INFO:tasks.ceph_manager:config epoch is 54 2024-07-26T08:35:22.416 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T08:35:22.416 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-26T08:35:22.416 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph mgr dump --format=json 2024-07-26T08:35:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:22 smithi171 ceph-mon[37789]: pgmap v383: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-26T08:35:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:22 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/741420474' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T08:35:22.902 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:22 smithi044 ceph-mon[32404]: pgmap v383: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-26T08:35:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:22 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/741420474' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T08:35:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:22 smithi116 ceph-mon[37097]: pgmap v383: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-26T08:35:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:22 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/741420474' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T08:35:24.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:24 smithi044 ceph-mon[32404]: pgmap v384: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:24 smithi171 ceph-mon[37789]: pgmap v384: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:24 smithi116 ceph-mon[37097]: pgmap v384: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:25.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 -- 172.21.15.44:0/1257183363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 msgr2=0x7fa83c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:25.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 --2- 172.21.15.44:0/1257183363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c10fea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa834009ca0 tx=0x7fa83402a8c0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 -- 172.21.15.44:0/1257183363 shutdown_connections 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 --2- 172.21.15.44:0/1257183363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c10fea0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 --2- 172.21.15.44:0/1257183363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c108d20 0x7fa83c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 --2- 172.21.15.44:0/1257183363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c1033e0 0x7fa83c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.305+0000 7fa843000640 1 -- 172.21.15.44:0/1257183363 >> 172.21.15.44:0/1257183363 conn(0x7fa83c0fee70 msgr2=0x7fa83c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.306+0000 7fa843000640 1 -- 172.21.15.44:0/1257183363 shutdown_connections 2024-07-26T08:35:25.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.306+0000 7fa843000640 1 -- 172.21.15.44:0/1257183363 wait complete. 2024-07-26T08:35:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.307+0000 7fa843000640 1 Processor -- start 2024-07-26T08:35:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 -- start start 2024-07-26T08:35:25.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c1033e0 0x7fa83c19bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:25.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c108d20 0x7fa83c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:25.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:25.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c113390 con 0x7fa83c108d20 2024-07-26T08:35:25.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c113210 con 0x7fa83c1033e0 2024-07-26T08:35:25.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.308+0000 7fa843000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c113510 con 0x7fa83c109670 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa841200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c108d20 0x7fa83c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa841c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c1033e0 0x7fa83c19bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa842600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa841200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c108d20 0x7fa83c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43136/0 (socket says 172.21.15.44:43136) 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa841c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c1033e0 0x7fa83c19bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:45612/0 (socket says 172.21.15.44:45612) 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa841200640 1 -- 172.21.15.44:0/3176935563 learned_addr learned my addr 172.21.15.44:0/3176935563 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa842600640 1 -- 172.21.15.44:0/3176935563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c1033e0 msgr2=0x7fa83c19bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa842600640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c1033e0 0x7fa83c19bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa842600640 1 -- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c108d20 msgr2=0x7fa83c19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa842600640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c108d20 0x7fa83c19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.309+0000 7fa842600640 1 -- 172.21.15.44:0/3176935563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c1a1e10 con 0x7fa83c109670 2024-07-26T08:35:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.310+0000 7fa842600640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c1a16e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa834005e30 tx=0x7fa8340317c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.310+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834003b70 con 0x7fa83c109670 2024-07-26T08:35:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.310+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83c1a20a0 con 0x7fa83c109670 2024-07-26T08:35:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.310+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa834003d10 con 0x7fa83c109670 2024-07-26T08:35:25.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.310+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83402f970 con 0x7fa83c109670 2024-07-26T08:35:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.310+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83c1a8860 con 0x7fa83c109670 2024-07-26T08:35:25.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.311+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83c1037f0 con 0x7fa83c109670 2024-07-26T08:35:25.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.314+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fa834004b40 con 0x7fa83c109670 2024-07-26T08:35:25.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.316+0000 7fa83a000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8240b3cb0 0x7fa8240b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:25.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.316+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fa834105550 con 0x7fa83c109670 2024-07-26T08:35:25.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.316+0000 7fa841c00640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8240b3cb0 0x7fa8240b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:25.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.317+0000 7fa841c00640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8240b3cb0 0x7fa8240b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa830009bd0 tx=0x7fa830005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:25.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.317+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa8340cc640 con 0x7fa83c109670 2024-07-26T08:35:25.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.572+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa83c1a8a90 con 0x7fa83c109670 2024-07-26T08:35:25.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.577+0000 7fa83a000640 1 -- 172.21.15.44:0/3176935563 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+291036 (secure 0 0 0) 0x7fa8340cbf40 con 0x7fa83c109670 2024-07-26T08:35:25.582 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:25.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8240b3cb0 msgr2=0x7fa8240b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:25.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8240b3cb0 0x7fa8240b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa830009bd0 tx=0x7fa830005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:25.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 msgr2=0x7fa83c1a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:25.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c1a16e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa834005e30 tx=0x7fa8340317c0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 shutdown_connections 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8240b3cb0 0x7fa8240b61c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa83c109670 0x7fa83c1a16e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa83c108d20 0x7fa83c19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 --2- 172.21.15.44:0/3176935563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa83c1033e0 0x7fa83c19bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 >> 172.21.15.44:0/3176935563 conn(0x7fa83c0fee70 msgr2=0x7fa83c1082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 shutdown_connections 2024-07-26T08:35:25.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:25.584+0000 7fa843000640 1 -- 172.21.15.44:0/3176935563 wait complete. 2024-07-26T08:35:26.566 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"flags":0,"active_gid":14211,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":2852968588},{"type":"v1","addr":"172.21.15.44:6801","nonce":2852968588}]},"active_addr":"172.21.15.44:6801/2852968588","active_change":"2024-07-26T08:25:12.576320+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24130,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24140,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.44:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1794351527}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":2191658667}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":4167935679}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":2617314357}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1172005782}]}]} 2024-07-26T08:35:26.573 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-26T08:35:26.573 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-26T08:35:26.573 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:35:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:26 smithi044 ceph-mon[32404]: pgmap v385: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:26 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3176935563' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T08:35:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:26 smithi171 ceph-mon[37789]: pgmap v385: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:26 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3176935563' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T08:35:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:26 smithi116 ceph-mon[37097]: pgmap v385: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:26 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3176935563' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T08:35:27.059 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:28 smithi171 ceph-mon[37789]: pgmap v386: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:28.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:28 smithi044 ceph-mon[32404]: pgmap v386: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:28 smithi116 ceph-mon[37097]: pgmap v386: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T08:35:29.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 -- 172.21.15.44:0/4075934994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 msgr2=0x7ff53c0751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:29.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 --2- 172.21.15.44:0/4075934994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c0751e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff524009900 tx=0x7ff52402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 -- 172.21.15.44:0/4075934994 shutdown_connections 2024-07-26T08:35:29.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 --2- 172.21.15.44:0/4075934994 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff53c113cb0 0x7ff53c116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 --2- 172.21.15.44:0/4075934994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c075800 0x7ff53c075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 --2- 172.21.15.44:0/4075934994 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c0751e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 -- 172.21.15.44:0/4075934994 >> 172.21.15.44:0/4075934994 conn(0x7ff53c0fee70 msgr2=0x7ff53c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:29.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 -- 172.21.15.44:0/4075934994 shutdown_connections 2024-07-26T08:35:29.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.469+0000 7ff541200640 1 -- 172.21.15.44:0/4075934994 wait complete. 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.471+0000 7ff541200640 1 Processor -- start 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.471+0000 7ff541200640 1 -- start start 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.471+0000 7ff541200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff53c075800 0x7ff53c1a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff541200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff541200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c113cb0 0x7ff53c1a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff541200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c1198f0 con 0x7ff53c076db0 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff541200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c119770 con 0x7ff53c113cb0 2024-07-26T08:35:29.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff541200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c119a70 con 0x7ff53c075800 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff53aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff53aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c1a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43140/0 (socket says 172.21.15.44:43140) 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff53aa00640 1 -- 172.21.15.44:0/2931228283 learned_addr learned my addr 172.21.15.44:0/2931228283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff53b400640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff53c075800 0x7ff53c1a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.472+0000 7ff53be00640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c113cb0 0x7ff53c1a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff53aa00640 1 -- 172.21.15.44:0/2931228283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff53c075800 msgr2=0x7ff53c1a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff53aa00640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff53c075800 0x7ff53c1a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff53aa00640 1 -- 172.21.15.44:0/2931228283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c113cb0 msgr2=0x7ff53c1a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:29.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff53aa00640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c113cb0 0x7ff53c1a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff53aa00640 1 -- 172.21.15.44:0/2931228283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c1a6330 con 0x7ff53c076db0 2024-07-26T08:35:29.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff53aa00640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c1a0960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00efd0 tx=0x7ff52c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:29.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52c003b70 con 0x7ff53c076db0 2024-07-26T08:35:29.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff52c003d10 con 0x7ff53c076db0 2024-07-26T08:35:29.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0189d0 con 0x7ff53c076db0 2024-07-26T08:35:29.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.473+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c1a6620 con 0x7ff53c076db0 2024-07-26T08:35:29.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.474+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c1acd80 con 0x7ff53c076db0 2024-07-26T08:35:29.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.475+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c075c10 con 0x7ff53c076db0 2024-07-26T08:35:29.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.477+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7ff52c018b70 con 0x7ff53c076db0 2024-07-26T08:35:29.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.479+0000 7ff533e00640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff5200b3d30 0x7ff5200b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:29.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.479+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff52c0e4b70 con 0x7ff53c076db0 2024-07-26T08:35:29.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.479+0000 7ff53b400640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff5200b3d30 0x7ff5200b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:29.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.480+0000 7ff53b400640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff5200b3d30 0x7ff5200b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff524031440 tx=0x7ff524009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:29.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.481+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff52c0abbe0 con 0x7ff53c076db0 2024-07-26T08:35:29.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.685+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff53c1a14f0 con 0x7ff53c076db0 2024-07-26T08:35:29.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.689+0000 7ff533e00640 1 -- 172.21.15.44:0/2931228283 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7ff52c0ab4e0 con 0x7ff53c076db0 2024-07-26T08:35:29.691 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:29.692 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:35:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff5200b3d30 msgr2=0x7ff5200b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff5200b3d30 0x7ff5200b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff524031440 tx=0x7ff524009560 comp rx=0 tx=0).stop 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 msgr2=0x7ff53c1a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c1a0960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00efd0 tx=0x7ff52c00c6a0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 shutdown_connections 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff5200b3d30 0x7ff5200b6240 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c113cb0 0x7ff53c1a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff53c076db0 0x7ff53c1a0960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 --2- 172.21.15.44:0/2931228283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff53c075800 0x7ff53c1a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:29.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 >> 172.21.15.44:0/2931228283 conn(0x7ff53c0fee70 msgr2=0x7ff53c10fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:29.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.692+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 shutdown_connections 2024-07-26T08:35:29.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:29.693+0000 7ff541200640 1 -- 172.21.15.44:0/2931228283 wait complete. 2024-07-26T08:35:30.677 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-26T08:35:30.677 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:35:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:30 smithi171 ceph-mon[37789]: pgmap v387: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T08:35:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:30 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2931228283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:30 smithi044 ceph-mon[32404]: pgmap v387: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T08:35:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:30 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2931228283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:30 smithi116 ceph-mon[37097]: pgmap v387: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T08:35:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:30 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2931228283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:31.182 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:32 smithi171 ceph-mon[37789]: pgmap v388: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:32.965 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:32 smithi044 ceph-mon[32404]: pgmap v388: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:32 smithi116 ceph-mon[37097]: pgmap v388: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.569+0000 7f0d3b200640 1 -- 172.21.15.44:0/1141946997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 msgr2=0x7f0d341165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 --2- 172.21.15.44:0/1141946997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 0x7f0d341165c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009ca0 tx=0x7f0d2c02a8c0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 -- 172.21.15.44:0/1141946997 shutdown_connections 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 --2- 172.21.15.44:0/1141946997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 0x7f0d34113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 --2- 172.21.15.44:0/1141946997 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 0x7f0d341165c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 --2- 172.21.15.44:0/1141946997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d34077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 -- 172.21.15.44:0/1141946997 >> 172.21.15.44:0/1141946997 conn(0x7f0d341011f0 msgr2=0x7f0d34103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 -- 172.21.15.44:0/1141946997 shutdown_connections 2024-07-26T08:35:33.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.570+0000 7f0d3b200640 1 -- 172.21.15.44:0/1141946997 wait complete. 2024-07-26T08:35:33.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.571+0000 7f0d3b200640 1 Processor -- start 2024-07-26T08:35:33.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.571+0000 7f0d3b200640 1 -- start start 2024-07-26T08:35:33.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d3b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d341a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d3b200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 0x7f0d341a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d3b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 0x7f0d341a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d3b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0d34119b00 con 0x7f0d341141a0 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d3b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d34119980 con 0x7f0d34077370 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d3b200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0d34119c80 con 0x7f0d34077cc0 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d341a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d341a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:38734/0 (socket says 172.21.15.44:38734) 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d33e00640 1 -- 172.21.15.44:0/3908010151 learned_addr learned my addr 172.21.15.44:0/3908010151 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d2b400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 0x7f0d341a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:33.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.572+0000 7f0d39200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 0x7f0d341a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d2b400640 1 -- 172.21.15.44:0/3908010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 msgr2=0x7f0d341a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d2b400640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d341a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d2b400640 1 -- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 msgr2=0x7f0d341a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d2b400640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 0x7f0d341a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d2b400640 1 -- 172.21.15.44:0/3908010151 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d341a6400 con 0x7f0d34077cc0 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d33e00640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d341a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:35:33.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d2b400640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 0x7f0d341a0a30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d3406c3a0 tx=0x7f0d1c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:33.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c00cc40 con 0x7f0d34077cc0 2024-07-26T08:35:33.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d1c00cde0 con 0x7f0d34077cc0 2024-07-26T08:35:33.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d341a66f0 con 0x7f0d34077cc0 2024-07-26T08:35:33.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.573+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c018780 con 0x7f0d34077cc0 2024-07-26T08:35:33.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.574+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d341ace50 con 0x7f0d34077cc0 2024-07-26T08:35:33.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.575+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d34077780 con 0x7f0d34077cc0 2024-07-26T08:35:33.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.577+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f0d1c010a90 con 0x7f0d34077cc0 2024-07-26T08:35:33.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.579+0000 7f0d31600640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0d180b3d30 0x7f0d180b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:33.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.579+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f0d1c0e4680 con 0x7f0d34077cc0 2024-07-26T08:35:33.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.580+0000 7f0d33e00640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0d180b3d30 0x7f0d180b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:33.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.580+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0d1c0ab010 con 0x7f0d34077cc0 2024-07-26T08:35:33.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.580+0000 7f0d33e00640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0d180b3d30 0x7f0d180b6240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0d24011550 tx=0x7f0d24005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:33.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.785+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d341ad0b0 con 0x7f0d34077cc0 2024-07-26T08:35:33.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.787+0000 7f0d31600640 1 -- 172.21.15.44:0/3908010151 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7f0d1c0af560 con 0x7f0d34077cc0 2024-07-26T08:35:33.789 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:35:33.789 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:35:33.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.789+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0d180b3d30 msgr2=0x7f0d180b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:33.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.789+0000 7f0d3b200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0d180b3d30 0x7f0d180b6240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0d24011550 tx=0x7f0d24005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:33.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.789+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 msgr2=0x7f0d341a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:33.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 0x7f0d341a0a30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d3406c3a0 tx=0x7f0d1c00ee50 comp rx=0 tx=0).stop 2024-07-26T08:35:33.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 shutdown_connections 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0d180b3d30 0x7f0d180b6240 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d34077cc0 0x7f0d341a0a30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0d341141a0 0x7f0d341a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 --2- 172.21.15.44:0/3908010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d34077370 0x7f0d341a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 >> 172.21.15.44:0/3908010151 conn(0x7f0d341011f0 msgr2=0x7f0d34103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 shutdown_connections 2024-07-26T08:35:33.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:33.790+0000 7f0d3b200640 1 -- 172.21.15.44:0/3908010151 wait complete. 2024-07-26T08:35:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:33 smithi044 ceph-mon[32404]: pgmap v389: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:33 smithi116 ceph-mon[37097]: pgmap v389: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:33 smithi171 ceph-mon[37789]: pgmap v389: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:34.644 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-26T08:35:34.645 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-26T08:35:34.645 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-26T08:35:34.645 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-26T08:35:34.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-26T08:35:34.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-26T08:35:34.647 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-26T08:35:34.647 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-26T08:35:34.649 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-26T08:35:34.649 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-26T08:35:34.650 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-26T08:35:34.650 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-26T08:35:34.811 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:34 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3908010151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:34.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:34 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3908010151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:34 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3908010151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:35:35.701 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:35.731 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:35.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:35 smithi044 ceph-mon[32404]: pgmap v390: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:35 smithi116 ceph-mon[37097]: pgmap v390: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:35 smithi171 ceph-mon[37789]: pgmap v390: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.153 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.796 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.800 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.802 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.937 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.938 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:37.940 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:38.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:37 smithi044 ceph-mon[32404]: pgmap v391: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:37 smithi116 ceph-mon[37097]: pgmap v391: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:37 smithi171 ceph-mon[37789]: pgmap v391: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:40.177 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:39 smithi044 ceph-mon[32404]: pgmap v392: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:39 smithi116 ceph-mon[37097]: pgmap v392: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:39 smithi171 ceph-mon[37789]: pgmap v392: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:41 smithi044 ceph-mon[32404]: pgmap v393: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:41 smithi116 ceph-mon[37097]: pgmap v393: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:41 smithi171 ceph-mon[37789]: pgmap v393: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:44.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:43 smithi044 ceph-mon[32404]: pgmap v394: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:43 smithi116 ceph-mon[37097]: pgmap v394: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:43 smithi171 ceph-mon[37789]: pgmap v394: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 -- 172.21.15.44:0/3336411370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4109670 msgr2=0x7f42f410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 --2- 172.21.15.44:0/3336411370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4109670 0x7f42f410fea0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f42ec009900 tx=0x7f42ec02a410 comp rx=0 tx=0).stop 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 -- 172.21.15.44:0/3336411370 shutdown_connections 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 --2- 172.21.15.44:0/3336411370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4109670 0x7f42f410fea0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 --2- 172.21.15.44:0/3336411370 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f4108d20 0x7f42f4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 --2- 172.21.15.44:0/3336411370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f41033e0 0x7f42f41037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 -- 172.21.15.44:0/3336411370 >> 172.21.15.44:0/3336411370 conn(0x7f42f40fee70 msgr2=0x7f42f41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 -- 172.21.15.44:0/3336411370 shutdown_connections 2024-07-26T08:35:44.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.444+0000 7f42fa600640 1 -- 172.21.15.44:0/3336411370 wait complete. 2024-07-26T08:35:44.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.446+0000 7f42fa600640 1 Processor -- start 2024-07-26T08:35:44.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.446+0000 7f42fa600640 1 -- start start 2024-07-26T08:35:44.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42fa600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f41033e0 0x7f42f4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:44.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42fa600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4108d20 0x7f42f4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:44.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42fa600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f4109670 0x7f42f4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42fa600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4113390 con 0x7f42f41033e0 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42fa600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4113210 con 0x7f42f4108d20 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42fa600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4113510 con 0x7f42f4109670 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f9200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f41033e0 0x7f42f4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f9200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f41033e0 0x7f42f4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60102/0 (socket says 172.21.15.44:60102) 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f9200640 1 -- 172.21.15.44:0/4063820053 learned_addr learned my addr 172.21.15.44:0/4063820053 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f3e00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4108d20 0x7f42f4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:44.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f9c00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f4109670 0x7f42f4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:44.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f3e00640 1 -- 172.21.15.44:0/4063820053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f4109670 msgr2=0x7f42f4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f3e00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42f4109670 0x7f42f4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f3e00640 1 -- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f41033e0 msgr2=0x7f42f4075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.447+0000 7f42f3e00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f41033e0 0x7f42f4075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42f3e00640 1 -- 172.21.15.44:0/4063820053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f41a7fe0 con 0x7f42f4108d20 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42f9200640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42f41033e0 0x7f42f4075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42f3e00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4108d20 0x7f42f4075680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f42dc005fd0 tx=0x7f42dc00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc009c00 con 0x7f42f4108d20 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42dc009da0 con 0x7f42f4108d20 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc003e10 con 0x7f42f4108d20 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.448+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f41a82d0 con 0x7f42f4108d20 2024-07-26T08:35:44.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.449+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f41a87c0 con 0x7f42f4108d20 2024-07-26T08:35:44.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.449+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f42f4063580 con 0x7f42f4108d20 2024-07-26T08:35:44.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.452+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f42dc004000 con 0x7f42f4108d20 2024-07-26T08:35:44.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.454+0000 7f42f1600640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42d80b3d30 0x7f42d80b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:44.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.454+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f42dc0e40b0 con 0x7f42f4108d20 2024-07-26T08:35:44.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.454+0000 7f42f9200640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42d80b3d30 0x7f42d80b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:44.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.454+0000 7f42f1600640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] conn(0x7f42d80bf3d0 0x7f42d80c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:44.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.454+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 --> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42dc015ec0 con 0x7f42d80bf3d0 2024-07-26T08:35:44.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.454+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f42dc0e4460 con 0x7f42f4108d20 2024-07-26T08:35:44.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.455+0000 7f42f9c00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] conn(0x7f42d80bf3d0 0x7f42d80c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:44.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.455+0000 7f42f9200640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42d80b3d30 0x7f42d80b6240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f42f4076b90 tx=0x7f42e4009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:44.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.456+0000 7f42f9c00640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] conn(0x7f42d80bf3d0 0x7f42d80c18c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:44.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.457+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== osd.7 v2:172.21.15.116:6816/776412453 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42dc015ec0 con 0x7f42d80bf3d0 2024-07-26T08:35:44.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.472+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 --> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f42f40639d0 con 0x7f42d80bf3d0 2024-07-26T08:35:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.473+0000 7f42f1600640 1 -- 172.21.15.44:0/4063820053 <== osd.7 v2:172.21.15.116:6816/776412453 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f42f40639d0 con 0x7f42d80bf3d0 2024-07-26T08:35:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.473+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] conn(0x7f42d80bf3d0 msgr2=0x7f42d80c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.473+0000 7f42fa600640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] conn(0x7f42d80bf3d0 0x7f42d80c18c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42d80b3d30 msgr2=0x7f42d80b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42fa600640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42d80b3d30 0x7f42d80b6240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f42f4076b90 tx=0x7f42e4009500 comp rx=0 tx=0).stop 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4108d20 msgr2=0x7f42f4075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42fa600640 1 --2- 172.21.15.44:0/4063820053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f4108d20 0x7f42f4075680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f42dc005fd0 tx=0x7f42dc00dde0 comp rx=0 tx=0).stop 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42f9c00640 1 -- 172.21.15.44:0/4063820053 reap_dead start 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 shutdown_connections 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.474+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 >> 172.21.15.44:0/4063820053 conn(0x7f42f40fee70 msgr2=0x7f42f41062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.475+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 shutdown_connections 2024-07-26T08:35:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:44.475+0000 7f42fa600640 1 -- 172.21.15.44:0/4063820053 wait complete. 2024-07-26T08:35:45.356 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 -- 172.21.15.44:0/1413915637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae8101e70 msgr2=0x7ffae8102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 --2- 172.21.15.44:0/1413915637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae8101e70 0x7ffae8102280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009900 tx=0x7ffad802a480 comp rx=0 tx=0).stop 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 -- 172.21.15.44:0/1413915637 shutdown_connections 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 --2- 172.21.15.44:0/1413915637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae810fa20 0x7ffae8111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 --2- 172.21.15.44:0/1413915637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffae81027c0 0x7ffae810f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 --2- 172.21.15.44:0/1413915637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae8101e70 0x7ffae8102280 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 -- 172.21.15.44:0/1413915637 >> 172.21.15.44:0/1413915637 conn(0x7ffae80fdcf0 msgr2=0x7ffae8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:45.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 -- 172.21.15.44:0/1413915637 shutdown_connections 2024-07-26T08:35:45.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.546+0000 7ffaed200640 1 -- 172.21.15.44:0/1413915637 wait complete. 2024-07-26T08:35:45.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.548+0000 7ffaed200640 1 Processor -- start 2024-07-26T08:35:45.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.548+0000 7ffaed200640 1 -- start start 2024-07-26T08:35:45.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.548+0000 7ffaed200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffae8101e70 0x7ffae819bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:45.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.548+0000 7ffaed200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae81027c0 0x7ffae819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:45.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.548+0000 7ffaed200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae810fa20 0x7ffae81a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:45.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.548+0000 7ffaed200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81154b0 con 0x7ffae8101e70 2024-07-26T08:35:45.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffaed200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8115330 con 0x7ffae810fa20 2024-07-26T08:35:45.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffaed200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8115630 con 0x7ffae81027c0 2024-07-26T08:35:45.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffae6a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae81027c0 0x7ffae819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:45.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffae6a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae81027c0 0x7ffae819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35598/0 (socket says 172.21.15.44:35598) 2024-07-26T08:35:45.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffae6a00640 1 -- 172.21.15.44:0/3642938278 learned_addr learned my addr 172.21.15.44:0/3642938278 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:45.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffae7e00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae810fa20 0x7ffae81a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:45.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.549+0000 7ffae7400640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffae8101e70 0x7ffae819bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffae6a00640 1 -- 172.21.15.44:0/3642938278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae810fa20 msgr2=0x7ffae81a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffae6a00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae810fa20 0x7ffae81a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffae6a00640 1 -- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffae8101e70 msgr2=0x7ffae819bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffae6a00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffae8101e70 0x7ffae819bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffae6a00640 1 -- 172.21.15.44:0/3642938278 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae81a1e10 con 0x7ffae81027c0 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffae6a00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae81027c0 0x7ffae819c440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffad400e9f0 tx=0x7ffad400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.550+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad400cd80 con 0x7ffae81027c0 2024-07-26T08:35:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.551+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffad40054d0 con 0x7ffae81027c0 2024-07-26T08:35:45.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.551+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae81a2100 con 0x7ffae81027c0 2024-07-26T08:35:45.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.551+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4010690 con 0x7ffae81027c0 2024-07-26T08:35:45.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.551+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae81a8860 con 0x7ffae81027c0 2024-07-26T08:35:45.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.552+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffae8111ec0 con 0x7ffae81027c0 2024-07-26T08:35:45.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.555+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7ffad4004e90 con 0x7ffae81027c0 2024-07-26T08:35:45.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.559+0000 7ffadfe00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffacc0b3d30 0x7ffacc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.559+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ffad40e4780 con 0x7ffae81027c0 2024-07-26T08:35:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.561+0000 7ffae7400640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffacc0b3d30 0x7ffacc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.561+0000 7ffadfe00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] conn(0x7ffacc0bf3d0 0x7ffacc0c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.561+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 --> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffad4005320 con 0x7ffacc0bf3d0 2024-07-26T08:35:45.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.561+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ffad40eb050 con 0x7ffae81027c0 2024-07-26T08:35:45.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.562+0000 7ffae7e00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] conn(0x7ffacc0bf3d0 0x7ffacc0c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.563+0000 7ffae7400640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffacc0b3d30 0x7ffacc0b6240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffad8031440 tx=0x7ffad8005d50 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.563+0000 7ffae7e00640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] conn(0x7ffacc0bf3d0 0x7ffacc0c18c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.564+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== osd.0 v2:172.21.15.44:6802/3315928929 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffad4005320 con 0x7ffacc0bf3d0 2024-07-26T08:35:45.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.581+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 --> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffae806af30 con 0x7ffacc0bf3d0 2024-07-26T08:35:45.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.582+0000 7ffadfe00640 1 -- 172.21.15.44:0/3642938278 <== osd.0 v2:172.21.15.44:6802/3315928929 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ffae806af30 con 0x7ffacc0bf3d0 2024-07-26T08:35:45.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] conn(0x7ffacc0bf3d0 msgr2=0x7ffacc0c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:45.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] conn(0x7ffacc0bf3d0 0x7ffacc0c18c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffacc0b3d30 msgr2=0x7ffacc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffacc0b3d30 0x7ffacc0b6240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffad8031440 tx=0x7ffad8005d50 comp rx=0 tx=0).stop 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae81027c0 msgr2=0x7ffae819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 --2- 172.21.15.44:0/3642938278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffae81027c0 0x7ffae819c440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffad400e9f0 tx=0x7ffad400edf0 comp rx=0 tx=0).stop 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffae7e00640 1 -- 172.21.15.44:0/3642938278 reap_dead start 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 shutdown_connections 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.583+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 >> 172.21.15.44:0/3642938278 conn(0x7ffae80fdcf0 msgr2=0x7ffae8100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:45.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.584+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 shutdown_connections 2024-07-26T08:35:45.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.585+0000 7ffaed200640 1 -- 172.21.15.44:0/3642938278 wait complete. 2024-07-26T08:35:46.235 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:45 smithi044 ceph-mon[32404]: pgmap v395: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:45 smithi116 ceph-mon[37097]: pgmap v395: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:45 smithi171 ceph-mon[37789]: pgmap v395: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.893+0000 7fd9ae000640 1 -- 172.21.15.44:0/2669181671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 msgr2=0x7fd9a810fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.893+0000 7fd9ae000640 1 --2- 172.21.15.44:0/2669181671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 0x7fd9a810fea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009be0 tx=0x7fd9a002a6c0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 -- 172.21.15.44:0/2669181671 shutdown_connections 2024-07-26T08:35:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 --2- 172.21.15.44:0/2669181671 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 0x7fd9a810fea0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 --2- 172.21.15.44:0/2669181671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a8108d20 0x7fd9a8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 --2- 172.21.15.44:0/2669181671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9a81033e0 0x7fd9a81037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 -- 172.21.15.44:0/2669181671 >> 172.21.15.44:0/2669181671 conn(0x7fd9a80fee70 msgr2=0x7fd9a81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 -- 172.21.15.44:0/2669181671 shutdown_connections 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.894+0000 7fd9ae000640 1 -- 172.21.15.44:0/2669181671 wait complete. 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.895+0000 7fd9ae000640 1 Processor -- start 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 -- start start 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a81033e0 0x7fd9a819bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9a8108d20 0x7fd9a819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 0x7fd9a81a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a8113320 con 0x7fd9a8108d20 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a81131a0 con 0x7fd9a81033e0 2024-07-26T08:35:46.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.896+0000 7fd9ae000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a81134a0 con 0x7fd9a8109670 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a81033e0 0x7fd9a819bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a81033e0 0x7fd9a819bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:57540/0 (socket says 172.21.15.44:57540) 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 -- 172.21.15.44:0/432446597 learned_addr learned my addr 172.21.15.44:0/432446597 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9a7e00640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9a8108d20 0x7fd9a819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9ad600640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 0x7fd9a81a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 -- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 msgr2=0x7fd9a81a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 0x7fd9a81a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 -- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9a8108d20 msgr2=0x7fd9a819c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd9a8108d20 0x7fd9a819c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9acc00640 1 -- 172.21.15.44:0/432446597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a81a1e60 con 0x7fd9a81033e0 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.897+0000 7fd9ad600640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9a8109670 0x7fd9a81a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.898+0000 7fd9acc00640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a81033e0 0x7fd9a819bef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00e9c0 tx=0x7fd99c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:46.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.898+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00cde0 con 0x7fd9a81033e0 2024-07-26T08:35:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.898+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd99c0035f0 con 0x7fd9a81033e0 2024-07-26T08:35:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.898+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c010690 con 0x7fd9a81033e0 2024-07-26T08:35:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.898+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a81a2150 con 0x7fd9a81033e0 2024-07-26T08:35:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.898+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a81a88b0 con 0x7fd9a81033e0 2024-07-26T08:35:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.899+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9a810f480 con 0x7fd9a81033e0 2024-07-26T08:35:46.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.902+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fd99c004000 con 0x7fd9a81033e0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.903+0000 7fd9a5600640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd9900b3d30 0x7fd9900b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.903+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fd99c0e48f0 con 0x7fd9a81033e0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.903+0000 7fd9a7e00640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd9900b3d30 0x7fd9900b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.904+0000 7fd9a5600640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] conn(0x7fd9900bf3d0 0x7fd9900c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.904+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 --> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd99c023e50 con 0x7fd9900bf3d0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.904+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd99c0e4ca0 con 0x7fd9a81033e0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.904+0000 7fd9a7e00640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd9900b3d30 0x7fd9900b6240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd9940095f0 tx=0x7fd994005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.904+0000 7fd9ad600640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] conn(0x7fd9900bf3d0 0x7fd9900c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.905+0000 7fd9ad600640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] conn(0x7fd9900bf3d0 0x7fd9900c18c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.906+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== osd.5 v2:172.21.15.171:6808/3188487524 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd99c023e50 con 0x7fd9900bf3d0 2024-07-26T08:35:46.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.922+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 --> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9a8104660 con 0x7fd9900bf3d0 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9a5600640 1 -- 172.21.15.44:0/432446597 <== osd.5 v2:172.21.15.171:6808/3188487524 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd9a8104660 con 0x7fd9900bf3d0 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] conn(0x7fd9900bf3d0 msgr2=0x7fd9900c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ae000640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] conn(0x7fd9900bf3d0 0x7fd9900c18c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd9900b3d30 msgr2=0x7fd9900b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ae000640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd9900b3d30 0x7fd9900b6240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd9940095f0 tx=0x7fd994005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a81033e0 msgr2=0x7fd9a819bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ae000640 1 --2- 172.21.15.44:0/432446597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9a81033e0 0x7fd9a819bef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00e9c0 tx=0x7fd99c00ee80 comp rx=0 tx=0).stop 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.923+0000 7fd9ad600640 1 -- 172.21.15.44:0/432446597 reap_dead start 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.924+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 shutdown_connections 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.924+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 >> 172.21.15.44:0/432446597 conn(0x7fd9a80fee70 msgr2=0x7fd9a8108350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.924+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 shutdown_connections 2024-07-26T08:35:46.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:45.924+0000 7fd9ae000640 1 -- 172.21.15.44:0/432446597 wait complete. 2024-07-26T08:35:48.165 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:47 smithi044 ceph-mon[32404]: pgmap v396: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:35:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:47 smithi116 ceph-mon[37097]: pgmap v396: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:35:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:47 smithi171 ceph-mon[37789]: pgmap v396: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:35:48.874 INFO:teuthology.orchestra.run.smithi044.stdout:163208757330 2024-07-26T08:35:48.875 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-26T08:35:49.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 -- 172.21.15.44:0/3736170488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 msgr2=0x7f48e4107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:49.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 --2- 172.21.15.44:0/3736170488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 0x7f48e4107ff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009860 tx=0x7f48cc02a340 comp rx=0 tx=0).stop 2024-07-26T08:35:49.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 -- 172.21.15.44:0/3736170488 shutdown_connections 2024-07-26T08:35:49.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 --2- 172.21.15.44:0/3736170488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 0x7f48e4107ff0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 --2- 172.21.15.44:0/3736170488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48e4108530 0x7f48e410ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 --2- 172.21.15.44:0/3736170488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e41022a0 0x7f48e41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 -- 172.21.15.44:0/3736170488 >> 172.21.15.44:0/3736170488 conn(0x7f48e40fdcf0 msgr2=0x7f48e4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:49.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 -- 172.21.15.44:0/3736170488 shutdown_connections 2024-07-26T08:35:49.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.592+0000 7f48ea400640 1 -- 172.21.15.44:0/3736170488 wait complete. 2024-07-26T08:35:49.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.594+0000 7f48ea400640 1 Processor -- start 2024-07-26T08:35:49.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 -- start start 2024-07-26T08:35:49.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48e41022a0 0x7f48e419bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:49.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 0x7f48e419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:49.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4108530 0x7f48e41a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:49.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41121c0 con 0x7f48e41022a0 2024-07-26T08:35:49.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4112040 con 0x7f48e4108530 2024-07-26T08:35:49.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48ea400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4112340 con 0x7f48e4107be0 2024-07-26T08:35:49.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48e9000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48e41022a0 0x7f48e419bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:49.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.595+0000 7f48dbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 0x7f48e419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:49.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48dbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 0x7f48e419c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35662/0 (socket says 172.21.15.44:35662) 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48dbe00640 1 -- 172.21.15.44:0/2427726475 learned_addr learned my addr 172.21.15.44:0/2427726475 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48e9a00640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4108530 0x7f48e41a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48e9000640 1 -- 172.21.15.44:0/2427726475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 msgr2=0x7f48e419c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48e9000640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4107be0 0x7f48e419c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48e9000640 1 -- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4108530 msgr2=0x7f48e41a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48e9000640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48e4108530 0x7f48e41a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.596+0000 7f48e9000640 1 -- 172.21.15.44:0/2427726475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e41a1f40 con 0x7f48e41022a0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.597+0000 7f48e9000640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48e41022a0 0x7f48e419bfd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f48d400e9f0 tx=0x7f48d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.597+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d400ccc0 con 0x7f48e41022a0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.597+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e41a2230 con 0x7f48e41022a0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.597+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d4005020 con 0x7f48e41022a0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.597+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4004a00 con 0x7f48e41022a0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.597+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e41a8990 con 0x7f48e41022a0 2024-07-26T08:35:49.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.598+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f48e4103dc0 con 0x7f48e41022a0 2024-07-26T08:35:49.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.601+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f48d4004ba0 con 0x7f48e41022a0 2024-07-26T08:35:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.602+0000 7f48e2000640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f48c80b3d30 0x7f48c80b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.602+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f48d40e4460 con 0x7f48e41022a0 2024-07-26T08:35:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.603+0000 7f48dbe00640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f48c80b3d30 0x7f48c80b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.603+0000 7f48e2000640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] conn(0x7f48c80bf3c0 0x7f48c80c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.603+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 --> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48d4023ea0 con 0x7f48c80bf3c0 2024-07-26T08:35:49.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.603+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f48d40e4810 con 0x7f48e41022a0 2024-07-26T08:35:49.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.603+0000 7f48e9a00640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] conn(0x7f48c80bf3c0 0x7f48c80c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:49.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.604+0000 7f48dbe00640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f48c80b3d30 0x7f48c80b6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0057c0 tx=0x7f48cc0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:49.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.604+0000 7f48e9a00640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] conn(0x7f48c80bf3c0 0x7f48c80c18b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:49.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.605+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== osd.1 v2:172.21.15.116:6800/2431548674 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f48d4023ea0 con 0x7f48c80bf3c0 2024-07-26T08:35:49.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.621+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 --> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f48e4103520 con 0x7f48c80bf3c0 2024-07-26T08:35:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48e2000640 1 -- 172.21.15.44:0/2427726475 <== osd.1 v2:172.21.15.116:6800/2431548674 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f48e4103520 con 0x7f48c80bf3c0 2024-07-26T08:35:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] conn(0x7f48c80bf3c0 msgr2=0x7f48c80c18b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48ea400640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] conn(0x7f48c80bf3c0 0x7f48c80c18b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f48c80b3d30 msgr2=0x7f48c80b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48ea400640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f48c80b3d30 0x7f48c80b6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0057c0 tx=0x7f48cc0058d0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48e41022a0 msgr2=0x7f48e419bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48ea400640 1 --2- 172.21.15.44:0/2427726475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f48e41022a0 0x7f48e419bfd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f48d400e9f0 tx=0x7f48d400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:35:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.622+0000 7f48e9a00640 1 -- 172.21.15.44:0/2427726475 reap_dead start 2024-07-26T08:35:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.623+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 shutdown_connections 2024-07-26T08:35:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.623+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 >> 172.21.15.44:0/2427726475 conn(0x7f48e40fdcf0 msgr2=0x7f48e4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.623+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 shutdown_connections 2024-07-26T08:35:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:49.623+0000 7f48ea400640 1 -- 172.21.15.44:0/2427726475 wait complete. 2024-07-26T08:35:50.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:49 smithi044 ceph-mon[32404]: pgmap v397: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:50.196 INFO:teuthology.orchestra.run.smithi044.stdout:38654705772 2024-07-26T08:35:50.197 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-26T08:35:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:49 smithi116 ceph-mon[37097]: pgmap v397: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:49 smithi171 ceph-mon[37789]: pgmap v397: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 -- 172.21.15.44:0/2842435684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd8109650 msgr2=0x7ffbd810fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 --2- 172.21.15.44:0/2842435684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd8109650 0x7ffbd810fe80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009ca0 tx=0x7ffbd002a8b0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 -- 172.21.15.44:0/2842435684 shutdown_connections 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 --2- 172.21.15.44:0/2842435684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd8109650 0x7ffbd810fe80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 --2- 172.21.15.44:0/2842435684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd8108d00 0x7ffbd8109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 --2- 172.21.15.44:0/2842435684 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd81033c0 0x7ffbd81037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 -- 172.21.15.44:0/2842435684 >> 172.21.15.44:0/2842435684 conn(0x7ffbd80fee10 msgr2=0x7ffbd8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 -- 172.21.15.44:0/2842435684 shutdown_connections 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.161+0000 7ffbdd800640 1 -- 172.21.15.44:0/2842435684 wait complete. 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.162+0000 7ffbdd800640 1 Processor -- start 2024-07-26T08:35:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.162+0000 7ffbdd800640 1 -- start start 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd81033c0 0x7ffbd8199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd8108d00 0x7ffbd819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdd800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd8109650 0x7ffbd819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdd800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd8113370 con 0x7ffbd81033c0 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdd800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd81131f0 con 0x7ffbd8108d00 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdd800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd81134f0 con 0x7ffbd8109650 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd81033c0 0x7ffbd8199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdce00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd8109650 0x7ffbd819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbd7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd8108d00 0x7ffbd819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdce00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd8109650 0x7ffbd819f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35646/0 (socket says 172.21.15.44:35646) 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbdce00640 1 -- 172.21.15.44:0/1632550976 learned_addr learned my addr 172.21.15.44:0/1632550976 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.163+0000 7ffbd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd81033c0 0x7ffbd8199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60148/0 (socket says 172.21.15.44:60148) 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbdce00640 1 -- 172.21.15.44:0/1632550976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd8108d00 msgr2=0x7ffbd819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbdce00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd8108d00 0x7ffbd819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbdce00640 1 -- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd81033c0 msgr2=0x7ffbd8199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbdce00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbd81033c0 0x7ffbd8199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbdce00640 1 -- 172.21.15.44:0/1632550976 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd819fc50 con 0x7ffbd8109650 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbd7400640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd8108d00 0x7ffbd819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.164+0000 7ffbdce00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd8109650 0x7ffbd819f520 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0005e30 tx=0x7ffbd0030dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.165+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0004040 con 0x7ffbd8109650 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.165+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd819fee0 con 0x7ffbd8109650 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.165+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbd0040070 con 0x7ffbd8109650 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.165+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0045a80 con 0x7ffbd8109650 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.165+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd81a66a0 con 0x7ffbd8109650 2024-07-26T08:35:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.166+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffbd8063540 con 0x7ffbd8109650 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.169+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7ffbd00041e0 con 0x7ffbd8109650 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.170+0000 7ffbd4c00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbbc0b3d30 0x7ffbbc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.170+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ffbd01058e0 con 0x7ffbd8109650 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.171+0000 7ffbd7e00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbbc0b3d30 0x7ffbbc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.171+0000 7ffbd4c00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] conn(0x7ffbbc0bf3d0 0x7ffbbc0c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.171+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 --> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffbd004ae50 con 0x7ffbbc0bf3d0 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.171+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ffbd0105c90 con 0x7ffbd8109650 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.171+0000 7ffbd7e00640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbbc0b3d30 0x7ffbbc0b6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009bd0 tx=0x7ffbc8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.180+0000 7ffbd7400640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] conn(0x7ffbbc0bf3d0 0x7ffbbc0c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.188+0000 7ffbd7400640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] conn(0x7ffbbc0bf3d0 0x7ffbbc0c18c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.189+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== osd.8 v2:172.21.15.171:6816/2299573831 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffbd004ae50 con 0x7ffbbc0bf3d0 2024-07-26T08:35:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.205+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 --> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffbd8063940 con 0x7ffbbc0bf3d0 2024-07-26T08:35:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.206+0000 7ffbd4c00640 1 -- 172.21.15.44:0/1632550976 <== osd.8 v2:172.21.15.171:6816/2299573831 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ffbd8063940 con 0x7ffbbc0bf3d0 2024-07-26T08:35:52.014 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:51 smithi044 ceph-mon[32404]: pgmap v398: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.206+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] conn(0x7ffbbc0bf3d0 msgr2=0x7ffbbc0c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.206+0000 7ffbdd800640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] conn(0x7ffbbc0bf3d0 0x7ffbbc0c18c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbbc0b3d30 msgr2=0x7ffbbc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbbc0b3d30 0x7ffbbc0b6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009bd0 tx=0x7ffbc8005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd8109650 msgr2=0x7ffbd819f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 --2- 172.21.15.44:0/1632550976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbd8109650 0x7ffbd819f520 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0005e30 tx=0x7ffbd0030dc0 comp rx=0 tx=0).stop 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdce00640 1 -- 172.21.15.44:0/1632550976 reap_dead start 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 shutdown_connections 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 >> 172.21.15.44:0/1632550976 conn(0x7ffbd80fee10 msgr2=0x7ffbd8108350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 shutdown_connections 2024-07-26T08:35:52.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:47.207+0000 7ffbdd800640 1 -- 172.21.15.44:0/1632550976 wait complete. 2024-07-26T08:35:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:51 smithi116 ceph-mon[37097]: pgmap v398: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:51 smithi171 ceph-mon[37789]: pgmap v398: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:35:53.274 INFO:teuthology.orchestra.run.smithi044.stdout:236223201347 2024-07-26T08:35:53.275 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 -- 172.21.15.44:0/463350176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 msgr2=0x7f9b84118580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 --2- 172.21.15.44:0/463350176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 0x7f9b84118580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009be0 tx=0x7f9b7c02a6c0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 -- 172.21.15.44:0/463350176 shutdown_connections 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 --2- 172.21.15.44:0/463350176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b8410a910 0x7f9b84115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 --2- 172.21.15.44:0/463350176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 0x7f9b84118580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 --2- 172.21.15.44:0/463350176 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b84109fc0 0x7f9b8410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 -- 172.21.15.44:0/463350176 >> 172.21.15.44:0/463350176 conn(0x7f9b84070d20 msgr2=0x7f9b84073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:53.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 -- 172.21.15.44:0/463350176 shutdown_connections 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.020+0000 7f9b89a00640 1 -- 172.21.15.44:0/463350176 wait complete. 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.021+0000 7f9b89a00640 1 Processor -- start 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 -- start start 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b84109fc0 0x7f9b8410b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b8410a910 0x7f9b8410bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 0x7f9b8410c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8411bd10 con 0x7f9b8410a910 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8411bb90 con 0x7f9b84116070 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.022+0000 7f9b89a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8411be90 con 0x7f9b84109fc0 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b8410a910 0x7f9b8410bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b84109fc0 0x7f9b8410b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:53.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b8410a910 0x7f9b8410bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56694/0 (socket says 172.21.15.44:56694) 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82000640 1 -- 172.21.15.44:0/2482040096 learned_addr learned my addr 172.21.15.44:0/2482040096 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b83400640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 0x7f9b8410c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82a00640 1 -- 172.21.15.44:0/2482040096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 msgr2=0x7f9b8410c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82a00640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b84116070 0x7f9b8410c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82a00640 1 -- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b8410a910 msgr2=0x7f9b8410bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82a00640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b8410a910 0x7f9b8410bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.023+0000 7f9b82a00640 1 -- 172.21.15.44:0/2482040096 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b841b09a0 con 0x7f9b84109fc0 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.024+0000 7f9b82a00640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b84109fc0 0x7f9b8410b8b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9b74005fd0 tx=0x7f9b7400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.024+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400cd00 con 0x7f9b84109fc0 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.024+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b841b0c90 con 0x7f9b84109fc0 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.024+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b740035f0 con 0x7f9b84109fc0 2024-07-26T08:35:53.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.024+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74010690 con 0x7f9b84109fc0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.024+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b841b11c0 con 0x7f9b84109fc0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.025+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b840737d0 con 0x7f9b84109fc0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.028+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f9b74003aa0 con 0x7f9b84109fc0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.029+0000 7f9b7b400640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b680b3cb0 0x7f9b680b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.030+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9b740e47f0 con 0x7f9b84109fc0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.030+0000 7f9b82000640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b680b3cb0 0x7f9b680b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.030+0000 7f9b7b400640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] conn(0x7f9b680bf350 0x7f9b680c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.030+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 --> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b74023e50 con 0x7f9b680bf350 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.030+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9b740e4ba0 con 0x7f9b84109fc0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.030+0000 7f9b83400640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] conn(0x7f9b680bf350 0x7f9b680c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.031+0000 7f9b82000640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b680b3cb0 0x7f9b680b61c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9b84118530 tx=0x7f9b70005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:53.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.031+0000 7f9b83400640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] conn(0x7f9b680bf350 0x7f9b680c1840 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:53.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.032+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== osd.2 v2:172.21.15.171:6800/2604165495 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b74023e50 con 0x7f9b680bf350 2024-07-26T08:35:53.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.047+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 --> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b8410b010 con 0x7f9b680bf350 2024-07-26T08:35:53.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.048+0000 7f9b7b400640 1 -- 172.21.15.44:0/2482040096 <== osd.2 v2:172.21.15.171:6800/2604165495 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9b8410b010 con 0x7f9b680bf350 2024-07-26T08:35:53.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] conn(0x7f9b680bf350 msgr2=0x7f9b680c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:53.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b89a00640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] conn(0x7f9b680bf350 0x7f9b680c1840 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:53.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b680b3cb0 msgr2=0x7f9b680b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b89a00640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b680b3cb0 0x7f9b680b61c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9b84118530 tx=0x7f9b70005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b84109fc0 msgr2=0x7f9b8410b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b89a00640 1 --2- 172.21.15.44:0/2482040096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b84109fc0 0x7f9b8410b8b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9b74005fd0 tx=0x7f9b7400ee50 comp rx=0 tx=0).stop 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.049+0000 7f9b83400640 1 -- 172.21.15.44:0/2482040096 reap_dead start 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.050+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 shutdown_connections 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.050+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 >> 172.21.15.44:0/2482040096 conn(0x7f9b84070d20 msgr2=0x7f9b84072d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.050+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 shutdown_connections 2024-07-26T08:35:53.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:53.050+0000 7f9b89a00640 1 -- 172.21.15.44:0/2482040096 wait complete. 2024-07-26T08:35:53.927 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:35:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:53 smithi044 ceph-mon[32404]: pgmap v399: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:53 smithi116 ceph-mon[37097]: pgmap v399: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:53 smithi171 ceph-mon[37789]: pgmap v399: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:35:55.514 INFO:teuthology.orchestra.run.smithi044.stdout:210453397576 2024-07-26T08:35:55.514 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-26T08:35:55.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:55 smithi044 ceph-mon[32404]: pgmap v400: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:55 smithi116 ceph-mon[37097]: pgmap v400: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:55 smithi171 ceph-mon[37789]: pgmap v400: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 -- 172.21.15.44:0/1778826592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 msgr2=0x7f395c109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 --2- 172.21.15.44:0/1778826592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 0x7f395c109e00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3950009900 tx=0x7f395002a410 comp rx=0 tx=0).stop 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 -- 172.21.15.44:0/1778826592 shutdown_connections 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 --2- 172.21.15.44:0/1778826592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10fc80 0x7f395c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 --2- 172.21.15.44:0/1778826592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f395c10f330 0x7f395c10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 --2- 172.21.15.44:0/1778826592 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 0x7f395c109e00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 -- 172.21.15.44:0/1778826592 >> 172.21.15.44:0/1778826592 conn(0x7f395c078ff0 msgr2=0x7f395c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:56.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 -- 172.21.15.44:0/1778826592 shutdown_connections 2024-07-26T08:35:56.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.349+0000 7f3962c00640 1 -- 172.21.15.44:0/1778826592 wait complete. 2024-07-26T08:35:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.350+0000 7f3962c00640 1 Processor -- start 2024-07-26T08:35:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.350+0000 7f3962c00640 1 -- start start 2024-07-26T08:35:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 0x7f395c1a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10f330 0x7f395c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f395c10fc80 0x7f395c1a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f395c119940 con 0x7f395c10fc80 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f395c1197c0 con 0x7f395c10f330 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962c00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f395c119ac0 con 0x7f395c1099f0 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3960e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10f330 0x7f395c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3962200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f395c10fc80 0x7f395c1a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3960e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10f330 0x7f395c1a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:48020/0 (socket says 172.21.15.44:48020) 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3961800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 0x7f395c1a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.351+0000 7f3960e00640 1 -- 172.21.15.44:0/1710261464 learned_addr learned my addr 172.21.15.44:0/1710261464 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.352+0000 7f3960e00640 1 -- 172.21.15.44:0/1710261464 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 msgr2=0x7f395c1a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.352+0000 7f3960e00640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f395c1099f0 0x7f395c1a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.352+0000 7f3960e00640 1 -- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f395c10fc80 msgr2=0x7f395c1a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.352+0000 7f3960e00640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f395c10fc80 0x7f395c1a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.352+0000 7f3960e00640 1 -- 172.21.15.44:0/1710261464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c1a62c0 con 0x7f395c10f330 2024-07-26T08:35:56.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.352+0000 7f3960e00640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10f330 0x7f395c1a08f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f394800e9f0 tx=0x7f394800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.353+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394800ccc0 con 0x7f395c10f330 2024-07-26T08:35:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.353+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395c1a65b0 con 0x7f395c10f330 2024-07-26T08:35:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.353+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39480108e0 con 0x7f395c10f330 2024-07-26T08:35:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.353+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948018880 con 0x7f395c10f330 2024-07-26T08:35:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.353+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395c1acd10 con 0x7f395c10f330 2024-07-26T08:35:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.354+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f393c001060 con 0x7f395c10f330 2024-07-26T08:35:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.330+0000 7fa688a00640 1 -- 172.21.15.44:0/3499400393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa684109670 msgr2=0x7fa68410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.330+0000 7fa688a00640 1 --2- 172.21.15.44:0/3499400393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa684109670 0x7fa68410fea0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009ca0 tx=0x7fa67c02a8c0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 -- 172.21.15.44:0/3499400393 shutdown_connections 2024-07-26T08:35:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 --2- 172.21.15.44:0/3499400393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa684109670 0x7fa68410fea0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 --2- 172.21.15.44:0/3499400393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 0x7fa684109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 --2- 172.21.15.44:0/3499400393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa6841033e0 0x7fa6841037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 -- 172.21.15.44:0/3499400393 >> 172.21.15.44:0/3499400393 conn(0x7fa6840fee30 msgr2=0x7fa684101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 -- 172.21.15.44:0/3499400393 shutdown_connections 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.331+0000 7fa688a00640 1 -- 172.21.15.44:0/3499400393 wait complete. 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.332+0000 7fa688a00640 1 Processor -- start 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 -- start start 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6841033e0 0x7fa684075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 0x7fa684075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa684109670 0x7fa684075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa684113320 con 0x7fa6841033e0 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841131a0 con 0x7fa684108d20 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.333+0000 7fa688a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841134a0 con 0x7fa684109670 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 0x7fa684075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa683400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6841033e0 0x7fa684075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 0x7fa684075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:48002/0 (socket says 172.21.15.44:48002) 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 -- 172.21.15.44:0/232166757 learned_addr learned my addr 172.21.15.44:0/232166757 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa683e00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa684109670 0x7fa684075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 -- 172.21.15.44:0/232166757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa684109670 msgr2=0x7fa684075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa684109670 0x7fa684075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 -- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6841033e0 msgr2=0x7fa684075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa6841033e0 0x7fa684075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.334+0000 7fa682a00640 1 -- 172.21.15.44:0/232166757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6841a7f50 con 0x7fa684108d20 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.335+0000 7fa682a00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 0x7fa684075680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa67000e9f0 tx=0x7fa67000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.335+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67000ccc0 con 0x7fa684108d20 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.335+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6841a8240 con 0x7fa684108d20 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.335+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6700108e0 con 0x7fa684108d20 2024-07-26T08:35:56.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.335+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670018860 con 0x7fa684108d20 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.335+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6841a87d0 con 0x7fa684108d20 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.336+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa684063580 con 0x7fa684108d20 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.340+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fa670010430 con 0x7fa684108d20 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.341+0000 7fa67be00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6680b3d30 0x7fa6680b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.342+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fa6700e4cb0 con 0x7fa684108d20 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.342+0000 7fa683400640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6680b3d30 0x7fa6680b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.342+0000 7fa67be00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] conn(0x7fa6680bf3d0 0x7fa6680c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.342+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 --> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6680c1f90 con 0x7fa6680bf3d0 2024-07-26T08:35:56.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.342+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa6700e5060 con 0x7fa684108d20 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.342+0000 7fa683e00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] conn(0x7fa6680bf3d0 0x7fa6680c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.343+0000 7fa683400640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6680b3d30 0x7fa6680b6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa684076b90 tx=0x7fa674005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.343+0000 7fa683e00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] conn(0x7fa6680bf3d0 0x7fa6680c18c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.344+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== osd.4 v2:172.21.15.116:6808/3661934334 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa6680c1f90 con 0x7fa6680bf3d0 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.360+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 --> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa684063980 con 0x7fa6680bf3d0 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.360+0000 7fa67be00640 1 -- 172.21.15.44:0/232166757 <== osd.4 v2:172.21.15.116:6808/3661934334 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa684063980 con 0x7fa6680bf3d0 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] conn(0x7fa6680bf3d0 msgr2=0x7fa6680c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] conn(0x7fa6680bf3d0 0x7fa6680c18c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6680b3d30 msgr2=0x7fa6680b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa6680b3d30 0x7fa6680b6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa684076b90 tx=0x7fa674005f90 comp rx=0 tx=0).stop 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 msgr2=0x7fa684075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 --2- 172.21.15.44:0/232166757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa684108d20 0x7fa684075680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa67000e9f0 tx=0x7fa67000eeb0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa683e00640 1 -- 172.21.15.44:0/232166757 reap_dead start 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 shutdown_connections 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.361+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 >> 172.21.15.44:0/232166757 conn(0x7fa6840fee30 msgr2=0x7fa684101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.362+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 shutdown_connections 2024-07-26T08:35:56.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:55.362+0000 7fa688a00640 1 -- 172.21.15.44:0/232166757 wait complete. 2024-07-26T08:35:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.358+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f3948010430 con 0x7f395c10f330 2024-07-26T08:35:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.359+0000 7f395a000640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39440b3d30 0x7f39440b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.360+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f39480e4c10 con 0x7f395c10f330 2024-07-26T08:35:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.360+0000 7f3961800640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39440b3d30 0x7f39440b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.360+0000 7f395a000640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] conn(0x7f39440bf3d0 0x7f39440c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:35:56.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.360+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 --> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3948028ea0 con 0x7f39440bf3d0 2024-07-26T08:35:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.360+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f39480e5060 con 0x7f395c10f330 2024-07-26T08:35:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.360+0000 7f3962200640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] conn(0x7f39440bf3d0 0x7f39440c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:35:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.361+0000 7f3961800640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39440b3d30 0x7f39440b6240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3950031440 tx=0x7f3950009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.361+0000 7f3962200640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] conn(0x7f39440bf3d0 0x7f39440c18c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:35:56.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.362+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== osd.10 v2:172.21.15.116:6824/1630121926 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3948028ea0 con 0x7f39440bf3d0 2024-07-26T08:35:56.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.378+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 --> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f393c002e00 con 0x7f39440bf3d0 2024-07-26T08:35:56.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.378+0000 7f395a000640 1 -- 172.21.15.44:0/1710261464 <== osd.10 v2:172.21.15.116:6824/1630121926 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f393c002e00 con 0x7f39440bf3d0 2024-07-26T08:35:56.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] conn(0x7f39440bf3d0 msgr2=0x7f39440c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962c00640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] conn(0x7f39440bf3d0 0x7f39440c18c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39440b3d30 msgr2=0x7f39440b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962c00640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39440b3d30 0x7f39440b6240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3950031440 tx=0x7f3950009560 comp rx=0 tx=0).stop 2024-07-26T08:35:56.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10f330 msgr2=0x7f395c1a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:35:56.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962c00640 1 --2- 172.21.15.44:0/1710261464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f395c10f330 0x7f395c1a08f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f394800e9f0 tx=0x7f394800edf0 comp rx=0 tx=0).stop 2024-07-26T08:35:56.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.379+0000 7f3962200640 1 -- 172.21.15.44:0/1710261464 reap_dead start 2024-07-26T08:35:56.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.380+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 shutdown_connections 2024-07-26T08:35:56.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.380+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 >> 172.21.15.44:0/1710261464 conn(0x7f395c078ff0 msgr2=0x7f395c107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:35:56.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.380+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 shutdown_connections 2024-07-26T08:35:56.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:35:56.380+0000 7f3962c00640 1 -- 172.21.15.44:0/1710261464 wait complete. 2024-07-26T08:35:57.528 INFO:teuthology.orchestra.run.smithi044.stdout:81604378726 2024-07-26T08:35:57.528 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-26T08:35:58.070 INFO:teuthology.orchestra.run.smithi044.stdout:60129542249 2024-07-26T08:35:58.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-26T08:35:58.214 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:57 smithi044 ceph-mon[32404]: pgmap v401: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T08:35:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:57 smithi116 ceph-mon[37097]: pgmap v401: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T08:35:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:57 smithi171 ceph-mon[37789]: pgmap v401: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T08:35:59.133 INFO:teuthology.orchestra.run.smithi044.stdout:283467841592 2024-07-26T08:35:59.133 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-26T08:35:59.765 INFO:teuthology.orchestra.run.smithi044.stdout:141733920857 2024-07-26T08:35:59.765 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-26T08:36:00.110 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:35:59 smithi044 ceph-mon[32404]: pgmap v402: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:35:59 smithi116 ceph-mon[37097]: pgmap v402: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 -- 172.21.15.44:0/3189199756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c841099f0 msgr2=0x7f8c84109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 --2- 172.21.15.44:0/3189199756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c841099f0 0x7f8c84109e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009900 tx=0x7f8c7402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 -- 172.21.15.44:0/3189199756 shutdown_connections 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 --2- 172.21.15.44:0/3189199756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8410fc80 0x7f8c841164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 --2- 172.21.15.44:0/3189199756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c8410f330 0x7f8c8410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 --2- 172.21.15.44:0/3189199756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c841099f0 0x7f8c84109e00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 -- 172.21.15.44:0/3189199756 >> 172.21.15.44:0/3189199756 conn(0x7f8c84078ff0 msgr2=0x7f8c84107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.305+0000 7f8c89200640 1 -- 172.21.15.44:0/3189199756 shutdown_connections 2024-07-26T08:36:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.306+0000 7f8c89200640 1 -- 172.21.15.44:0/3189199756 wait complete. 2024-07-26T08:36:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.307+0000 7f8c89200640 1 Processor -- start 2024-07-26T08:36:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.307+0000 7f8c89200640 1 -- start start 2024-07-26T08:36:00.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c89200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c841099f0 0x7f8c841a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c89200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c8410f330 0x7f8c841a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c89200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8410fc80 0x7f8c841a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c89200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84119990 con 0x7f8c8410fc80 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c89200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84119810 con 0x7f8c841099f0 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c89200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84119b10 con 0x7f8c8410f330 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c83400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c841099f0 0x7f8c841a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c83400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c841099f0 0x7f8c841a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:48032/0 (socket says 172.21.15.44:48032) 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c83400640 1 -- 172.21.15.44:0/1809515948 learned_addr learned my addr 172.21.15.44:0/1809515948 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c83e00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8410fc80 0x7f8c841a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:00.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.308+0000 7f8c7be00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c8410f330 0x7f8c841a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.309+0000 7f8c83400640 1 -- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c8410f330 msgr2=0x7f8c841a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:00.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.309+0000 7f8c83400640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c8410f330 0x7f8c841a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.309+0000 7f8c83400640 1 -- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8410fc80 msgr2=0x7f8c841a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.309+0000 7f8c83400640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8c8410fc80 0x7f8c841a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.309+0000 7f8c83400640 1 -- 172.21.15.44:0/1809515948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c841a6270 con 0x7f8c841099f0 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.309+0000 7f8c83400640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c841099f0 0x7f8c841a0300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c74005ab0 tx=0x7f8c74003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.310+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7403f070 con 0x7f8c841099f0 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.310+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c841a6500 con 0x7f8c841099f0 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.310+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c74031b60 con 0x7f8c841099f0 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.310+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74035030 con 0x7f8c841099f0 2024-07-26T08:36:00.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.310+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c841accc0 con 0x7f8c841099f0 2024-07-26T08:36:00.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.311+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8c84063580 con 0x7f8c841099f0 2024-07-26T08:36:00.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.314+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f8c74042070 con 0x7f8c841099f0 2024-07-26T08:36:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.316+0000 7f8c80c00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8c640b3cb0 0x7f8c640b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.316+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8c74105060 con 0x7f8c841099f0 2024-07-26T08:36:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.316+0000 7f8c7be00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8c640b3cb0 0x7f8c640b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.316+0000 7f8c80c00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] conn(0x7f8c640bf350 0x7f8c640c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:00.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.316+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 --> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c640c1f10 con 0x7f8c640bf350 2024-07-26T08:36:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.316+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8c74105410 con 0x7f8c841099f0 2024-07-26T08:36:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.317+0000 7f8c83e00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] conn(0x7f8c640bf350 0x7f8c640c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.317+0000 7f8c7be00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8c640b3cb0 0x7f8c640b61c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c8410fa50 tx=0x7f8c6c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:00.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.317+0000 7f8c83e00640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] conn(0x7f8c640bf350 0x7f8c640c1840 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:00.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.318+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== osd.11 v2:172.21.15.171:6824/4225876845 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c640c1f10 con 0x7f8c640bf350 2024-07-26T08:36:00.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.334+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 --> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8c84063980 con 0x7f8c640bf350 2024-07-26T08:36:00.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.334+0000 7f8c80c00640 1 -- 172.21.15.44:0/1809515948 <== osd.11 v2:172.21.15.171:6824/4225876845 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8c84063980 con 0x7f8c640bf350 2024-07-26T08:36:00.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] conn(0x7f8c640bf350 msgr2=0x7f8c640c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:00.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] conn(0x7f8c640bf350 0x7f8c640c1840 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:35:59 smithi171 ceph-mon[37789]: pgmap v402: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:00.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8c640b3cb0 msgr2=0x7f8c640b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:00.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8c640b3cb0 0x7f8c640b61c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c8410fa50 tx=0x7f8c6c005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:00.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c841099f0 msgr2=0x7f8c841a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:00.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 --2- 172.21.15.44:0/1809515948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c841099f0 0x7f8c841a0300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c74005ab0 tx=0x7f8c74003c10 comp rx=0 tx=0).stop 2024-07-26T08:36:00.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c83e00640 1 -- 172.21.15.44:0/1809515948 reap_dead start 2024-07-26T08:36:00.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 shutdown_connections 2024-07-26T08:36:00.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.335+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 >> 172.21.15.44:0/1809515948 conn(0x7f8c84078ff0 msgr2=0x7f8c840796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:00.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.336+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 shutdown_connections 2024-07-26T08:36:00.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:00.336+0000 7f8c89200640 1 -- 172.21.15.44:0/1809515948 wait complete. 2024-07-26T08:36:01.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 -- 172.21.15.44:0/1445960064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b541035b0 msgr2=0x7f9b541039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:01.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 --2- 172.21.15.44:0/1445960064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b541035b0 0x7f9b541039c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009900 tx=0x7f9b4802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 -- 172.21.15.44:0/1445960064 shutdown_connections 2024-07-26T08:36:01.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 --2- 172.21.15.44:0/1445960064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b54109840 0x7f9b54110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 --2- 172.21.15.44:0/1445960064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b54108ef0 0x7f9b54109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 --2- 172.21.15.44:0/1445960064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b541035b0 0x7f9b541039c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 -- 172.21.15.44:0/1445960064 >> 172.21.15.44:0/1445960064 conn(0x7f9b540ff000 msgr2=0x7f9b54101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:01.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 -- 172.21.15.44:0/1445960064 shutdown_connections 2024-07-26T08:36:01.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.607+0000 7f9b5b600640 1 -- 172.21.15.44:0/1445960064 wait complete. 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 Processor -- start 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 -- start start 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b541035b0 0x7f9b54199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b54108ef0 0x7f9b5419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54109840 0x7f9b5419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9b54113560 con 0x7f9b54108ef0 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b541133e0 con 0x7f9b54109840 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.609+0000 7f9b5b600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9b541136e0 con 0x7f9b541035b0 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b541035b0 0x7f9b54199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:01.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b541035b0 0x7f9b54199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:44628/0 (socket says 172.21.15.44:44628) 2024-07-26T08:36:01.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 -- 172.21.15.44:0/2757939260 learned_addr learned my addr 172.21.15.44:0/2757939260 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:01.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b53e00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b54108ef0 0x7f9b5419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:01.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b59600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54109840 0x7f9b5419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 -- 172.21.15.44:0/2757939260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54109840 msgr2=0x7f9b5419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54109840 0x7f9b5419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 -- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b54108ef0 msgr2=0x7f9b5419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b54108ef0 0x7f9b5419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b58c00640 1 -- 172.21.15.44:0/2757939260 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5419fe60 con 0x7f9b541035b0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.610+0000 7f9b53e00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b54108ef0 0x7f9b5419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.611+0000 7f9b58c00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b541035b0 0x7f9b54199ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b48005ab0 tx=0x7f9b48003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.611+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4803f070 con 0x7f9b541035b0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.611+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b541a00f0 con 0x7f9b541035b0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.611+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b48031b60 con 0x7f9b541035b0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.611+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48035030 con 0x7f9b541035b0 2024-07-26T08:36:01.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.611+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b541a68b0 con 0x7f9b541035b0 2024-07-26T08:36:01.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.612+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b54063540 con 0x7f9b541035b0 2024-07-26T08:36:01.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.615+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f9b48042070 con 0x7f9b541035b0 2024-07-26T08:36:01.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.616+0000 7f9b51600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b3c0b3d30 0x7f9b3c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:01.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.616+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9b48104e70 con 0x7f9b541035b0 2024-07-26T08:36:01.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.616+0000 7f9b53e00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b3c0b3d30 0x7f9b3c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:01.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.617+0000 7f9b51600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] conn(0x7f9b3c0bf3d0 0x7f9b3c0c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:01.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.617+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 --> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b48009080 con 0x7f9b3c0bf3d0 2024-07-26T08:36:01.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.617+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9b48105220 con 0x7f9b541035b0 2024-07-26T08:36:01.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.617+0000 7f9b59600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] conn(0x7f9b3c0bf3d0 0x7f9b3c0c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:01.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.617+0000 7f9b53e00640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b3c0b3d30 0x7f9b3c0b6240 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b54108a80 tx=0x7f9b40005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:01.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.618+0000 7f9b59600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] conn(0x7f9b3c0bf3d0 0x7f9b3c0c18c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:01.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.619+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== osd.9 v2:172.21.15.44:6826/2869135265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b48009080 con 0x7f9b3c0bf3d0 2024-07-26T08:36:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.635+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 --> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b54063970 con 0x7f9b3c0bf3d0 2024-07-26T08:36:01.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.635+0000 7f9b51600640 1 -- 172.21.15.44:0/2757939260 <== osd.9 v2:172.21.15.44:6826/2869135265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b54063970 con 0x7f9b3c0bf3d0 2024-07-26T08:36:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] conn(0x7f9b3c0bf3d0 msgr2=0x7f9b3c0c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:01.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] conn(0x7f9b3c0bf3d0 0x7f9b3c0c18c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b3c0b3d30 msgr2=0x7f9b3c0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b3c0b3d30 0x7f9b3c0b6240 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b54108a80 tx=0x7f9b40005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b541035b0 msgr2=0x7f9b54199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 --2- 172.21.15.44:0/2757939260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b541035b0 0x7f9b54199ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b48005ab0 tx=0x7f9b48003c10 comp rx=0 tx=0).stop 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b59600640 1 -- 172.21.15.44:0/2757939260 reap_dead start 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 shutdown_connections 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 >> 172.21.15.44:0/2757939260 conn(0x7f9b540ff000 msgr2=0x7f9b54101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 shutdown_connections 2024-07-26T08:36:01.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:01.636+0000 7f9b5b600640 1 -- 172.21.15.44:0/2757939260 wait complete. 2024-07-26T08:36:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:01 smithi044 ceph-mon[32404]: pgmap v403: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:01 smithi116 ceph-mon[37097]: pgmap v403: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 -- 172.21.15.44:0/2240210232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc108ee0 msgr2=0x7f65dc1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 --2- 172.21.15.44:0/2240210232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc108ee0 0x7f65dc1092f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009860 tx=0x7f65c402a340 comp rx=0 tx=0).stop 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 -- 172.21.15.44:0/2240210232 shutdown_connections 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 --2- 172.21.15.44:0/2240210232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65dc109830 0x7f65dc110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 --2- 172.21.15.44:0/2240210232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc108ee0 0x7f65dc1092f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 --2- 172.21.15.44:0/2240210232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc1035a0 0x7f65dc1039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 -- 172.21.15.44:0/2240210232 >> 172.21.15.44:0/2240210232 conn(0x7f65dc0ff010 msgr2=0x7f65dc101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 -- 172.21.15.44:0/2240210232 shutdown_connections 2024-07-26T08:36:02.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.300+0000 7f65e2c00640 1 -- 172.21.15.44:0/2240210232 wait complete. 2024-07-26T08:36:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.302+0000 7f65e2c00640 1 Processor -- start 2024-07-26T08:36:02.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.302+0000 7f65e2c00640 1 -- start start 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.302+0000 7f65e2c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65dc1035a0 0x7f65dc10dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.302+0000 7f65e2c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc108ee0 0x7f65dc10a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.302+0000 7f65e2c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc109830 0x7f65dc10a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65e2c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc1134c0 con 0x7f65dc1035a0 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65e2c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc113340 con 0x7f65dc109830 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65e2c00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc113640 con 0x7f65dc108ee0 2024-07-26T08:36:02.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65d3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc108ee0 0x7f65dc10a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:02.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65d3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc108ee0 0x7f65dc10a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:44634/0 (socket says 172.21.15.44:44634) 2024-07-26T08:36:02.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65d3e00640 1 -- 172.21.15.44:0/3915531839 learned_addr learned my addr 172.21.15.44:0/3915531839 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:02.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65dbe00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65dc1035a0 0x7f65dc10dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.303+0000 7f65e0c00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc109830 0x7f65dc10a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65d3e00640 1 -- 172.21.15.44:0/3915531839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc109830 msgr2=0x7f65dc10a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65d3e00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc109830 0x7f65dc10a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65d3e00640 1 -- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65dc1035a0 msgr2=0x7f65dc10dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65d3e00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65dc1035a0 0x7f65dc10dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65d3e00640 1 -- 172.21.15.44:0/3915531839 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65dc1ac6c0 con 0x7f65dc108ee0 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65dbe00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65dc1035a0 0x7f65dc10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:02.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.304+0000 7f65d3e00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc108ee0 0x7f65dc10a030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f65c4035040 tx=0x7f65c4032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.305+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4040020 con 0x7f65dc108ee0 2024-07-26T08:36:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.305+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65c4030e10 con 0x7f65dc108ee0 2024-07-26T08:36:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.305+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4037040 con 0x7f65dc108ee0 2024-07-26T08:36:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.306+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65dc1ac890 con 0x7f65dc108ee0 2024-07-26T08:36:02.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.306+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65dc1acd50 con 0x7f65dc108ee0 2024-07-26T08:36:02.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.307+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65dc10f640 con 0x7f65dc108ee0 2024-07-26T08:36:02.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.310+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f65c4043020 con 0x7f65dc108ee0 2024-07-26T08:36:02.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.311+0000 7f65d9600640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f65c00b3d30 0x7f65c00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:02.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.312+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f65c4104500 con 0x7f65dc108ee0 2024-07-26T08:36:02.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.312+0000 7f65dbe00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f65c00b3d30 0x7f65c00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:02.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.312+0000 7f65d9600640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] conn(0x7f65c00bf3d0 0x7f65c00c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:02.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.312+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 --> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65c4040e80 con 0x7f65c00bf3d0 2024-07-26T08:36:02.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.312+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f65c4104950 con 0x7f65dc108ee0 2024-07-26T08:36:02.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.313+0000 7f65e0c00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] conn(0x7f65c00bf3d0 0x7f65c00c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:02.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.314+0000 7f65e0c00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] conn(0x7f65c00bf3d0 0x7f65c00c18c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:02.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.315+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== osd.3 v2:172.21.15.44:6810/2033336910 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65c4040e80 con 0x7f65c00bf3d0 2024-07-26T08:36:02.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.315+0000 7f65dbe00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f65c00b3d30 0x7f65c00b6240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f65dc10daa0 tx=0x7f65cc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:02.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.331+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 --> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f65dc104820 con 0x7f65c00bf3d0 2024-07-26T08:36:02.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.332+0000 7f65d9600640 1 -- 172.21.15.44:0/3915531839 <== osd.3 v2:172.21.15.44:6810/2033336910 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f65dc104820 con 0x7f65c00bf3d0 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.332+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] conn(0x7f65c00bf3d0 msgr2=0x7f65c00c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.332+0000 7f65e2c00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] conn(0x7f65c00bf3d0 0x7f65c00c18c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.332+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f65c00b3d30 msgr2=0x7f65c00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.332+0000 7f65e2c00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f65c00b3d30 0x7f65c00b6240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f65dc10daa0 tx=0x7f65cc005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc108ee0 msgr2=0x7f65dc10a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e2c00640 1 --2- 172.21.15.44:0/3915531839 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65dc108ee0 0x7f65dc10a030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f65c4035040 tx=0x7f65c4032d50 comp rx=0 tx=0).stop 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e0c00640 1 -- 172.21.15.44:0/3915531839 reap_dead start 2024-07-26T08:36:02.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 shutdown_connections 2024-07-26T08:36:02.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 >> 172.21.15.44:0/3915531839 conn(0x7f65dc0ff010 msgr2=0x7f65dc106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:02.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 shutdown_connections 2024-07-26T08:36:02.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:02.333+0000 7f65e2c00640 1 -- 172.21.15.44:0/3915531839 wait complete. 2024-07-26T08:36:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:01 smithi171 ceph-mon[37789]: pgmap v403: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:02.912 INFO:teuthology.orchestra.run.smithi044.stdout:309237645365 2024-07-26T08:36:02.912 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-26T08:36:03.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:03.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:03.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:36:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:36:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:36:03.580 INFO:teuthology.orchestra.run.smithi044.stdout:261993005119 2024-07-26T08:36:03.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-26T08:36:03.905 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:03.906 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 -- 172.21.15.44:0/466872407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010ba40 msgr2=0x7fc3a0112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 --2- 172.21.15.44:0/466872407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010ba40 0x7fc3a0112270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc398009be0 tx=0x7fc39802a6c0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 -- 172.21.15.44:0/466872407 shutdown_connections 2024-07-26T08:36:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 --2- 172.21.15.44:0/466872407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010ba40 0x7fc3a0112270 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 --2- 172.21.15.44:0/466872407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3a010b0f0 0x7fc3a010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 --2- 172.21.15.44:0/466872407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3a01057b0 0x7fc3a0105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.889+0000 7fc3a8600640 1 -- 172.21.15.44:0/466872407 >> 172.21.15.44:0/466872407 conn(0x7fc3a0101240 msgr2=0x7fc3a0103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.890+0000 7fc3a8600640 1 -- 172.21.15.44:0/466872407 shutdown_connections 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.890+0000 7fc3a8600640 1 -- 172.21.15.44:0/466872407 wait complete. 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.895+0000 7fc3a8600640 1 Processor -- start 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.895+0000 7fc3a8600640 1 -- start start 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a8600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3a01057b0 0x7fc3a010f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a8600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010b0f0 0x7fc3a010fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a8600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3a010ba40 0x7fc3a0110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a8600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0115710 con 0x7fc3a01057b0 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a8600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0115590 con 0x7fc3a010b0f0 2024-07-26T08:36:04.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a8600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0115890 con 0x7fc3a010ba40 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.896+0000 7fc3a6600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3a010ba40 0x7fc3a0110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a6600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3a010ba40 0x7fc3a0110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:44664/0 (socket says 172.21.15.44:44664) 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a6600640 1 -- 172.21.15.44:0/1534930673 learned_addr learned my addr 172.21.15.44:0/1534930673 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5c00640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3a01057b0 0x7fc3a010f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5200640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010b0f0 0x7fc3a010fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5c00640 1 -- 172.21.15.44:0/1534930673 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3a010ba40 msgr2=0x7fc3a0110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5c00640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3a010ba40 0x7fc3a0110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5c00640 1 -- 172.21.15.44:0/1534930673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010b0f0 msgr2=0x7fc3a010fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5c00640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a010b0f0 0x7fc3a010fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.897+0000 7fc3a5c00640 1 -- 172.21.15.44:0/1534930673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a019ad10 con 0x7fc3a01057b0 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.898+0000 7fc3a5c00640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3a01057b0 0x7fc3a010f6f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc39400e9f0 tx=0x7fc39400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:04.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.898+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39400ccc0 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.898+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc394005020 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.898+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394004a00 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.898+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a019b000 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.898+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a019b4f0 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.900+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3a0063580 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.905+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fc3940051d0 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.907+0000 7fc39e000640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc3880b3d30 0x7fc3880b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.907+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fc3940e4570 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.908+0000 7fc3a5200640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc3880b3d30 0x7fc3880b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.908+0000 7fc39e000640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] conn(0x7fc3880bf3d0 0x7fc3880c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.908+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 --> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc394023ea0 con 0x7fc3880bf3d0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.908+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc3940e49c0 con 0x7fc3a01057b0 2024-07-26T08:36:04.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.909+0000 7fc3a5200640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc3880b3d30 0x7fc3880b6240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0104940 tx=0x7fc38c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:04.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.910+0000 7fc3a6600640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] conn(0x7fc3880bf3d0 0x7fc3880c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:04.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.910+0000 7fc3a6600640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] conn(0x7fc3880bf3d0 0x7fc3880c18c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:04.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.912+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== osd.6 v2:172.21.15.44:6818/51483018 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc394023ea0 con 0x7fc3880bf3d0 2024-07-26T08:36:04.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.943+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 --> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc3a0063a00 con 0x7fc3880bf3d0 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.944+0000 7fc39e000640 1 -- 172.21.15.44:0/1534930673 <== osd.6 v2:172.21.15.44:6818/51483018 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc3a0063a00 con 0x7fc3880bf3d0 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.944+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] conn(0x7fc3880bf3d0 msgr2=0x7fc3880c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.944+0000 7fc3a8600640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] conn(0x7fc3880bf3d0 0x7fc3880c18c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc3880b3d30 msgr2=0x7fc3880b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc3880b3d30 0x7fc3880b6240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0104940 tx=0x7fc38c009260 comp rx=0 tx=0).stop 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3a01057b0 msgr2=0x7fc3a010f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 --2- 172.21.15.44:0/1534930673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc3a01057b0 0x7fc3a010f6f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc39400e9f0 tx=0x7fc39400edf0 comp rx=0 tx=0).stop 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a6600640 1 -- 172.21.15.44:0/1534930673 reap_dead start 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 shutdown_connections 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 >> 172.21.15.44:0/1534930673 conn(0x7fc3a0101240 msgr2=0x7fc3a010a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:04.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.945+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 shutdown_connections 2024-07-26T08:36:04.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:03.946+0000 7fc3a8600640 1 -- 172.21.15.44:0/1534930673 wait complete. 2024-07-26T08:36:04.146 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:03 smithi044 ceph-mon[32404]: pgmap v404: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:03 smithi116 ceph-mon[37097]: pgmap v404: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:03 smithi171 ceph-mon[37789]: pgmap v404: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:04.900 INFO:teuthology.orchestra.run.smithi044.stdout:115964117088 2024-07-26T08:36:04.901 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-26T08:36:05.141 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:06.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:05 smithi044 ceph-mon[32404]: pgmap v405: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T08:36:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:05 smithi116 ceph-mon[37097]: pgmap v405: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T08:36:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:05 smithi171 ceph-mon[37789]: pgmap v405: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T08:36:07.619 INFO:teuthology.orchestra.run.smithi044.stdout:188978561103 2024-07-26T08:36:07.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-26T08:36:07.701 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:07.703 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:07.705 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:08.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:07 smithi044 ceph-mon[32404]: pgmap v406: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:07 smithi116 ceph-mon[37097]: pgmap v406: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:07 smithi171 ceph-mon[37789]: pgmap v406: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:10.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 -- 172.21.15.44:0/1947971321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0114250 msgr2=0x7f1ec0116670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 --2- 172.21.15.44:0/1947971321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0114250 0x7f1ec0116670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009ca0 tx=0x7f1eb802a8c0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 -- 172.21.15.44:0/1947971321 shutdown_connections 2024-07-26T08:36:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 --2- 172.21.15.44:0/1947971321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0114250 0x7f1ec0116670 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 --2- 172.21.15.44:0/1947971321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ec0077cc0 0x7f1ec0113d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 --2- 172.21.15.44:0/1947971321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 0x7f1ec0077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 -- 172.21.15.44:0/1947971321 >> 172.21.15.44:0/1947971321 conn(0x7f1ec0101340 msgr2=0x7f1ec0103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 -- 172.21.15.44:0/1947971321 shutdown_connections 2024-07-26T08:36:10.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.040+0000 7f1ec7400640 1 -- 172.21.15.44:0/1947971321 wait complete. 2024-07-26T08:36:10.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 Processor -- start 2024-07-26T08:36:10.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 -- start start 2024-07-26T08:36:10.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 0x7f1ec01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:10.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0077cc0 0x7f1ec01a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:10.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ec0114250 0x7f1ec01a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:10.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0119c70 con 0x7f1ec0114250 2024-07-26T08:36:10.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0119af0 con 0x7f1ec0077370 2024-07-26T08:36:10.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.042+0000 7f1ec7400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec0119df0 con 0x7f1ec0077cc0 2024-07-26T08:36:10.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ec5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ec0114250 0x7f1ec01a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:10.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ec4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 0x7f1ec01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:10.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0077cc0 0x7f1ec01a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:10.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ec4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 0x7f1ec01a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47772/0 (socket says 172.21.15.44:47772) 2024-07-26T08:36:10.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ec4a00640 1 -- 172.21.15.44:0/208859025 learned_addr learned my addr 172.21.15.44:0/208859025 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:10.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 -- 172.21.15.44:0/208859025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 msgr2=0x7f1ec01a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:10.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 0x7f1ec01a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 -- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ec0114250 msgr2=0x7f1ec01a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:10.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ec0114250 0x7f1ec01a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 -- 172.21.15.44:0/208859025 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec01a6510 con 0x7f1ec0077cc0 2024-07-26T08:36:10.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.043+0000 7f1ebfe00640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0077cc0 0x7f1ec01a0b40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1ec006a4c0 tx=0x7f1eac00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:10.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.044+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00cc40 con 0x7f1ec0077cc0 2024-07-26T08:36:10.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.044+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1eac00cde0 con 0x7f1ec0077cc0 2024-07-26T08:36:10.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.044+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec01a6800 con 0x7f1ec0077cc0 2024-07-26T08:36:10.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.044+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac018780 con 0x7f1ec0077cc0 2024-07-26T08:36:10.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.044+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec01acf60 con 0x7f1ec0077cc0 2024-07-26T08:36:10.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.045+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec0077780 con 0x7f1ec0077cc0 2024-07-26T08:36:10.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.048+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f1eac010a90 con 0x7f1ec0077cc0 2024-07-26T08:36:10.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.050+0000 7f1ebd600640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1ea80b3d30 0x7f1ea80b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:10.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.050+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f1eac0e4680 con 0x7f1ec0077cc0 2024-07-26T08:36:10.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.050+0000 7f1ec4a00640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1ea80b3d30 0x7f1ea80b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:10.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.051+0000 7f1ec4a00640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1ea80b3d30 0x7f1ea80b6240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40095f0 tx=0x7f1eb4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:10.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.051+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1eac0ab010 con 0x7f1ec0077cc0 2024-07-26T08:36:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:09 smithi044 ceph-mon[32404]: pgmap v407: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T08:36:10.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.267+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f1ec010f3e0 con 0x7f1ec0077cc0 2024-07-26T08:36:10.271 INFO:teuthology.orchestra.run.smithi044.stdout:236223201351 2024-07-26T08:36:10.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.268+0000 7f1ebd600640 1 -- 172.21.15.44:0/208859025 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1eac0af560 con 0x7f1ec0077cc0 2024-07-26T08:36:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:09 smithi116 ceph-mon[37097]: pgmap v407: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1ea80b3d30 msgr2=0x7f1ea80b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1ea80b3d30 0x7f1ea80b6240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40095f0 tx=0x7f1eb4005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0077cc0 msgr2=0x7f1ec01a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0077cc0 0x7f1ec01a0b40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1ec006a4c0 tx=0x7f1eac00ee50 comp rx=0 tx=0).stop 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 shutdown_connections 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1ea80b3d30 0x7f1ea80b6240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ec0077cc0 0x7f1ec01a0b40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ec0114250 0x7f1ec01a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.272+0000 7f1ec7400640 1 --2- 172.21.15.44:0/208859025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ec0077370 0x7f1ec01a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:10.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.273+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 >> 172.21.15.44:0/208859025 conn(0x7f1ec0101340 msgr2=0x7f1ec0116270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:10.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.273+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 shutdown_connections 2024-07-26T08:36:10.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:10.273+0000 7f1ec7400640 1 -- 172.21.15.44:0/208859025 wait complete. 2024-07-26T08:36:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:09 smithi171 ceph-mon[37789]: pgmap v407: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T08:36:10.730 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:10 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/208859025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T08:36:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:10 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/208859025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T08:36:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:10 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/208859025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T08:36:11.383 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:11.913 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:11.959 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201347 got 236223201351 for osd.8 2024-07-26T08:36:11.960 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:11 smithi044 ceph-mon[32404]: pgmap v408: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:11 smithi116 ceph-mon[37097]: pgmap v408: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:11 smithi171 ceph-mon[37789]: pgmap v408: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:13 smithi044 ceph-mon[32404]: pgmap v409: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:13 smithi116 ceph-mon[37097]: pgmap v409: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:13 smithi171 ceph-mon[37789]: pgmap v409: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:15.244 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:15 smithi044 ceph-mon[32404]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T08:36:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:15 smithi116 ceph-mon[37097]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T08:36:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:15 smithi171 ceph-mon[37789]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T08:36:16.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:16 smithi044 ceph-mon[32404]: pgmap v410: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T08:36:16.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:16 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2033171601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T08:36:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:16 smithi116 ceph-mon[37097]: pgmap v410: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T08:36:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:16 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2033171601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T08:36:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:16 smithi171 ceph-mon[37789]: pgmap v410: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T08:36:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:16 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2033171601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T08:36:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.366+0000 7fca4d200640 1 -- 172.21.15.44:0/1982088826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca4810fc50 msgr2=0x7fca48112070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.366+0000 7fca4d200640 1 --2- 172.21.15.44:0/1982088826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca4810fc50 0x7fca48112070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fca40009be0 tx=0x7fca4002a750 comp rx=0 tx=0).stop 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 -- 172.21.15.44:0/1982088826 shutdown_connections 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 --2- 172.21.15.44:0/1982088826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca481029f0 0x7fca4810f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 --2- 172.21.15.44:0/1982088826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca4810fc50 0x7fca48112070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 --2- 172.21.15.44:0/1982088826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca481020a0 0x7fca481024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 -- 172.21.15.44:0/1982088826 >> 172.21.15.44:0/1982088826 conn(0x7fca48079170 msgr2=0x7fca48079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 -- 172.21.15.44:0/1982088826 shutdown_connections 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.367+0000 7fca4d200640 1 -- 172.21.15.44:0/1982088826 wait complete. 2024-07-26T08:36:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.368+0000 7fca4d200640 1 Processor -- start 2024-07-26T08:36:16.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 -- start start 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 0x7fca4819c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca481029f0 0x7fca4819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4810fc50 0x7fca481a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca48115690 con 0x7fca481020a0 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca48115510 con 0x7fca4810fc50 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca4d200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fca48115810 con 0x7fca481029f0 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.369+0000 7fca45e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 0x7fca4819c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 0x7fca4819c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46742/0 (socket says 172.21.15.44:46742) 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 -- 172.21.15.44:0/2600341877 learned_addr learned my addr 172.21.15.44:0/2600341877 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca3d400640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca481029f0 0x7fca4819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca46800640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4810fc50 0x7fca481a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 -- 172.21.15.44:0/2600341877 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca481029f0 msgr2=0x7fca4819c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca481029f0 0x7fca4819c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 -- 172.21.15.44:0/2600341877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4810fc50 msgr2=0x7fca481a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4810fc50 0x7fca481a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.370+0000 7fca45e00640 1 -- 172.21.15.44:0/2600341877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca481a1fb0 con 0x7fca481020a0 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.371+0000 7fca45e00640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 0x7fca4819c040 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fca3800e9c0 tx=0x7fca3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.371+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3800cde0 con 0x7fca481020a0 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.371+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca380035f0 con 0x7fca481020a0 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.371+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca481a22a0 con 0x7fca481020a0 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.371+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38010690 con 0x7fca481020a0 2024-07-26T08:36:16.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.371+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca481a8a00 con 0x7fca481020a0 2024-07-26T08:36:16.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.372+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca4806bbf0 con 0x7fca481020a0 2024-07-26T08:36:16.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.375+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fca38004000 con 0x7fca481020a0 2024-07-26T08:36:16.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.376+0000 7fca3f400640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fca2c0b3ca0 0x7fca2c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.377+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fca380e43c0 con 0x7fca481020a0 2024-07-26T08:36:16.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.377+0000 7fca3d400640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fca2c0b3ca0 0x7fca2c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.378+0000 7fca3d400640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fca2c0b3ca0 0x7fca2c0b61b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fca48111b80 tx=0x7fca30005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:16.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.381+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fca38009510 con 0x7fca481020a0 2024-07-26T08:36:16.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 -- 172.21.15.44:0/354702068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 msgr2=0x7ff898109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 --2- 172.21.15.44:0/354702068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898109110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff880009900 tx=0x7ff88002a410 comp rx=0 tx=0).stop 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 -- 172.21.15.44:0/354702068 shutdown_connections 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 --2- 172.21.15.44:0/354702068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff898109650 0x7ff89810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 --2- 172.21.15.44:0/354702068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898109110 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 --2- 172.21.15.44:0/354702068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 0x7ff8981037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 -- 172.21.15.44:0/354702068 >> 172.21.15.44:0/354702068 conn(0x7ff8980fee10 msgr2=0x7ff898101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 -- 172.21.15.44:0/354702068 shutdown_connections 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.617+0000 7ff89de00640 1 -- 172.21.15.44:0/354702068 wait complete. 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.618+0000 7ff89de00640 1 Processor -- start 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.618+0000 7ff89de00640 1 -- start start 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89de00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 0x7ff898075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff898109650 0x7ff898075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89de00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff898113330 con 0x7ff898108d00 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8981131b0 con 0x7ff898109650 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89de00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff8981134b0 con 0x7ff8981033c0 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff88fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff88fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46728/0 (socket says 172.21.15.44:46728) 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff88fe00640 1 -- 172.21.15.44:0/2033171601 learned_addr learned my addr 172.21.15.44:0/2033171601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89ca00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 0x7ff898075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.619+0000 7ff89d400640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff898109650 0x7ff898075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff88fe00640 1 -- 172.21.15.44:0/2033171601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 msgr2=0x7ff898075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff88fe00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 0x7ff898075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff88fe00640 1 -- 172.21.15.44:0/2033171601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff898109650 msgr2=0x7ff898075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff88fe00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff898109650 0x7ff898075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff88fe00640 1 -- 172.21.15.44:0/2033171601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8981a7fe0 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff89ca00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 0x7ff898075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.620+0000 7ff88fe00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898075680 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff880005ab0 tx=0x7ff880003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.621+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88003f070 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.621+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff880031b50 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.621+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880035030 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.621+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8981a8270 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.621+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8981a8700 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.622+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8981037d0 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.624+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7ff880042070 con 0x7ff898108d00 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.628+0000 7ff896000640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff87c0b3c20 0x7ff87c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:16.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.628+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff880104a20 con 0x7ff898108d00 2024-07-26T08:36:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.628+0000 7ff89ca00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff87c0b3c20 0x7ff87c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.628+0000 7ff89ca00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff87c0b3c20 0x7ff87c0b6130 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff898076b90 tx=0x7ff888005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.630+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff8800cb570 con 0x7ff898108d00 2024-07-26T08:36:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.836+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff89802e0c0 con 0x7ff898108d00 2024-07-26T08:36:16.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.837+0000 7ff896000640 1 -- 172.21.15.44:0/2033171601 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff8800cfac0 con 0x7ff898108d00 2024-07-26T08:36:16.477 INFO:teuthology.orchestra.run.smithi044.stdout:163208757336 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff87c0b3c20 msgr2=0x7ff87c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff87c0b3c20 0x7ff87c0b6130 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff898076b90 tx=0x7ff888005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 msgr2=0x7ff898075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898075680 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff880005ab0 tx=0x7ff880003b90 comp rx=0 tx=0).stop 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 shutdown_connections 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff87c0b3c20 0x7ff87c0b6130 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff898109650 0x7ff898075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff898108d00 0x7ff898075680 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 --2- 172.21.15.44:0/2033171601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8981033c0 0x7ff898075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 >> 172.21.15.44:0/2033171601 conn(0x7ff8980fee10 msgr2=0x7ff898101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 shutdown_connections 2024-07-26T08:36:16.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:15.841+0000 7ff89de00640 1 -- 172.21.15.44:0/2033171601 wait complete. 2024-07-26T08:36:16.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.590+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fca481a8da0 con 0x7fca481020a0 2024-07-26T08:36:16.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.590+0000 7fca3f400640 1 -- 172.21.15.44:0/2600341877 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fca380ab500 con 0x7fca481020a0 2024-07-26T08:36:16.592 INFO:teuthology.orchestra.run.smithi044.stdout:38654705778 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fca2c0b3ca0 msgr2=0x7fca2c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fca2c0b3ca0 0x7fca2c0b61b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fca48111b80 tx=0x7fca30005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 msgr2=0x7fca4819c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 0x7fca4819c040 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fca3800e9c0 tx=0x7fca3800ee80 comp rx=0 tx=0).stop 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 shutdown_connections 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fca2c0b3ca0 0x7fca2c0b61b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fca481029f0 0x7fca4819c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4810fc50 0x7fca481a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 --2- 172.21.15.44:0/2600341877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca481020a0 0x7fca4819c040 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 >> 172.21.15.44:0/2600341877 conn(0x7fca48079170 msgr2=0x7fca481086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:16.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.593+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 shutdown_connections 2024-07-26T08:36:16.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:16.594+0000 7fca4d200640 1 -- 172.21.15.44:0/2600341877 wait complete. 2024-07-26T08:36:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:17 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2600341877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T08:36:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:17 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2600341877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T08:36:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:17 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2600341877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T08:36:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 -- 172.21.15.44:0/885552501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45281055f0 msgr2=0x7f4528105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 --2- 172.21.15.44:0/885552501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45281055f0 0x7f4528105a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4518009900 tx=0x7f451802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 -- 172.21.15.44:0/885552501 shutdown_connections 2024-07-26T08:36:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 --2- 172.21.15.44:0/885552501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f452810af30 0x7f452810b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 --2- 172.21.15.44:0/885552501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452810b880 0x7f45281120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 --2- 172.21.15.44:0/885552501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45281055f0 0x7f4528105a00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.648+0000 7f452e000640 1 -- 172.21.15.44:0/885552501 >> 172.21.15.44:0/885552501 conn(0x7f4528101060 msgr2=0x7f45281034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:17.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.649+0000 7f452e000640 1 -- 172.21.15.44:0/885552501 shutdown_connections 2024-07-26T08:36:17.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.649+0000 7f452e000640 1 -- 172.21.15.44:0/885552501 wait complete. 2024-07-26T08:36:17.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.650+0000 7f452e000640 1 Processor -- start 2024-07-26T08:36:17.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.650+0000 7f452e000640 1 -- start start 2024-07-26T08:36:17.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 0x7f452819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:17.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452e000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f452810af30 0x7f452819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:17.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452e000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f452810b880 0x7f45281a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:17.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452e000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4528115530 con 0x7f452810af30 2024-07-26T08:36:17.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45281153b0 con 0x7f45281055f0 2024-07-26T08:36:17.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452e000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45281156b0 con 0x7f452810b880 2024-07-26T08:36:17.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 0x7f452819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 0x7f452819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:49342/0 (socket says 172.21.15.44:49342) 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452cc00640 1 -- 172.21.15.44:0/1131861198 learned_addr learned my addr 172.21.15.44:0/1131861198 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f451fe00640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f452810af30 0x7f452819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.651+0000 7f452d600640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f452810b880 0x7f45281a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452cc00640 1 -- 172.21.15.44:0/1131861198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f452810b880 msgr2=0x7f45281a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452cc00640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f452810b880 0x7f45281a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452cc00640 1 -- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f452810af30 msgr2=0x7f452819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452cc00640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f452810af30 0x7f452819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452cc00640 1 -- 172.21.15.44:0/1131861198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45281a1eb0 con 0x7f45281055f0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452cc00640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 0x7f452819bf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4518005940 tx=0x7f45180040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451803f030 con 0x7f45281055f0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45281a2140 con 0x7f45281055f0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45180044c0 con 0x7f45281055f0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.652+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518035070 con 0x7f45281055f0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.653+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45281a8900 con 0x7f45281055f0 2024-07-26T08:36:17.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.654+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4528105a00 con 0x7f45281055f0 2024-07-26T08:36:17.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.656+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f4518042030 con 0x7f45281055f0 2024-07-26T08:36:17.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.658+0000 7f4526000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f450c0b3c20 0x7f450c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:17.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.658+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4518104720 con 0x7f45281055f0 2024-07-26T08:36:17.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.658+0000 7f451fe00640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f450c0b3c20 0x7f450c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:17.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.659+0000 7f451fe00640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f450c0b3c20 0x7f450c0b6130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f45100095f0 tx=0x7f4510005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:17.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.659+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f45180cb210 con 0x7f45281055f0 2024-07-26T08:36:17.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.871+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f452810d070 con 0x7f45281055f0 2024-07-26T08:36:17.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.872+0000 7f4526000640 1 -- 172.21.15.44:0/1131861198 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f45180cf760 con 0x7f45281055f0 2024-07-26T08:36:17.874 INFO:teuthology.orchestra.run.smithi044.stdout:283467841596 2024-07-26T08:36:17.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.874+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f450c0b3c20 msgr2=0x7f450c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:17.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f450c0b3c20 0x7f450c0b6130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f45100095f0 tx=0x7f4510005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:17.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 msgr2=0x7f452819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:17.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 0x7f452819bf40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4518005940 tx=0x7f45180040c0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 shutdown_connections 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f450c0b3c20 0x7f450c0b6130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f452810b880 0x7f45281a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f452810af30 0x7f452819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 --2- 172.21.15.44:0/1131861198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45281055f0 0x7f452819bf40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 >> 172.21.15.44:0/1131861198 conn(0x7f4528101060 msgr2=0x7f4528103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 shutdown_connections 2024-07-26T08:36:17.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:17.875+0000 7f452e000640 1 -- 172.21.15.44:0/1131861198 wait complete. 2024-07-26T08:36:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:18 smithi044 ceph-mon[32404]: pgmap v411: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T08:36:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:18 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1131861198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T08:36:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:18 smithi171 ceph-mon[37789]: pgmap v411: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T08:36:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:18 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1131861198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T08:36:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:18 smithi116 ceph-mon[37097]: pgmap v411: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T08:36:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:18 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1131861198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 -- 172.21.15.44:0/2038951494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8108d00 msgr2=0x7f2ef8109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 --2- 172.21.15.44:0/2038951494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8108d00 0x7f2ef8109110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009900 tx=0x7f2ee402a410 comp rx=0 tx=0).stop 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 -- 172.21.15.44:0/2038951494 shutdown_connections 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 --2- 172.21.15.44:0/2038951494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef8109650 0x7f2ef810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 --2- 172.21.15.44:0/2038951494 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8108d00 0x7f2ef8109110 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 --2- 172.21.15.44:0/2038951494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ef81033c0 0x7f2ef81037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 -- 172.21.15.44:0/2038951494 >> 172.21.15.44:0/2038951494 conn(0x7f2ef80fee10 msgr2=0x7f2ef8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 -- 172.21.15.44:0/2038951494 shutdown_connections 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.074+0000 7f2efe200640 1 -- 172.21.15.44:0/2038951494 wait complete. 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.075+0000 7f2efe200640 1 Processor -- start 2024-07-26T08:36:19.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 -- start start 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef81033c0 0x7f2ef819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ef8108d00 0x7f2ef819c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 0x7f2ef81a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81132e0 con 0x7f2ef8109650 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef8113160 con 0x7f2ef81033c0 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efe200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef8113460 con 0x7f2ef8108d00 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efd800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 0x7f2ef81a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef81033c0 0x7f2ef819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efd800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 0x7f2ef81a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46774/0 (socket says 172.21.15.44:46774) 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.076+0000 7f2efd800640 1 -- 172.21.15.44:0/2785497595 learned_addr learned my addr 172.21.15.44:0/2785497595 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2ef7e00640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ef8108d00 0x7f2ef819c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:19.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efce00640 1 -- 172.21.15.44:0/2785497595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ef8108d00 msgr2=0x7f2ef819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efce00640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ef8108d00 0x7f2ef819c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efce00640 1 -- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 msgr2=0x7f2ef81a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efce00640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 0x7f2ef81a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efce00640 1 -- 172.21.15.44:0/2785497595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef81a1f40 con 0x7f2ef81033c0 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efd800640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 0x7f2ef81a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.077+0000 7f2efce00640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef81033c0 0x7f2ef819bfd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e9f0 tx=0x7f2eec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.078+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec00ccc0 con 0x7f2ef81033c0 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.078+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eec005020 con 0x7f2ef81033c0 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.078+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef81a2230 con 0x7f2ef81033c0 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.078+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec004a00 con 0x7f2ef81033c0 2024-07-26T08:36:19.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.078+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef81a8990 con 0x7f2ef81033c0 2024-07-26T08:36:19.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.079+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef81037d0 con 0x7f2ef81033c0 2024-07-26T08:36:19.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.081+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f2eec004ba0 con 0x7f2ef81033c0 2024-07-26T08:36:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.083+0000 7f2ef5600640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2ee00b3ca0 0x7f2ee00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.083+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2eec0e4070 con 0x7f2ef81033c0 2024-07-26T08:36:19.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.083+0000 7f2ef7e00640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2ee00b3ca0 0x7f2ee00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:19.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.084+0000 7f2ef7e00640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2ee00b3ca0 0x7f2ee00b61b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009230 tx=0x7f2ee4031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:19.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.087+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2eec0ab1b0 con 0x7f2ef81033c0 2024-07-26T08:36:19.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.297+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2ef81a8c80 con 0x7f2ef81033c0 2024-07-26T08:36:19.299 INFO:teuthology.orchestra.run.smithi044.stdout:141733920862 2024-07-26T08:36:19.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.298+0000 7f2ef5600640 1 -- 172.21.15.44:0/2785497595 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2eec0aaab0 con 0x7f2ef81033c0 2024-07-26T08:36:19.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2ee00b3ca0 msgr2=0x7f2ee00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:19.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2ee00b3ca0 0x7f2ee00b61b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009230 tx=0x7f2ee4031400 comp rx=0 tx=0).stop 2024-07-26T08:36:19.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef81033c0 msgr2=0x7f2ef819bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef81033c0 0x7f2ef819bfd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e9f0 tx=0x7f2eec00eeb0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 shutdown_connections 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2ee00b3ca0 0x7f2ee00b61b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ef8108d00 0x7f2ef819c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2ef8109650 0x7f2ef81a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 --2- 172.21.15.44:0/2785497595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ef81033c0 0x7f2ef819bfd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 >> 172.21.15.44:0/2785497595 conn(0x7f2ef80fee10 msgr2=0x7f2ef8106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 shutdown_connections 2024-07-26T08:36:19.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:19.301+0000 7f2efe200640 1 -- 172.21.15.44:0/2785497595 wait complete. 2024-07-26T08:36:20.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:20 smithi044 ceph-mon[32404]: pgmap v412: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T08:36:20.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:20 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2785497595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T08:36:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:20 smithi171 ceph-mon[37789]: pgmap v412: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T08:36:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:20 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2785497595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T08:36:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:20 smithi116 ceph-mon[37097]: pgmap v412: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T08:36:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:20 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2785497595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T08:36:21.687 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757330 got 163208757336 for osd.5 2024-07-26T08:36:21.687 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:22 smithi171 ceph-mon[37789]: pgmap v413: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T08:36:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:22 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/57946265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T08:36:22.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:22 smithi044 ceph-mon[32404]: pgmap v413: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T08:36:22.367 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:22 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/57946265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T08:36:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:22 smithi116 ceph-mon[37097]: pgmap v413: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T08:36:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:22 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/57946265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T08:36:22.727 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:22.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705772 got 38654705778 for osd.0 2024-07-26T08:36:22.789 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.943+0000 7ff3e5800640 1 -- 172.21.15.44:0/1605412008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0075800 msgr2=0x7ff3e0075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.943+0000 7ff3e5800640 1 --2- 172.21.15.44:0/1605412008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0075800 0x7ff3e0075c10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009900 tx=0x7ff3c802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 -- 172.21.15.44:0/1605412008 shutdown_connections 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 --2- 172.21.15.44:0/1605412008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0113cb0 0x7ff3e0116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 --2- 172.21.15.44:0/1605412008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0075800 0x7ff3e0075c10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 --2- 172.21.15.44:0/1605412008 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3e0076db0 0x7ff3e00751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 -- 172.21.15.44:0/1605412008 >> 172.21.15.44:0/1605412008 conn(0x7ff3e00fee70 msgr2=0x7ff3e01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:23.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 -- 172.21.15.44:0/1605412008 shutdown_connections 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.944+0000 7ff3e5800640 1 -- 172.21.15.44:0/1605412008 wait complete. 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.945+0000 7ff3e5800640 1 Processor -- start 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.945+0000 7ff3e5800640 1 -- start start 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3e5800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3e0075800 0x7ff3e01a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3e5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 0x7ff3e01a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3e5800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0113cb0 0x7ff3e01a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3e5800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01198f0 con 0x7ff3e0113cb0 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3e5800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e0119770 con 0x7ff3e0076db0 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3e5800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e0119a70 con 0x7ff3e0075800 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 0x7ff3e01a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:23.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 0x7ff3e01a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:49364/0 (socket says 172.21.15.44:49364) 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.946+0000 7ff3d7e00640 1 -- 172.21.15.44:0/57946265 learned_addr learned my addr 172.21.15.44:0/57946265 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3e4e00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0113cb0 0x7ff3e01a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3dfe00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3e0075800 0x7ff3e01a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3e4e00640 1 -- 172.21.15.44:0/57946265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3e0075800 msgr2=0x7ff3e01a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3e4e00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3e0075800 0x7ff3e01a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3e4e00640 1 -- 172.21.15.44:0/57946265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 msgr2=0x7ff3e01a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3e4e00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 0x7ff3e01a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3e4e00640 1 -- 172.21.15.44:0/57946265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e01a6290 con 0x7ff3e0113cb0 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.947+0000 7ff3d7e00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 0x7ff3e01a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.948+0000 7ff3e4e00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0113cb0 0x7ff3e01a5b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009fd0 tx=0x7ff3d800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.948+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800ce20 con 0x7ff3e0113cb0 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.948+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3d8005020 con 0x7ff3e0113cb0 2024-07-26T08:36:23.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.948+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e01a6580 con 0x7ff3e0113cb0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.948+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8010690 con 0x7ff3e0113cb0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.948+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e01acce0 con 0x7ff3e0113cb0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.949+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e0075c10 con 0x7ff3e0113cb0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.952+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7ff3d80054d0 con 0x7ff3e0113cb0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.953+0000 7ff3dd600640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3c40b3c20 0x7ff3c40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.954+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff3d80e47e0 con 0x7ff3e0113cb0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.954+0000 7ff3dfe00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3c40b3c20 0x7ff3c40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.955+0000 7ff3dfe00640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3c40b3c20 0x7ff3c40b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009bd0 tx=0x7ff3d0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:23.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:20.955+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff3d80ab920 con 0x7ff3e0113cb0 2024-07-26T08:36:23.481 INFO:teuthology.orchestra.run.smithi044.stdout:115964117091 2024-07-26T08:36:23.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.164+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff3e0112450 con 0x7ff3e0113cb0 2024-07-26T08:36:23.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.165+0000 7ff3dd600640 1 -- 172.21.15.44:0/57946265 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff3d80ab220 con 0x7ff3e0113cb0 2024-07-26T08:36:23.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.167+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3c40b3c20 msgr2=0x7ff3c40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:23.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.167+0000 7ff3e5800640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3c40b3c20 0x7ff3c40b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0009bd0 tx=0x7ff3d0005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:23.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.167+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0113cb0 msgr2=0x7ff3e01a5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.167+0000 7ff3e5800640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0113cb0 0x7ff3e01a5b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009fd0 tx=0x7ff3d800ef10 comp rx=0 tx=0).stop 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 shutdown_connections 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff3c40b3c20 0x7ff3c40b6130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3e0113cb0 0x7ff3e01a5b60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3e0076db0 0x7ff3e01a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 --2- 172.21.15.44:0/57946265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3e0075800 0x7ff3e01a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 >> 172.21.15.44:0/57946265 conn(0x7ff3e00fee70 msgr2=0x7ff3e010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 shutdown_connections 2024-07-26T08:36:23.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:21.168+0000 7ff3e5800640 1 -- 172.21.15.44:0/57946265 wait complete. 2024-07-26T08:36:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:24 smithi171 ceph-mon[37789]: pgmap v414: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T08:36:24.413 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:24 smithi044 ceph-mon[32404]: pgmap v414: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T08:36:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:24 smithi116 ceph-mon[37097]: pgmap v414: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 -- 172.21.15.44:0/1238179141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 msgr2=0x7fee041120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 --2- 172.21.15.44:0/1238179141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 0x7fee041120b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009ca0 tx=0x7fedfc02a8b0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 -- 172.21.15.44:0/1238179141 shutdown_connections 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 --2- 172.21.15.44:0/1238179141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 0x7fee041120b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 --2- 172.21.15.44:0/1238179141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee0410af30 0x7fee0410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 --2- 172.21.15.44:0/1238179141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee041055f0 0x7fee04105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 -- 172.21.15.44:0/1238179141 >> 172.21.15.44:0/1238179141 conn(0x7fee04101060 msgr2=0x7fee041034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:24.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.619+0000 7fee08a00640 1 -- 172.21.15.44:0/1238179141 shutdown_connections 2024-07-26T08:36:24.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.620+0000 7fee08a00640 1 -- 172.21.15.44:0/1238179141 wait complete. 2024-07-26T08:36:24.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.621+0000 7fee08a00640 1 Processor -- start 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.621+0000 7fee08a00640 1 -- start start 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee08a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 0x7fee0419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee08a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee0410af30 0x7fee0419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee08a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 0x7fee041a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee08a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee04115530 con 0x7fee0410b880 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee08a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee041153b0 con 0x7fee041055f0 2024-07-26T08:36:24.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee08a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fee041156b0 con 0x7fee0410af30 2024-07-26T08:36:24.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee03400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 0x7fee0419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:24.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee03400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 0x7fee0419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:33212/0 (socket says 172.21.15.44:33212) 2024-07-26T08:36:24.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee03400640 1 -- 172.21.15.44:0/1961085128 learned_addr learned my addr 172.21.15.44:0/1961085128 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:24.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee03e00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 0x7fee041a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.622+0000 7fee02a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee0410af30 0x7fee0419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.623+0000 7fee03400640 1 -- 172.21.15.44:0/1961085128 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee0410af30 msgr2=0x7fee0419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.623+0000 7fee03400640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee0410af30 0x7fee0419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.623+0000 7fee03400640 1 -- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 msgr2=0x7fee041a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.623+0000 7fee03400640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 0x7fee041a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.623+0000 7fee03400640 1 -- 172.21.15.44:0/1961085128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee041a1e10 con 0x7fee041055f0 2024-07-26T08:36:24.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.623+0000 7fee03400640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 0x7fee0419bea0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedf400d950 tx=0x7fedf400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:24.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.624+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4017030 con 0x7fee041055f0 2024-07-26T08:36:24.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.624+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee041a2100 con 0x7fee041055f0 2024-07-26T08:36:24.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.624+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedf4004020 con 0x7fee041055f0 2024-07-26T08:36:24.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.624+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf40047a0 con 0x7fee041055f0 2024-07-26T08:36:24.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.624+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee041a8860 con 0x7fee041055f0 2024-07-26T08:36:24.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.625+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee04105a00 con 0x7fee041055f0 2024-07-26T08:36:24.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.628+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fedf4009a30 con 0x7fee041055f0 2024-07-26T08:36:24.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.629+0000 7fedfbe00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fede80b3c20 0x7fede80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:24.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.629+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fedf40e3c90 con 0x7fee041055f0 2024-07-26T08:36:24.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.629+0000 7fee02a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fede80b3c20 0x7fede80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:24.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.630+0000 7fee02a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fede80b3c20 0x7fede80b6130 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fee04107890 tx=0x7fedf000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:24.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.631+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fedf40aadd0 con 0x7fee041055f0 2024-07-26T08:36:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.849+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fee0419cfd0 con 0x7fee041055f0 2024-07-26T08:36:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.849+0000 7fedfbe00640 1 -- 172.21.15.44:0/1961085128 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fedf40aa6d0 con 0x7fee041055f0 2024-07-26T08:36:24.854 INFO:teuthology.orchestra.run.smithi044.stdout:81604378732 2024-07-26T08:36:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fede80b3c20 msgr2=0x7fede80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fede80b3c20 0x7fede80b6130 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fee04107890 tx=0x7fedf000b410 comp rx=0 tx=0).stop 2024-07-26T08:36:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 msgr2=0x7fee0419bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 0x7fee0419bea0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedf400d950 tx=0x7fedf400de10 comp rx=0 tx=0).stop 2024-07-26T08:36:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 shutdown_connections 2024-07-26T08:36:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fede80b3c20 0x7fede80b6130 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee0410af30 0x7fee0419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee0410b880 0x7fee041a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 --2- 172.21.15.44:0/1961085128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee041055f0 0x7fee0419bea0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 >> 172.21.15.44:0/1961085128 conn(0x7fee04101060 msgr2=0x7fee04103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 shutdown_connections 2024-07-26T08:36:24.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:24.852+0000 7fee08a00640 1 -- 172.21.15.44:0/1961085128 wait complete. 2024-07-26T08:36:24.912 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920857 got 141733920862 for osd.4 2024-07-26T08:36:24.913 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:24.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841592 got 283467841596 for osd.10 2024-07-26T08:36:24.914 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:25 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1961085128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T08:36:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:25 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1961085128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T08:36:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:25 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1961085128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T08:36:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:26 smithi171 ceph-mon[37789]: pgmap v415: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T08:36:26.343 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:26 smithi044 ceph-mon[32404]: pgmap v415: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T08:36:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:26 smithi116 ceph-mon[37097]: pgmap v415: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T08:36:27.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 -- 172.21.15.44:0/743194114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2010d660 msgr2=0x7feb2010fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 --2- 172.21.15.44:0/743194114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2010d660 0x7feb2010fb50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feb18009be0 tx=0x7feb1802a6c0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 -- 172.21.15.44:0/743194114 shutdown_connections 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 --2- 172.21.15.44:0/743194114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2010d660 0x7feb2010fb50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 --2- 172.21.15.44:0/743194114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104b50 0x7feb2010d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 --2- 172.21.15.44:0/743194114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb20104200 0x7feb20104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 -- 172.21.15.44:0/743194114 >> 172.21.15.44:0/743194114 conn(0x7feb200fdd00 msgr2=0x7feb20100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 -- 172.21.15.44:0/743194114 shutdown_connections 2024-07-26T08:36:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.135+0000 7feb26800640 1 -- 172.21.15.44:0/743194114 wait complete. 2024-07-26T08:36:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.137+0000 7feb26800640 1 Processor -- start 2024-07-26T08:36:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.137+0000 7feb26800640 1 -- start start 2024-07-26T08:36:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb26800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 0x7feb20199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb26800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb20104b50 0x7feb2019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb26800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb2010d660 0x7feb2019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb26800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb201132d0 con 0x7feb20104b50 2024-07-26T08:36:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb26800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb20113150 con 0x7feb20104200 2024-07-26T08:36:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb26800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feb20113450 con 0x7feb2010d660 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 0x7feb20199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb24a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb20104b50 0x7feb2019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 0x7feb20199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:33232/0 (socket says 172.21.15.44:33232) 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 -- 172.21.15.44:0/2895890887 learned_addr learned my addr 172.21.15.44:0/2895890887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25e00640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb2010d660 0x7feb2019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 -- 172.21.15.44:0/2895890887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb2010d660 msgr2=0x7feb2019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb2010d660 0x7feb2019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 -- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb20104b50 msgr2=0x7feb2019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb20104b50 0x7feb2019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.138+0000 7feb25400640 1 -- 172.21.15.44:0/2895890887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2019fbd0 con 0x7feb20104200 2024-07-26T08:36:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.139+0000 7feb25400640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 0x7feb20199c60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb1400e9c0 tx=0x7feb1400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.139+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1400cde0 con 0x7feb20104200 2024-07-26T08:36:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.139+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb140035f0 con 0x7feb20104200 2024-07-26T08:36:27.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.139+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14010690 con 0x7feb20104200 2024-07-26T08:36:27.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.139+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2019fec0 con 0x7feb20104200 2024-07-26T08:36:27.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.140+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb20101e20 con 0x7feb20104200 2024-07-26T08:36:27.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.141+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb20104610 con 0x7feb20104200 2024-07-26T08:36:27.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.143+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7feb14004000 con 0x7feb20104200 2024-07-26T08:36:27.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.146+0000 7feb1e000640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feb080b3c20 0x7feb080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.146+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7feb140e4450 con 0x7feb20104200 2024-07-26T08:36:27.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.146+0000 7feb24a00640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feb080b3c20 0x7feb080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.147+0000 7feb24a00640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feb080b3c20 0x7feb080b6130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feb2010f9e0 tx=0x7feb0c012040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:27.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.149+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feb14009510 con 0x7feb20104200 2024-07-26T08:36:27.283 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:27.323 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117088 got 115964117091 for osd.3 2024-07-26T08:36:27.323 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:27.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.369+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7feb2019ad90 con 0x7feb20104200 2024-07-26T08:36:27.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.370+0000 7feb1e000640 1 -- 172.21.15.44:0/2895890887 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7feb140ab590 con 0x7feb20104200 2024-07-26T08:36:27.371 INFO:teuthology.orchestra.run.smithi044.stdout:210453397584 2024-07-26T08:36:27.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feb080b3c20 msgr2=0x7feb080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feb080b3c20 0x7feb080b6130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feb2010f9e0 tx=0x7feb0c012040 comp rx=0 tx=0).stop 2024-07-26T08:36:27.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 msgr2=0x7feb20199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 0x7feb20199c60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb1400e9c0 tx=0x7feb1400ee80 comp rx=0 tx=0).stop 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 shutdown_connections 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feb080b3c20 0x7feb080b6130 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb2010d660 0x7feb2019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb20104b50 0x7feb2019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 --2- 172.21.15.44:0/2895890887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20104200 0x7feb20199c60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 >> 172.21.15.44:0/2895890887 conn(0x7feb200fdd00 msgr2=0x7feb20105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 shutdown_connections 2024-07-26T08:36:27.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.373+0000 7feb26800640 1 -- 172.21.15.44:0/2895890887 wait complete. 2024-07-26T08:36:27.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 -- 172.21.15.44:0/208751724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d001055d0 msgr2=0x7f5d001059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 --2- 172.21.15.44:0/208751724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d001055d0 0x7f5d001059e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009ca0 tx=0x7f5cf802a8c0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 -- 172.21.15.44:0/208751724 shutdown_connections 2024-07-26T08:36:27.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 --2- 172.21.15.44:0/208751724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d0010af10 0x7f5d0010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 --2- 172.21.15.44:0/208751724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d0010b860 0x7f5d00112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 --2- 172.21.15.44:0/208751724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d001055d0 0x7f5d001059e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 -- 172.21.15.44:0/208751724 >> 172.21.15.44:0/208751724 conn(0x7f5d00101040 msgr2=0x7f5d00103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 -- 172.21.15.44:0/208751724 shutdown_connections 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.604+0000 7f5cffe00640 1 -- 172.21.15.44:0/208751724 wait complete. 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.605+0000 7f5cffe00640 1 Processor -- start 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.605+0000 7f5cffe00640 1 -- start start 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cffe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d001055d0 0x7f5d0019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cffe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 0x7f5d0019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cffe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d0010b860 0x7f5d001a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cffe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00115530 con 0x7f5d0010b860 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cffe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001153b0 con 0x7f5d0010af10 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cffe00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5d001156b0 con 0x7f5d001055d0 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cff400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d0010b860 0x7f5d001a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 0x7f5d0019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d001055d0 0x7f5d0019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 0x7f5d0019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:33242/0 (socket says 172.21.15.44:33242) 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 -- 172.21.15.44:0/4114972365 learned_addr learned my addr 172.21.15.44:0/4114972365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 -- 172.21.15.44:0/4114972365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d001055d0 msgr2=0x7f5d0019bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d001055d0 0x7f5d0019bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 -- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d0010b860 msgr2=0x7f5d001a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d0010b860 0x7f5d001a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.606+0000 7f5cfe000640 1 -- 172.21.15.44:0/4114972365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d001a1e90 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.607+0000 7f5cfe000640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 0x7f5d0019c4c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800e9c0 tx=0x7f5ce800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.607+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800ccc0 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.607+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce80054d0 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.607+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d001a2180 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.607+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8010430 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.607+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d001a88e0 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.608+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d001059e0 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.611+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f5ce8005000 con 0x7f5d0010af10 2024-07-26T08:36:27.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.613+0000 7f5cf7400640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5ce40b3c20 0x7f5ce40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.613+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f5ce80e4680 con 0x7f5d0010af10 2024-07-26T08:36:27.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.613+0000 7f5cfea00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5ce40b3c20 0x7f5ce40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.614+0000 7f5cfea00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5ce40b3c20 0x7f5ce40b6130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8005e30 tx=0x7f5cf8009360 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:27.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.614+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5ce8055600 con 0x7f5d0010af10 2024-07-26T08:36:27.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.819+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5d0002e0c0 con 0x7f5d0010af10 2024-07-26T08:36:27.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.820+0000 7f5cf7400640 1 -- 172.21.15.44:0/4114972365 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5ce80052f0 con 0x7f5d0010af10 2024-07-26T08:36:27.821 INFO:teuthology.orchestra.run.smithi044.stdout:60129542256 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.822+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5ce40b3c20 msgr2=0x7f5ce40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5ce40b3c20 0x7f5ce40b6130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8005e30 tx=0x7f5cf8009360 comp rx=0 tx=0).stop 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 msgr2=0x7f5d0019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 0x7f5d0019c4c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800e9c0 tx=0x7f5ce800ee80 comp rx=0 tx=0).stop 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 shutdown_connections 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5ce40b3c20 0x7f5ce40b6130 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d0010b860 0x7f5d001a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d0010af10 0x7f5d0019c4c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 --2- 172.21.15.44:0/4114972365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d001055d0 0x7f5d0019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 >> 172.21.15.44:0/4114972365 conn(0x7f5d00101040 msgr2=0x7f5d00103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:27.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 shutdown_connections 2024-07-26T08:36:27.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.823+0000 7f5cffe00640 1 -- 172.21.15.44:0/4114972365 wait complete. 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 -- 172.21.15.44:0/2623879747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba01057b0 msgr2=0x7f9ba0105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 --2- 172.21.15.44:0/2623879747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba01057b0 0x7f9ba0105bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009900 tx=0x7f9b9002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 -- 172.21.15.44:0/2623879747 shutdown_connections 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 --2- 172.21.15.44:0/2623879747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba010b0f0 0x7f9ba010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 --2- 172.21.15.44:0/2623879747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 0x7f9ba0112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 --2- 172.21.15.44:0/2623879747 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba01057b0 0x7f9ba0105bc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 -- 172.21.15.44:0/2623879747 >> 172.21.15.44:0/2623879747 conn(0x7f9ba0101240 msgr2=0x7f9ba0103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:27.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 -- 172.21.15.44:0/2623879747 shutdown_connections 2024-07-26T08:36:27.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.974+0000 7f9ba8400640 1 -- 172.21.15.44:0/2623879747 wait complete. 2024-07-26T08:36:27.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.975+0000 7f9ba8400640 1 Processor -- start 2024-07-26T08:36:27.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 -- start start 2024-07-26T08:36:27.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba01057b0 0x7f9ba019c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 0x7f9ba019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 0x7f9ba01a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba0115740 con 0x7f9ba010b0f0 2024-07-26T08:36:27.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01155c0 con 0x7f9ba010ba40 2024-07-26T08:36:27.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba8400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01158c0 con 0x7f9ba01057b0 2024-07-26T08:36:27.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.976+0000 7f9ba5000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 0x7f9ba019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 0x7f9ba019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55848/0 (socket says 172.21.15.44:55848) 2024-07-26T08:36:27.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5000640 1 -- 172.21.15.44:0/786282465 learned_addr learned my addr 172.21.15.44:0/786282465 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:27.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5a00640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba01057b0 0x7f9ba019c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba6400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 0x7f9ba01a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5a00640 1 -- 172.21.15.44:0/786282465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 msgr2=0x7f9ba01a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5a00640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 0x7f9ba01a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5a00640 1 -- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 msgr2=0x7f9ba019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5a00640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 0x7f9ba019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5a00640 1 -- 172.21.15.44:0/786282465 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba01a20c0 con 0x7f9ba01057b0 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba6400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 0x7f9ba01a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.977+0000 7f9ba5000640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 0x7f9ba019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.978+0000 7f9ba5a00640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba01057b0 0x7f9ba019c150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9b9002a9e0 tx=0x7f9b90003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.978+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9003f070 con 0x7f9ba01057b0 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.978+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba01a2350 con 0x7f9ba01057b0 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.978+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b90031b50 con 0x7f9ba01057b0 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.978+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90035030 con 0x7f9ba01057b0 2024-07-26T08:36:27.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.978+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01a8b10 con 0x7f9ba01057b0 2024-07-26T08:36:27.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.979+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba0105bc0 con 0x7f9ba01057b0 2024-07-26T08:36:27.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.982+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f9b90042070 con 0x7f9ba01057b0 2024-07-26T08:36:27.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.984+0000 7f9b9e000640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b840b3c20 0x7f9b840b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:27.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.984+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9b901046f0 con 0x7f9ba01057b0 2024-07-26T08:36:27.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.984+0000 7f9ba5000640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b840b3c20 0x7f9b840b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:27.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.985+0000 7f9ba5000640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b840b3c20 0x7f9b840b6130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9ba010ac40 tx=0x7f9b8c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:27.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:27.988+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9b90031d00 con 0x7f9ba01057b0 2024-07-26T08:36:28.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.193+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f9ba002e0c0 con 0x7f9ba01057b0 2024-07-26T08:36:28.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:28 smithi044 ceph-mon[32404]: pgmap v416: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:36:28.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:28 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2895890887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T08:36:28.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:28 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4114972365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T08:36:28.196 INFO:teuthology.orchestra.run.smithi044.stdout:309237645371 2024-07-26T08:36:28.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.194+0000 7f9b9e000640 1 -- 172.21.15.44:0/786282465 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f9b900cb1e0 con 0x7f9ba01057b0 2024-07-26T08:36:28.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.197+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b840b3c20 msgr2=0x7f9b840b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:28.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.197+0000 7f9ba8400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b840b3c20 0x7f9b840b6130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9ba010ac40 tx=0x7f9b8c009500 comp rx=0 tx=0).stop 2024-07-26T08:36:28.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.197+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba01057b0 msgr2=0x7f9ba019c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:28.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.197+0000 7f9ba8400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba01057b0 0x7f9ba019c150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9b9002a9e0 tx=0x7f9b90003b90 comp rx=0 tx=0).stop 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 shutdown_connections 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9b840b3c20 0x7f9b840b6130 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ba010ba40 0x7f9ba01a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ba010b0f0 0x7f9ba019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 --2- 172.21.15.44:0/786282465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ba01057b0 0x7f9ba019c150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 >> 172.21.15.44:0/786282465 conn(0x7f9ba0101240 msgr2=0x7f9ba0103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 shutdown_connections 2024-07-26T08:36:28.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:28.198+0000 7f9ba8400640 1 -- 172.21.15.44:0/786282465 wait complete. 2024-07-26T08:36:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:28 smithi171 ceph-mon[37789]: pgmap v416: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:36:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:28 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2895890887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T08:36:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:28 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4114972365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T08:36:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:28 smithi116 ceph-mon[37097]: pgmap v416: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T08:36:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:28 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2895890887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T08:36:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:28 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4114972365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T08:36:29.257 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:29 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/786282465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T08:36:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:29 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/786282465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T08:36:29.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378726 got 81604378732 for osd.2 2024-07-26T08:36:29.489 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:29 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/786282465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T08:36:29.955 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397576 got 210453397584 for osd.7 2024-07-26T08:36:29.956 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:30.238 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542249 got 60129542256 for osd.1 2024-07-26T08:36:30.239 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:30 smithi171 ceph-mon[37789]: pgmap v417: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T08:36:30.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:30 smithi044 ceph-mon[32404]: pgmap v417: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T08:36:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:30 smithi116 ceph-mon[37097]: pgmap v417: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T08:36:30.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645365 got 309237645371 for osd.11 2024-07-26T08:36:30.573 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:31.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 -- 172.21.15.44:0/3704267044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075ad0 msgr2=0x7fbbc010f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 --2- 172.21.15.44:0/3704267044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075ad0 0x7fbbc010f640 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009900 tx=0x7fbbac02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 -- 172.21.15.44:0/3704267044 shutdown_connections 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 --2- 172.21.15.44:0/3704267044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 0x7fbbc0111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 --2- 172.21.15.44:0/3704267044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075ad0 0x7fbbc010f640 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 --2- 172.21.15.44:0/3704267044 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbc0075180 0x7fbbc0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 -- 172.21.15.44:0/3704267044 >> 172.21.15.44:0/3704267044 conn(0x7fbbc00ff010 msgr2=0x7fbbc0101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.094+0000 7fbbc6800640 1 -- 172.21.15.44:0/3704267044 shutdown_connections 2024-07-26T08:36:31.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.095+0000 7fbbc6800640 1 -- 172.21.15.44:0/3704267044 wait complete. 2024-07-26T08:36:31.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.096+0000 7fbbc6800640 1 Processor -- start 2024-07-26T08:36:31.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.096+0000 7fbbc6800640 1 -- start start 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 0x7fbbc019c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbc0075ad0 0x7fbbc019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 0x7fbbc01a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbc6800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc01156f0 con 0x7fbbc010fb80 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbc6800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc0115570 con 0x7fbbc0075180 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbc6800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc0115870 con 0x7fbbc0075ad0 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbbf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 0x7fbbc019c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.097+0000 7fbbbf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 0x7fbbc019c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:33280/0 (socket says 172.21.15.44:33280) 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbf400640 1 -- 172.21.15.44:0/2600172037 learned_addr learned my addr 172.21.15.44:0/2600172037 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:31.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbfe00640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 0x7fbbc01a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:31.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbea00640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbc0075ad0 0x7fbbc019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbf400640 1 -- 172.21.15.44:0/2600172037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbc0075ad0 msgr2=0x7fbbc019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbf400640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbc0075ad0 0x7fbbc019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbf400640 1 -- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 msgr2=0x7fbbc01a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbf400640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 0x7fbbc01a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbf400640 1 -- 172.21.15.44:0/2600172037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc01a2030 con 0x7fbbc0075180 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.098+0000 7fbbbfe00640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 0x7fbbc01a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.099+0000 7fbbbf400640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 0x7fbbc019c0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000e9f0 tx=0x7fbbb000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.099+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb000cd80 con 0x7fbbc0075180 2024-07-26T08:36:31.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.099+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbb00108e0 con 0x7fbbc0075180 2024-07-26T08:36:31.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.099+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0018980 con 0x7fbbc0075180 2024-07-26T08:36:31.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.099+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc01a2320 con 0x7fbbc0075180 2024-07-26T08:36:31.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.099+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc01a8a80 con 0x7fbbc0075180 2024-07-26T08:36:31.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.100+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc0076690 con 0x7fbbc0075180 2024-07-26T08:36:31.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.103+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fbbb0010430 con 0x7fbbc0075180 2024-07-26T08:36:31.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.105+0000 7fbbb7e00640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbba40b3ca0 0x7fbba40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:31.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.105+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fbbb00e46e0 con 0x7fbbc0075180 2024-07-26T08:36:31.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.105+0000 7fbbbea00640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbba40b3ca0 0x7fbba40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:31.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.106+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbbb00ab820 con 0x7fbbc0075180 2024-07-26T08:36:31.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.107+0000 7fbbbea00640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbba40b3ca0 0x7fbba40b61b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbbac02a9e0 tx=0x7fbbac005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:31.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.321+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fbbc0075590 con 0x7fbbc0075180 2024-07-26T08:36:31.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.322+0000 7fbbb7e00640 1 -- 172.21.15.44:0/2600172037 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbbb00ab120 con 0x7fbbc0075180 2024-07-26T08:36:31.323 INFO:teuthology.orchestra.run.smithi044.stdout:261993005124 2024-07-26T08:36:31.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.324+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbba40b3ca0 msgr2=0x7fbba40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.324+0000 7fbbc6800640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbba40b3ca0 0x7fbba40b61b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbbac02a9e0 tx=0x7fbbac005a20 comp rx=0 tx=0).stop 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.324+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 msgr2=0x7fbbc019c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.324+0000 7fbbc6800640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 0x7fbbc019c0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000e9f0 tx=0x7fbbb000eeb0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 shutdown_connections 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbba40b3ca0 0x7fbba40b61b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbbc0075ad0 0x7fbbc019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbbc010fb80 0x7fbbc01a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 --2- 172.21.15.44:0/2600172037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0075180 0x7fbbc019c0c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:31.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 >> 172.21.15.44:0/2600172037 conn(0x7fbbc00ff010 msgr2=0x7fbbc0109720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:31.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 shutdown_connections 2024-07-26T08:36:31.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:31.325+0000 7fbbc6800640 1 -- 172.21.15.44:0/2600172037 wait complete. 2024-07-26T08:36:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:32 smithi171 ceph-mon[37789]: pgmap v418: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:32 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2600172037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T08:36:32.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:32 smithi044 ceph-mon[32404]: pgmap v418: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:32.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:32 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2600172037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T08:36:32.475 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005119 got 261993005124 for osd.9 2024-07-26T08:36:32.475 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:32 smithi116 ceph-mon[37097]: pgmap v418: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:32 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2600172037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T08:36:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 -- 172.21.15.44:0/1310218773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 msgr2=0x7fe38c1166a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 --2- 172.21.15.44:0/1310218773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1166a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe384009ca0 tx=0x7fe38402a8c0 comp rx=0 tx=0).stop 2024-07-26T08:36:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 -- 172.21.15.44:0/1310218773 shutdown_connections 2024-07-26T08:36:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 --2- 172.21.15.44:0/1310218773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe38c10f520 0x7fe38c10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 --2- 172.21.15.44:0/1310218773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1166a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:33.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 --2- 172.21.15.44:0/1310218773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 0x7fe38c109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:33.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 -- 172.21.15.44:0/1310218773 >> 172.21.15.44:0/1310218773 conn(0x7fe38c077b10 msgr2=0x7fe38c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:33.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 -- 172.21.15.44:0/1310218773 shutdown_connections 2024-07-26T08:36:33.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.811+0000 7fe391600640 1 -- 172.21.15.44:0/1310218773 wait complete. 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.812+0000 7fe391600640 1 Processor -- start 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.812+0000 7fe391600640 1 -- start start 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe391600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 0x7fe38c1a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe391600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe38c10f520 0x7fe38c1a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe391600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe391600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c119b80 con 0x7fe38c109be0 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe391600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c119a00 con 0x7fe38c10fe70 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe391600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c119d00 con 0x7fe38c10f520 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe38aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 0x7fe38c1a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe38b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:33.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe38aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 0x7fe38c1a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36342/0 (socket says 172.21.15.44:36342) 2024-07-26T08:36:33.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe38aa00640 1 -- 172.21.15.44:0/683325437 learned_addr learned my addr 172.21.15.44:0/683325437 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:33.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.813+0000 7fe383e00640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe38c10f520 0x7fe38c1a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.814+0000 7fe38b400640 1 -- 172.21.15.44:0/683325437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe38c10f520 msgr2=0x7fe38c1a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.814+0000 7fe38b400640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe38c10f520 0x7fe38c1a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.814+0000 7fe38b400640 1 -- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 msgr2=0x7fe38c1a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.814+0000 7fe38b400640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 0x7fe38c1a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.814+0000 7fe38b400640 1 -- 172.21.15.44:0/683325437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38c1a6400 con 0x7fe38c10fe70 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.814+0000 7fe38b400640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1a5cd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe384005e30 tx=0x7fe3840317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.815+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384003b70 con 0x7fe38c10fe70 2024-07-26T08:36:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.815+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe384003d10 con 0x7fe38c10fe70 2024-07-26T08:36:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.815+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38402f970 con 0x7fe38c10fe70 2024-07-26T08:36:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.815+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38c1a6690 con 0x7fe38c10fe70 2024-07-26T08:36:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.815+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe38c1ace50 con 0x7fe38c10fe70 2024-07-26T08:36:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.816+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe38c10ae60 con 0x7fe38c10fe70 2024-07-26T08:36:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.819+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fe384004b40 con 0x7fe38c10fe70 2024-07-26T08:36:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.820+0000 7fe383400640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe3700b3ca0 0x7fe3700b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.820+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fe384105460 con 0x7fe38c10fe70 2024-07-26T08:36:33.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.820+0000 7fe38aa00640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe3700b3ca0 0x7fe3700b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:33.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.821+0000 7fe38aa00640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe3700b3ca0 0x7fe3700b61b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009c90 tx=0x7fe37c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:33.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:33.822+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe3840cc5a0 con 0x7fe38c10fe70 2024-07-26T08:36:34.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.033+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe38c111c60 con 0x7fe38c10fe70 2024-07-26T08:36:34.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.034+0000 7fe383400640 1 -- 172.21.15.44:0/683325437 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe3840cbea0 con 0x7fe38c10fe70 2024-07-26T08:36:34.035 INFO:teuthology.orchestra.run.smithi044.stdout:188978561109 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe3700b3ca0 msgr2=0x7fe3700b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe3700b3ca0 0x7fe3700b61b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009c90 tx=0x7fe37c005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 msgr2=0x7fe38c1a5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1a5cd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe384005e30 tx=0x7fe3840317c0 comp rx=0 tx=0).stop 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 shutdown_connections 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe3700b3ca0 0x7fe3700b61b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe38c10f520 0x7fe38c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe38c10fe70 0x7fe38c1a5cd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 --2- 172.21.15.44:0/683325437 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe38c109be0 0x7fe38c1a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 >> 172.21.15.44:0/683325437 conn(0x7fe38c077b10 msgr2=0x7fe38c10ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 shutdown_connections 2024-07-26T08:36:34.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:34.037+0000 7fe391600640 1 -- 172.21.15.44:0/683325437 wait complete. 2024-07-26T08:36:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:34 smithi044 ceph-mon[32404]: pgmap v419: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:34.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:34 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/683325437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T08:36:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:34 smithi116 ceph-mon[37097]: pgmap v419: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:34 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/683325437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T08:36:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:34 smithi171 ceph-mon[37789]: pgmap v419: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:34 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/683325437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T08:36:34.868 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561103 got 188978561109 for osd.6 2024-07-26T08:36:34.868 DEBUG:teuthology.parallel:result is None 2024-07-26T08:36:34.868 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T08:36:34.869 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T08:36:35.351 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:36 smithi044 ceph-mon[32404]: pgmap v420: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:36 smithi116 ceph-mon[37097]: pgmap v420: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:36 smithi171 ceph-mon[37789]: pgmap v420: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:37.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 -- 172.21.15.44:0/359253678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5501099f0 msgr2=0x7fc550109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:37.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 --2- 172.21.15.44:0/359253678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5501099f0 0x7fc550109e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc540009900 tx=0x7fc54002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 -- 172.21.15.44:0/359253678 shutdown_connections 2024-07-26T08:36:37.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 --2- 172.21.15.44:0/359253678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc55010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 --2- 172.21.15.44:0/359253678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55010fc80 0x7fc5501164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 --2- 172.21.15.44:0/359253678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5501099f0 0x7fc550109e00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 -- 172.21.15.44:0/359253678 >> 172.21.15.44:0/359253678 conn(0x7fc550078ff0 msgr2=0x7fc550107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:37.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 -- 172.21.15.44:0/359253678 shutdown_connections 2024-07-26T08:36:37.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.740+0000 7fc555800640 1 -- 172.21.15.44:0/359253678 wait complete. 2024-07-26T08:36:37.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.741+0000 7fc555800640 1 Processor -- start 2024-07-26T08:36:37.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 -- start start 2024-07-26T08:36:37.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5501099f0 0x7fc5501a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc5501a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc55010fc80 0x7fc5501a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc550119940 con 0x7fc55010fc80 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5501197c0 con 0x7fc5501099f0 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.742+0000 7fc555800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc550119ac0 con 0x7fc55010f330 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc5501a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:37.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc554e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc55010fc80 0x7fc5501a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:37.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc5501a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35840/0 (socket says 172.21.15.44:35840) 2024-07-26T08:36:37.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 -- 172.21.15.44:0/3443984127 learned_addr learned my addr 172.21.15.44:0/3443984127 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:37.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc54fe00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5501099f0 0x7fc5501a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:37.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 -- 172.21.15.44:0/3443984127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5501099f0 msgr2=0x7fc5501a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:37.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5501099f0 0x7fc5501a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 -- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc55010fc80 msgr2=0x7fc5501a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc55010fc80 0x7fc5501a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc547e00640 1 -- 172.21.15.44:0/3443984127 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5501a62e0 con 0x7fc55010f330 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.743+0000 7fc554e00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc55010fc80 0x7fc5501a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.744+0000 7fc547e00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc5501a0910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc53800e9f0 tx=0x7fc53800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.744+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53800ccc0 con 0x7fc55010f330 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.744+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5380108e0 con 0x7fc55010f330 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.744+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5501a65d0 con 0x7fc55010f330 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.744+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538018860 con 0x7fc55010f330 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.744+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5501acd30 con 0x7fc55010f330 2024-07-26T08:36:37.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.745+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc550109e00 con 0x7fc55010f330 2024-07-26T08:36:37.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.748+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fc538010430 con 0x7fc55010f330 2024-07-26T08:36:37.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.750+0000 7fc54d600640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc5340b3c20 0x7fc5340b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:37.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.750+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fc5380e48f0 con 0x7fc55010f330 2024-07-26T08:36:37.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.750+0000 7fc54fe00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc5340b3c20 0x7fc5340b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:37.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.751+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc5380ab3e0 con 0x7fc55010f330 2024-07-26T08:36:37.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.751+0000 7fc54fe00640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc5340b3c20 0x7fc5340b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc540031440 tx=0x7fc540009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:37.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.958+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc550063230 con 0x7fc5340b3c20 2024-07-26T08:36:37.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.969+0000 7fc54d600640 1 -- 172.21.15.44:0/3443984127 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340804 (secure 0 0 0) 0x7fc550063230 con 0x7fc5340b3c20 2024-07-26T08:36:37.976 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:36:37.978 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:36:37.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc5340b3c20 msgr2=0x7fc5340b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:37.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc5340b3c20 0x7fc5340b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc540031440 tx=0x7fc540009560 comp rx=0 tx=0).stop 2024-07-26T08:36:37.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 msgr2=0x7fc5501a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc5501a0910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc53800e9f0 tx=0x7fc53800eeb0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 shutdown_connections 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fc5340b3c20 0x7fc5340b6130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc55010f330 0x7fc5501a0910 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc55010fc80 0x7fc5501a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.976+0000 7fc555800640 1 --2- 172.21.15.44:0/3443984127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5501099f0 0x7fc5501a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.977+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 >> 172.21.15.44:0/3443984127 conn(0x7fc550078ff0 msgr2=0x7fc550107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.977+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 shutdown_connections 2024-07-26T08:36:37.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:37.977+0000 7fc555800640 1 -- 172.21.15.44:0/3443984127 wait complete. 2024-07-26T08:36:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:38 smithi044 ceph-mon[32404]: pgmap v421: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:38.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:38 smithi171 ceph-mon[37789]: pgmap v421: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:38 smithi116 ceph-mon[37097]: pgmap v421: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:38.825 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-07-26T08:36:36.850435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665371,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1349,"num_read_kb":786,"num_write":387,"num_write_kb":2308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":820,"ondisk_log_size":820,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365728,"kb_used_data":12416,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576096,"statfs":{"total":1151940427776,"available":1151565922304,"internally_reserved":0,"allocated":12713984,"data_stored":8440763,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":60,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007186"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":184,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:34.407675+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:36:34.407675+0000","last_peered":"2024-07-26T08:36:34.407675+0000","last_clean":"2024-07-26T08:36:34.407675+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:36:34.407675+0000","last_undegraded":"2024-07-26T08:36:34.407675+0000","last_fullsized":"2024-07-26T08:36:34.407675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":125,"num_read_kb":125,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746213+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:36:12.746213+0000","last_peered":"2024-07-26T08:36:12.746213+0000","last_clean":"2024-07-26T08:36:12.746213+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:36:12.746213+0000","last_undegraded":"2024-07-26T08:36:12.746213+0000","last_fullsized":"2024-07-26T08:36:12.746213+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965699+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:34:13.965699+0000","last_peered":"2024-07-26T08:34:13.965699+0000","last_clean":"2024-07-26T08:34:13.965699+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:34:13.965699+0000","last_undegraded":"2024-07-26T08:34:13.965699+0000","last_fullsized":"2024-07-26T08:34:13.965699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:29.071793+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:29.071793+0000","last_peered":"2024-07-26T08:34:29.071793+0000","last_clean":"2024-07-26T08:34:29.071793+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:29.071793+0000","last_undegraded":"2024-07-26T08:34:29.071793+0000","last_fullsized":"2024-07-26T08:34:29.071793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740925+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:36:12.740925+0000","last_peered":"2024-07-26T08:36:12.740925+0000","last_clean":"2024-07-26T08:36:12.740925+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:36:12.740925+0000","last_undegraded":"2024-07-26T08:36:12.740925+0000","last_fullsized":"2024-07-26T08:36:12.740925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.593535+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:34:14.593535+0000","last_peered":"2024-07-26T08:34:14.593535+0000","last_clean":"2024-07-26T08:34:14.593535+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:34:14.593535+0000","last_undegraded":"2024-07-26T08:34:14.593535+0000","last_fullsized":"2024-07-26T08:34:14.593535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:33.090130+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:34:33.090130+0000","last_peered":"2024-07-26T08:34:33.090130+0000","last_clean":"2024-07-26T08:34:33.090130+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:34:33.090130+0000","last_undegraded":"2024-07-26T08:34:33.090130+0000","last_fullsized":"2024-07-26T08:34:33.090130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741267+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:36:12.741267+0000","last_peered":"2024-07-26T08:36:12.741267+0000","last_clean":"2024-07-26T08:36:12.741267+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:36:12.741267+0000","last_undegraded":"2024-07-26T08:36:12.741267+0000","last_fullsized":"2024-07-26T08:36:12.741267+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955356+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:34:13.955356+0000","last_peered":"2024-07-26T08:34:13.955356+0000","last_clean":"2024-07-26T08:34:13.955356+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:34:13.955356+0000","last_undegraded":"2024-07-26T08:34:13.955356+0000","last_fullsized":"2024-07-26T08:34:13.955356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.113096+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:34:32.113096+0000","last_peered":"2024-07-26T08:34:32.113096+0000","last_clean":"2024-07-26T08:34:32.113096+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:34:32.113096+0000","last_undegraded":"2024-07-26T08:34:32.113096+0000","last_fullsized":"2024-07-26T08:34:32.113096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734672+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:36:12.734672+0000","last_peered":"2024-07-26T08:36:12.734672+0000","last_clean":"2024-07-26T08:36:12.734672+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:36:12.734672+0000","last_undegraded":"2024-07-26T08:36:12.734672+0000","last_fullsized":"2024-07-26T08:36:12.734672+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954873+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:34:13.954873+0000","last_peered":"2024-07-26T08:34:13.954873+0000","last_clean":"2024-07-26T08:34:13.954873+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:34:13.954873+0000","last_undegraded":"2024-07-26T08:34:13.954873+0000","last_fullsized":"2024-07-26T08:34:13.954873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742090+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:36:12.742090+0000","last_peered":"2024-07-26T08:36:12.742090+0000","last_clean":"2024-07-26T08:36:12.742090+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:36:12.742090+0000","last_undegraded":"2024-07-26T08:36:12.742090+0000","last_fullsized":"2024-07-26T08:36:12.742090+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106175+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:34:32.106175+0000","last_peered":"2024-07-26T08:34:32.106175+0000","last_clean":"2024-07-26T08:34:32.106175+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:34:32.106175+0000","last_undegraded":"2024-07-26T08:34:32.106175+0000","last_fullsized":"2024-07-26T08:34:32.106175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955043+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.955043+0000","last_peered":"2024-07-26T08:34:13.955043+0000","last_clean":"2024-07-26T08:34:13.955043+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:34:13.955043+0000","last_undegraded":"2024-07-26T08:34:13.955043+0000","last_fullsized":"2024-07-26T08:34:13.955043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743210+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:36:12.743210+0000","last_peered":"2024-07-26T08:36:12.743210+0000","last_clean":"2024-07-26T08:36:12.743210+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:36:12.743210+0000","last_undegraded":"2024-07-26T08:36:12.743210+0000","last_fullsized":"2024-07-26T08:36:12.743210+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.079193+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:34:31.079193+0000","last_peered":"2024-07-26T08:34:31.079193+0000","last_clean":"2024-07-26T08:34:31.079193+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:34:31.079193+0000","last_undegraded":"2024-07-26T08:34:31.079193+0000","last_fullsized":"2024-07-26T08:34:31.079193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:53.482447+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:34:53.482447+0000","last_peered":"2024-07-26T08:34:53.482447+0000","last_clean":"2024-07-26T08:34:53.482447+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:34:53.482447+0000","last_undegraded":"2024-07-26T08:34:53.482447+0000","last_fullsized":"2024-07-26T08:34:53.482447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740202+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:36:12.740202+0000","last_peered":"2024-07-26T08:36:12.740202+0000","last_clean":"2024-07-26T08:36:12.740202+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:36:12.740202+0000","last_undegraded":"2024-07-26T08:36:12.740202+0000","last_fullsized":"2024-07-26T08:36:12.740202+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097156+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:34:31.097156+0000","last_peered":"2024-07-26T08:34:31.097156+0000","last_clean":"2024-07-26T08:34:31.097156+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:34:31.097156+0000","last_undegraded":"2024-07-26T08:34:31.097156+0000","last_fullsized":"2024-07-26T08:34:31.097156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.596626+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:34:14.596626+0000","last_peered":"2024-07-26T08:34:14.596626+0000","last_clean":"2024-07-26T08:34:14.596626+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:34:14.596626+0000","last_undegraded":"2024-07-26T08:34:14.596626+0000","last_fullsized":"2024-07-26T08:34:14.596626+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.693280+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:36:12.693280+0000","last_peered":"2024-07-26T08:36:12.693280+0000","last_clean":"2024-07-26T08:36:12.693280+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:36:12.693280+0000","last_undegraded":"2024-07-26T08:36:12.693280+0000","last_fullsized":"2024-07-26T08:36:12.693280+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:28.060952+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:34:28.060952+0000","last_peered":"2024-07-26T08:34:28.060952+0000","last_clean":"2024-07-26T08:34:28.060952+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:34:28.060952+0000","last_undegraded":"2024-07-26T08:34:28.060952+0000","last_fullsized":"2024-07-26T08:34:28.060952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955242+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:34:13.955242+0000","last_peered":"2024-07-26T08:34:13.955242+0000","last_clean":"2024-07-26T08:34:13.955242+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:34:13.955242+0000","last_undegraded":"2024-07-26T08:34:13.955242+0000","last_fullsized":"2024-07-26T08:34:13.955242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744261+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:36:12.744261+0000","last_peered":"2024-07-26T08:36:12.744261+0000","last_clean":"2024-07-26T08:36:12.744261+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:36:12.744261+0000","last_undegraded":"2024-07-26T08:36:12.744261+0000","last_fullsized":"2024-07-26T08:36:12.744261+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108189+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:34:32.108189+0000","last_peered":"2024-07-26T08:34:32.108189+0000","last_clean":"2024-07-26T08:34:32.108189+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:34:32.108189+0000","last_undegraded":"2024-07-26T08:34:32.108189+0000","last_fullsized":"2024-07-26T08:34:32.108189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966025+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:34:13.966025+0000","last_peered":"2024-07-26T08:34:13.966025+0000","last_clean":"2024-07-26T08:34:13.966025+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:34:13.966025+0000","last_undegraded":"2024-07-26T08:34:13.966025+0000","last_fullsized":"2024-07-26T08:34:13.966025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728411+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:36:12.728411+0000","last_peered":"2024-07-26T08:36:12.728411+0000","last_clean":"2024-07-26T08:36:12.728411+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:36:12.728411+0000","last_undegraded":"2024-07-26T08:36:12.728411+0000","last_fullsized":"2024-07-26T08:36:12.728411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'17","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:11.341058+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:35:11.341058+0000","last_peered":"2024-07-26T08:35:11.341058+0000","last_clean":"2024-07-26T08:35:11.341058+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:35:11.341058+0000","last_undegraded":"2024-07-26T08:35:11.341058+0000","last_fullsized":"2024-07-26T08:35:11.341058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591736+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:34:14.591736+0000","last_peered":"2024-07-26T08:34:14.591736+0000","last_clean":"2024-07-26T08:34:14.591736+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:34:14.591736+0000","last_undegraded":"2024-07-26T08:34:14.591736+0000","last_fullsized":"2024-07-26T08:34:14.591736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733504+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:36:12.733504+0000","last_peered":"2024-07-26T08:36:12.733504+0000","last_clean":"2024-07-26T08:36:12.733504+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:36:12.733504+0000","last_undegraded":"2024-07-26T08:36:12.733504+0000","last_fullsized":"2024-07-26T08:36:12.733504+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108436+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:34:32.108436+0000","last_peered":"2024-07-26T08:34:32.108436+0000","last_clean":"2024-07-26T08:34:32.108436+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:34:32.108436+0000","last_undegraded":"2024-07-26T08:34:32.108436+0000","last_fullsized":"2024-07-26T08:34:32.108436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961966+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:34:13.961966+0000","last_peered":"2024-07-26T08:34:13.961966+0000","last_clean":"2024-07-26T08:34:13.961966+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:34:13.961966+0000","last_undegraded":"2024-07-26T08:34:13.961966+0000","last_fullsized":"2024-07-26T08:34:13.961966+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743910+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:36:12.743910+0000","last_peered":"2024-07-26T08:36:12.743910+0000","last_clean":"2024-07-26T08:36:12.743910+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:36:12.743910+0000","last_undegraded":"2024-07-26T08:36:12.743910+0000","last_fullsized":"2024-07-26T08:36:12.743910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.095298+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:34:30.095298+0000","last_peered":"2024-07-26T08:34:30.095298+0000","last_clean":"2024-07-26T08:34:30.095298+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:34:30.095298+0000","last_undegraded":"2024-07-26T08:34:30.095298+0000","last_fullsized":"2024-07-26T08:34:30.095298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965933+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:34:13.965933+0000","last_peered":"2024-07-26T08:34:13.965933+0000","last_clean":"2024-07-26T08:34:13.965933+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:34:13.965933+0000","last_undegraded":"2024-07-26T08:34:13.965933+0000","last_fullsized":"2024-07-26T08:34:13.965933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739889+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:36:12.739889+0000","last_peered":"2024-07-26T08:36:12.739889+0000","last_clean":"2024-07-26T08:36:12.739889+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:36:12.739889+0000","last_undegraded":"2024-07-26T08:36:12.739889+0000","last_fullsized":"2024-07-26T08:36:12.739889+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":173,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:31.400634+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:36:31.400634+0000","last_peered":"2024-07-26T08:36:31.400634+0000","last_clean":"2024-07-26T08:36:31.400634+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:36:31.400634+0000","last_undegraded":"2024-07-26T08:36:31.400634+0000","last_fullsized":"2024-07-26T08:36:31.400634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328912+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:34:14.328912+0000","last_peered":"2024-07-26T08:34:14.328912+0000","last_clean":"2024-07-26T08:34:14.328912+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:34:14.328912+0000","last_undegraded":"2024-07-26T08:34:14.328912+0000","last_fullsized":"2024-07-26T08:34:14.328912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729525+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:36:12.729525+0000","last_peered":"2024-07-26T08:36:12.729525+0000","last_clean":"2024-07-26T08:36:12.729525+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:36:12.729525+0000","last_undegraded":"2024-07-26T08:36:12.729525+0000","last_fullsized":"2024-07-26T08:36:12.729525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":178,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:35.445196+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:36:35.445196+0000","last_peered":"2024-07-26T08:36:35.445196+0000","last_clean":"2024-07-26T08:36:35.445196+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:36:35.445196+0000","last_undegraded":"2024-07-26T08:36:35.445196+0000","last_fullsized":"2024-07-26T08:36:35.445196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329123+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:34:14.329123+0000","last_peered":"2024-07-26T08:34:14.329123+0000","last_clean":"2024-07-26T08:34:14.329123+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:34:14.329123+0000","last_undegraded":"2024-07-26T08:34:14.329123+0000","last_fullsized":"2024-07-26T08:34:14.329123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733921+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:36:12.733921+0000","last_peered":"2024-07-26T08:36:12.733921+0000","last_clean":"2024-07-26T08:36:12.733921+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:36:12.733921+0000","last_undegraded":"2024-07-26T08:36:12.733921+0000","last_fullsized":"2024-07-26T08:36:12.733921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096346+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:31.096346+0000","last_peered":"2024-07-26T08:34:31.096346+0000","last_clean":"2024-07-26T08:34:31.096346+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:31.096346+0000","last_undegraded":"2024-07-26T08:34:31.096346+0000","last_fullsized":"2024-07-26T08:34:31.096346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953794+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.953794+0000","last_peered":"2024-07-26T08:34:13.953794+0000","last_clean":"2024-07-26T08:34:13.953794+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:34:13.953794+0000","last_undegraded":"2024-07-26T08:34:13.953794+0000","last_fullsized":"2024-07-26T08:34:13.953794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745169+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:36:12.745169+0000","last_peered":"2024-07-26T08:36:12.745169+0000","last_clean":"2024-07-26T08:36:12.745169+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:36:12.745169+0000","last_undegraded":"2024-07-26T08:36:12.745169+0000","last_fullsized":"2024-07-26T08:36:12.745169+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.109130+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:34:32.109130+0000","last_peered":"2024-07-26T08:34:32.109130+0000","last_clean":"2024-07-26T08:34:32.109130+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:34:32.109130+0000","last_undegraded":"2024-07-26T08:34:32.109130+0000","last_fullsized":"2024-07-26T08:34:32.109130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252777+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:34:14.252777+0000","last_peered":"2024-07-26T08:34:14.252777+0000","last_clean":"2024-07-26T08:34:14.252777+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:34:14.252777+0000","last_undegraded":"2024-07-26T08:34:14.252777+0000","last_fullsized":"2024-07-26T08:34:14.252777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:45.489565+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:34:45.489565+0000","last_peered":"2024-07-26T08:34:45.489565+0000","last_clean":"2024-07-26T08:34:45.489565+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:34:45.489565+0000","last_undegraded":"2024-07-26T08:34:45.489565+0000","last_fullsized":"2024-07-26T08:34:45.489565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741925+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:36:12.741925+0000","last_peered":"2024-07-26T08:36:12.741925+0000","last_clean":"2024-07-26T08:36:12.741925+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:36:12.741925+0000","last_undegraded":"2024-07-26T08:36:12.741925+0000","last_fullsized":"2024-07-26T08:36:12.741925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":180,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:33.301933+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:36:33.301933+0000","last_peered":"2024-07-26T08:36:33.301933+0000","last_clean":"2024-07-26T08:36:33.301933+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:36:33.301933+0000","last_undegraded":"2024-07-26T08:36:33.301933+0000","last_fullsized":"2024-07-26T08:36:33.301933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952699+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:34:13.952699+0000","last_peered":"2024-07-26T08:34:13.952699+0000","last_clean":"2024-07-26T08:34:13.952699+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:34:13.952699+0000","last_undegraded":"2024-07-26T08:34:13.952699+0000","last_fullsized":"2024-07-26T08:34:13.952699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744004+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:36:12.744004+0000","last_peered":"2024-07-26T08:36:12.744004+0000","last_clean":"2024-07-26T08:36:12.744004+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:36:12.744004+0000","last_undegraded":"2024-07-26T08:36:12.744004+0000","last_fullsized":"2024-07-26T08:36:12.744004+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106683+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:34:32.106683+0000","last_peered":"2024-07-26T08:34:32.106683+0000","last_clean":"2024-07-26T08:34:32.106683+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:34:32.106683+0000","last_undegraded":"2024-07-26T08:34:32.106683+0000","last_fullsized":"2024-07-26T08:34:32.106683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967150+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:34:13.967150+0000","last_peered":"2024-07-26T08:34:13.967150+0000","last_clean":"2024-07-26T08:34:13.967150+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:34:13.967150+0000","last_undegraded":"2024-07-26T08:34:13.967150+0000","last_fullsized":"2024-07-26T08:34:13.967150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.726718+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:36:12.726718+0000","last_peered":"2024-07-26T08:36:12.726718+0000","last_clean":"2024-07-26T08:36:12.726718+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:36:12.726718+0000","last_undegraded":"2024-07-26T08:36:12.726718+0000","last_fullsized":"2024-07-26T08:36:12.726718+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096563+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:34:31.096563+0000","last_peered":"2024-07-26T08:34:31.096563+0000","last_clean":"2024-07-26T08:34:31.096563+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:34:31.096563+0000","last_undegraded":"2024-07-26T08:34:31.096563+0000","last_fullsized":"2024-07-26T08:34:31.096563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953156+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:34:13.953156+0000","last_peered":"2024-07-26T08:34:13.953156+0000","last_clean":"2024-07-26T08:34:13.953156+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:34:13.953156+0000","last_undegraded":"2024-07-26T08:34:13.953156+0000","last_fullsized":"2024-07-26T08:34:13.953156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.720115+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:36:12.720115+0000","last_peered":"2024-07-26T08:36:12.720115+0000","last_clean":"2024-07-26T08:36:12.720115+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:36:12.720115+0000","last_undegraded":"2024-07-26T08:36:12.720115+0000","last_fullsized":"2024-07-26T08:36:12.720115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106297+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:34:32.106297+0000","last_peered":"2024-07-26T08:34:32.106297+0000","last_clean":"2024-07-26T08:34:32.106297+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:34:32.106297+0000","last_undegraded":"2024-07-26T08:34:32.106297+0000","last_fullsized":"2024-07-26T08:34:32.106297+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953681+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:34:13.953681+0000","last_peered":"2024-07-26T08:34:13.953681+0000","last_clean":"2024-07-26T08:34:13.953681+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:34:13.953681+0000","last_undegraded":"2024-07-26T08:34:13.953681+0000","last_fullsized":"2024-07-26T08:34:13.953681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733613+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:36:12.733613+0000","last_peered":"2024-07-26T08:36:12.733613+0000","last_clean":"2024-07-26T08:36:12.733613+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:36:12.733613+0000","last_undegraded":"2024-07-26T08:36:12.733613+0000","last_fullsized":"2024-07-26T08:36:12.733613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108992+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:32.108992+0000","last_peered":"2024-07-26T08:34:32.108992+0000","last_clean":"2024-07-26T08:34:32.108992+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:32.108992+0000","last_undegraded":"2024-07-26T08:34:32.108992+0000","last_fullsized":"2024-07-26T08:34:32.108992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329087+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:34:14.329087+0000","last_peered":"2024-07-26T08:34:14.329087+0000","last_clean":"2024-07-26T08:34:14.329087+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:34:14.329087+0000","last_undegraded":"2024-07-26T08:34:14.329087+0000","last_fullsized":"2024-07-26T08:34:14.329087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.731525+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:36:12.731525+0000","last_peered":"2024-07-26T08:36:12.731525+0000","last_clean":"2024-07-26T08:36:12.731525+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:36:12.731525+0000","last_undegraded":"2024-07-26T08:36:12.731525+0000","last_fullsized":"2024-07-26T08:36:12.731525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:32.436856+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:36:32.436856+0000","last_peered":"2024-07-26T08:36:32.436856+0000","last_clean":"2024-07-26T08:36:32.436856+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:36:32.436856+0000","last_undegraded":"2024-07-26T08:36:32.436856+0000","last_fullsized":"2024-07-26T08:36:32.436856+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.957436+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:34:13.957436+0000","last_peered":"2024-07-26T08:34:13.957436+0000","last_clean":"2024-07-26T08:34:13.957436+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:34:13.957436+0000","last_undegraded":"2024-07-26T08:34:13.957436+0000","last_fullsized":"2024-07-26T08:34:13.957436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733851+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:36:12.733851+0000","last_peered":"2024-07-26T08:36:12.733851+0000","last_clean":"2024-07-26T08:36:12.733851+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:36:12.733851+0000","last_undegraded":"2024-07-26T08:36:12.733851+0000","last_fullsized":"2024-07-26T08:36:12.733851+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106170+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:34:32.106170+0000","last_peered":"2024-07-26T08:34:32.106170+0000","last_clean":"2024-07-26T08:34:32.106170+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:34:32.106170+0000","last_undegraded":"2024-07-26T08:34:32.106170+0000","last_fullsized":"2024-07-26T08:34:32.106170+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961991+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:34:13.961991+0000","last_peered":"2024-07-26T08:34:13.961991+0000","last_clean":"2024-07-26T08:34:13.961991+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:34:13.961991+0000","last_undegraded":"2024-07-26T08:34:13.961991+0000","last_fullsized":"2024-07-26T08:34:13.961991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740754+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:36:12.740754+0000","last_peered":"2024-07-26T08:36:12.740754+0000","last_clean":"2024-07-26T08:36:12.740754+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:36:12.740754+0000","last_undegraded":"2024-07-26T08:36:12.740754+0000","last_fullsized":"2024-07-26T08:36:12.740754+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108272+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:34:32.108272+0000","last_peered":"2024-07-26T08:34:32.108272+0000","last_clean":"2024-07-26T08:34:32.108272+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:34:32.108272+0000","last_undegraded":"2024-07-26T08:34:32.108272+0000","last_fullsized":"2024-07-26T08:34:32.108272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953564+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:34:13.953564+0000","last_peered":"2024-07-26T08:34:13.953564+0000","last_clean":"2024-07-26T08:34:13.953564+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:34:13.953564+0000","last_undegraded":"2024-07-26T08:34:13.953564+0000","last_fullsized":"2024-07-26T08:34:13.953564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.710145+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:36:12.710145+0000","last_peered":"2024-07-26T08:36:12.710145+0000","last_clean":"2024-07-26T08:36:12.710145+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:36:12.710145+0000","last_undegraded":"2024-07-26T08:36:12.710145+0000","last_fullsized":"2024-07-26T08:36:12.710145+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'41","reported_seq":339,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:32.299668+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:36:32.299668+0000","last_peered":"2024-07-26T08:36:32.299668+0000","last_clean":"2024-07-26T08:36:32.299668+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:36:32.299668+0000","last_undegraded":"2024-07-26T08:36:32.299668+0000","last_fullsized":"2024-07-26T08:36:32.299668+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67283,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":192,"num_write":43,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955409+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:34:13.955409+0000","last_peered":"2024-07-26T08:34:13.955409+0000","last_clean":"2024-07-26T08:34:13.955409+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:34:13.955409+0000","last_undegraded":"2024-07-26T08:34:13.955409+0000","last_fullsized":"2024-07-26T08:34:13.955409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738435+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:36:12.738435+0000","last_peered":"2024-07-26T08:36:12.738435+0000","last_clean":"2024-07-26T08:36:12.738435+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:36:12.738435+0000","last_undegraded":"2024-07-26T08:36:12.738435+0000","last_fullsized":"2024-07-26T08:36:12.738435+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.111302+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:34:32.111302+0000","last_peered":"2024-07-26T08:34:32.111302+0000","last_clean":"2024-07-26T08:34:32.111302+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:34:32.111302+0000","last_undegraded":"2024-07-26T08:34:32.111302+0000","last_fullsized":"2024-07-26T08:34:32.111302+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.969909+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:34:13.969909+0000","last_peered":"2024-07-26T08:34:13.969909+0000","last_clean":"2024-07-26T08:34:13.969909+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:34:13.969909+0000","last_undegraded":"2024-07-26T08:34:13.969909+0000","last_fullsized":"2024-07-26T08:34:13.969909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735248+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:36:12.735248+0000","last_peered":"2024-07-26T08:36:12.735248+0000","last_clean":"2024-07-26T08:36:12.735248+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:36:12.735248+0000","last_undegraded":"2024-07-26T08:36:12.735248+0000","last_fullsized":"2024-07-26T08:36:12.735248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097187+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:34:31.097187+0000","last_peered":"2024-07-26T08:34:31.097187+0000","last_clean":"2024-07-26T08:34:31.097187+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:34:31.097187+0000","last_undegraded":"2024-07-26T08:34:31.097187+0000","last_fullsized":"2024-07-26T08:34:31.097187+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954260+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:34:13.954260+0000","last_peered":"2024-07-26T08:34:13.954260+0000","last_clean":"2024-07-26T08:34:13.954260+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:34:13.954260+0000","last_undegraded":"2024-07-26T08:34:13.954260+0000","last_fullsized":"2024-07-26T08:34:13.954260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741011+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:36:12.741011+0000","last_peered":"2024-07-26T08:36:12.741011+0000","last_clean":"2024-07-26T08:36:12.741011+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:36:12.741011+0000","last_undegraded":"2024-07-26T08:36:12.741011+0000","last_fullsized":"2024-07-26T08:36:12.741011+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":293,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:32.299634+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:36:32.299634+0000","last_peered":"2024-07-26T08:36:32.299634+0000","last_clean":"2024-07-26T08:36:32.299634+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:36:32.299634+0000","last_undegraded":"2024-07-26T08:36:32.299634+0000","last_fullsized":"2024-07-26T08:36:32.299634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591838+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:34:14.591838+0000","last_peered":"2024-07-26T08:34:14.591838+0000","last_clean":"2024-07-26T08:34:14.591838+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:34:14.591838+0000","last_undegraded":"2024-07-26T08:34:14.591838+0000","last_fullsized":"2024-07-26T08:34:14.591838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967620+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:34:13.967620+0000","last_peered":"2024-07-26T08:34:13.967620+0000","last_clean":"2024-07-26T08:34:13.967620+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:34:13.967620+0000","last_undegraded":"2024-07-26T08:34:13.967620+0000","last_fullsized":"2024-07-26T08:34:13.967620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746247+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:36:12.746247+0000","last_peered":"2024-07-26T08:36:12.746247+0000","last_clean":"2024-07-26T08:36:12.746247+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:36:12.746247+0000","last_undegraded":"2024-07-26T08:36:12.746247+0000","last_fullsized":"2024-07-26T08:36:12.746247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.111667+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:34:32.111667+0000","last_peered":"2024-07-26T08:34:32.111667+0000","last_clean":"2024-07-26T08:34:32.111667+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:34:32.111667+0000","last_undegraded":"2024-07-26T08:34:32.111667+0000","last_fullsized":"2024-07-26T08:34:32.111667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962233+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:34:13.962233+0000","last_peered":"2024-07-26T08:34:13.962233+0000","last_clean":"2024-07-26T08:34:13.962233+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:34:13.962233+0000","last_undegraded":"2024-07-26T08:34:13.962233+0000","last_fullsized":"2024-07-26T08:34:13.962233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741760+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:36:12.741760+0000","last_peered":"2024-07-26T08:36:12.741760+0000","last_clean":"2024-07-26T08:36:12.741760+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:36:12.741760+0000","last_undegraded":"2024-07-26T08:36:12.741760+0000","last_fullsized":"2024-07-26T08:36:12.741760+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096093+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:34:31.096093+0000","last_peered":"2024-07-26T08:34:31.096093+0000","last_clean":"2024-07-26T08:34:31.096093+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:34:31.096093+0000","last_undegraded":"2024-07-26T08:34:31.096093+0000","last_fullsized":"2024-07-26T08:34:31.096093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.091831+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:34:30.091831+0000","last_peered":"2024-07-26T08:34:30.091831+0000","last_clean":"2024-07-26T08:34:30.091831+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:34:30.091831+0000","last_undegraded":"2024-07-26T08:34:30.091831+0000","last_fullsized":"2024-07-26T08:34:30.091831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740795+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:36:12.740795+0000","last_peered":"2024-07-26T08:36:12.740795+0000","last_clean":"2024-07-26T08:36:12.740795+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:36:12.740795+0000","last_undegraded":"2024-07-26T08:36:12.740795+0000","last_fullsized":"2024-07-26T08:36:12.740795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.964602+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:34:13.964602+0000","last_peered":"2024-07-26T08:34:13.964602+0000","last_clean":"2024-07-26T08:34:13.964602+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:34:13.964602+0000","last_undegraded":"2024-07-26T08:34:13.964602+0000","last_fullsized":"2024-07-26T08:34:13.964602+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":178,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:35.304900+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:36:35.304900+0000","last_peered":"2024-07-26T08:36:35.304900+0000","last_clean":"2024-07-26T08:36:35.304900+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:36:35.304900+0000","last_undegraded":"2024-07-26T08:36:35.304900+0000","last_fullsized":"2024-07-26T08:36:35.304900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727268+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:36:12.727268+0000","last_peered":"2024-07-26T08:36:12.727268+0000","last_clean":"2024-07-26T08:36:12.727268+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:36:12.727268+0000","last_undegraded":"2024-07-26T08:36:12.727268+0000","last_fullsized":"2024-07-26T08:36:12.727268+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965555+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:34:13.965555+0000","last_peered":"2024-07-26T08:34:13.965555+0000","last_clean":"2024-07-26T08:34:13.965555+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:34:13.965555+0000","last_undegraded":"2024-07-26T08:34:13.965555+0000","last_fullsized":"2024-07-26T08:34:13.965555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723604+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:36:12.723604+0000","last_peered":"2024-07-26T08:36:12.723604+0000","last_clean":"2024-07-26T08:36:12.723604+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:36:12.723604+0000","last_undegraded":"2024-07-26T08:36:12.723604+0000","last_fullsized":"2024-07-26T08:36:12.723604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721726+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:36:12.721726+0000","last_peered":"2024-07-26T08:36:12.721726+0000","last_clean":"2024-07-26T08:36:12.721726+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:36:12.721726+0000","last_undegraded":"2024-07-26T08:36:12.721726+0000","last_fullsized":"2024-07-26T08:36:12.721726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737335+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:36:12.737335+0000","last_peered":"2024-07-26T08:36:12.737335+0000","last_clean":"2024-07-26T08:36:12.737335+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:36:12.737335+0000","last_undegraded":"2024-07-26T08:36:12.737335+0000","last_fullsized":"2024-07-26T08:36:12.737335+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729035+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:36:12.729035+0000","last_peered":"2024-07-26T08:36:12.729035+0000","last_clean":"2024-07-26T08:36:12.729035+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:36:12.729035+0000","last_undegraded":"2024-07-26T08:36:12.729035+0000","last_fullsized":"2024-07-26T08:36:12.729035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737901+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:36:12.737901+0000","last_peered":"2024-07-26T08:36:12.737901+0000","last_clean":"2024-07-26T08:36:12.737901+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:36:12.737901+0000","last_undegraded":"2024-07-26T08:36:12.737901+0000","last_fullsized":"2024-07-26T08:36:12.737901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718361+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:36:12.718361+0000","last_peered":"2024-07-26T08:36:12.718361+0000","last_clean":"2024-07-26T08:36:12.718361+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:36:12.718361+0000","last_undegraded":"2024-07-26T08:36:12.718361+0000","last_fullsized":"2024-07-26T08:36:12.718361+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.732615+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:36:12.732615+0000","last_peered":"2024-07-26T08:36:12.732615+0000","last_clean":"2024-07-26T08:36:12.732615+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:36:12.732615+0000","last_undegraded":"2024-07-26T08:36:12.732615+0000","last_fullsized":"2024-07-26T08:36:12.732615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743216+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:36:12.743216+0000","last_peered":"2024-07-26T08:36:12.743216+0000","last_clean":"2024-07-26T08:36:12.743216+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:36:12.743216+0000","last_undegraded":"2024-07-26T08:36:12.743216+0000","last_fullsized":"2024-07-26T08:36:12.743216+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745514+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:36:12.745514+0000","last_peered":"2024-07-26T08:36:12.745514+0000","last_clean":"2024-07-26T08:36:12.745514+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:36:12.745514+0000","last_undegraded":"2024-07-26T08:36:12.745514+0000","last_fullsized":"2024-07-26T08:36:12.745514+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742101+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:36:12.742101+0000","last_peered":"2024-07-26T08:36:12.742101+0000","last_clean":"2024-07-26T08:36:12.742101+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:36:12.742101+0000","last_undegraded":"2024-07-26T08:36:12.742101+0000","last_fullsized":"2024-07-26T08:36:12.742101+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746400+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:36:12.746400+0000","last_peered":"2024-07-26T08:36:12.746400+0000","last_clean":"2024-07-26T08:36:12.746400+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:36:12.746400+0000","last_undegraded":"2024-07-26T08:36:12.746400+0000","last_fullsized":"2024-07-26T08:36:12.746400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742305+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:36:12.742305+0000","last_peered":"2024-07-26T08:36:12.742305+0000","last_clean":"2024-07-26T08:36:12.742305+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:36:12.742305+0000","last_undegraded":"2024-07-26T08:36:12.742305+0000","last_fullsized":"2024-07-26T08:36:12.742305+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723067+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:36:12.723067+0000","last_peered":"2024-07-26T08:36:12.723067+0000","last_clean":"2024-07-26T08:36:12.723067+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:36:12.723067+0000","last_undegraded":"2024-07-26T08:36:12.723067+0000","last_fullsized":"2024-07-26T08:36:12.723067+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735951+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:36:12.735951+0000","last_peered":"2024-07-26T08:36:12.735951+0000","last_clean":"2024-07-26T08:36:12.735951+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:36:12.735951+0000","last_undegraded":"2024-07-26T08:36:12.735951+0000","last_fullsized":"2024-07-26T08:36:12.735951+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733519+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:36:12.733519+0000","last_peered":"2024-07-26T08:36:12.733519+0000","last_clean":"2024-07-26T08:36:12.733519+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:36:12.733519+0000","last_undegraded":"2024-07-26T08:36:12.733519+0000","last_fullsized":"2024-07-26T08:36:12.733519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740709+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:36:12.740709+0000","last_peered":"2024-07-26T08:36:12.740709+0000","last_clean":"2024-07-26T08:36:12.740709+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:36:12.740709+0000","last_undegraded":"2024-07-26T08:36:12.740709+0000","last_fullsized":"2024-07-26T08:36:12.740709+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743025+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:36:12.743025+0000","last_peered":"2024-07-26T08:36:12.743025+0000","last_clean":"2024-07-26T08:36:12.743025+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:36:12.743025+0000","last_undegraded":"2024-07-26T08:36:12.743025+0000","last_fullsized":"2024-07-26T08:36:12.743025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735949+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:36:12.735949+0000","last_peered":"2024-07-26T08:36:12.735949+0000","last_clean":"2024-07-26T08:36:12.735949+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:36:12.735949+0000","last_undegraded":"2024-07-26T08:36:12.735949+0000","last_fullsized":"2024-07-26T08:36:12.735949+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736238+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:36:12.736238+0000","last_peered":"2024-07-26T08:36:12.736238+0000","last_clean":"2024-07-26T08:36:12.736238+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:36:12.736238+0000","last_undegraded":"2024-07-26T08:36:12.736238+0000","last_fullsized":"2024-07-26T08:36:12.736238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728244+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:36:12.728244+0000","last_peered":"2024-07-26T08:36:12.728244+0000","last_clean":"2024-07-26T08:36:12.728244+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:36:12.728244+0000","last_undegraded":"2024-07-26T08:36:12.728244+0000","last_fullsized":"2024-07-26T08:36:12.728244+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718869+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:36:12.718869+0000","last_peered":"2024-07-26T08:36:12.718869+0000","last_clean":"2024-07-26T08:36:12.718869+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:36:12.718869+0000","last_undegraded":"2024-07-26T08:36:12.718869+0000","last_fullsized":"2024-07-26T08:36:12.718869+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742060+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:36:12.742060+0000","last_peered":"2024-07-26T08:36:12.742060+0000","last_clean":"2024-07-26T08:36:12.742060+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:36:12.742060+0000","last_undegraded":"2024-07-26T08:36:12.742060+0000","last_fullsized":"2024-07-26T08:36:12.742060+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721655+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:36:12.721655+0000","last_peered":"2024-07-26T08:36:12.721655+0000","last_clean":"2024-07-26T08:36:12.721655+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:36:12.721655+0000","last_undegraded":"2024-07-26T08:36:12.721655+0000","last_fullsized":"2024-07-26T08:36:12.721655+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740155+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:36:12.740155+0000","last_peered":"2024-07-26T08:36:12.740155+0000","last_clean":"2024-07-26T08:36:12.740155+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:36:12.740155+0000","last_undegraded":"2024-07-26T08:36:12.740155+0000","last_fullsized":"2024-07-26T08:36:12.740155+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744881+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:36:12.744881+0000","last_peered":"2024-07-26T08:36:12.744881+0000","last_clean":"2024-07-26T08:36:12.744881+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:36:12.744881+0000","last_undegraded":"2024-07-26T08:36:12.744881+0000","last_fullsized":"2024-07-26T08:36:12.744881+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739021+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:36:12.739021+0000","last_peered":"2024-07-26T08:36:12.739021+0000","last_clean":"2024-07-26T08:36:12.739021+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:36:12.739021+0000","last_undegraded":"2024-07-26T08:36:12.739021+0000","last_fullsized":"2024-07-26T08:36:12.739021+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728498+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:36:12.728498+0000","last_peered":"2024-07-26T08:36:12.728498+0000","last_clean":"2024-07-26T08:36:12.728498+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:36:12.728498+0000","last_undegraded":"2024-07-26T08:36:12.728498+0000","last_fullsized":"2024-07-26T08:36:12.728498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743776+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:36:12.743776+0000","last_peered":"2024-07-26T08:36:12.743776+0000","last_clean":"2024-07-26T08:36:12.743776+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:36:12.743776+0000","last_undegraded":"2024-07-26T08:36:12.743776+0000","last_fullsized":"2024-07-26T08:36:12.743776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719531+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:36:12.719531+0000","last_peered":"2024-07-26T08:36:12.719531+0000","last_clean":"2024-07-26T08:36:12.719531+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:36:12.719531+0000","last_undegraded":"2024-07-26T08:36:12.719531+0000","last_fullsized":"2024-07-26T08:36:12.719531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735860+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:36:12.735860+0000","last_peered":"2024-07-26T08:36:12.735860+0000","last_clean":"2024-07-26T08:36:12.735860+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:36:12.735860+0000","last_undegraded":"2024-07-26T08:36:12.735860+0000","last_fullsized":"2024-07-26T08:36:12.735860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.725277+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:36:12.725277+0000","last_peered":"2024-07-26T08:36:12.725277+0000","last_clean":"2024-07-26T08:36:12.725277+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:36:12.725277+0000","last_undegraded":"2024-07-26T08:36:12.725277+0000","last_fullsized":"2024-07-26T08:36:12.725277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740147+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:36:12.740147+0000","last_peered":"2024-07-26T08:36:12.740147+0000","last_clean":"2024-07-26T08:36:12.740147+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:36:12.740147+0000","last_undegraded":"2024-07-26T08:36:12.740147+0000","last_fullsized":"2024-07-26T08:36:12.740147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74769,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1268,"num_read_kb":717,"num_write":262,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":224307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645373,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540314,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841600,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1198397,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005126,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397586,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136616,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561110,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757341,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920865,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539710,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378735,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":604169,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542258,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1085440,"data_stored":603417,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":537092,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:36:38.831 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T08:36:39.366 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:39.378 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:39 smithi044 ceph-mon[32404]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:36:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:39 smithi116 ceph-mon[37097]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:36:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:39 smithi171 ceph-mon[37789]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:36:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:40 smithi044 ceph-mon[32404]: pgmap v422: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:40 smithi116 ceph-mon[37097]: pgmap v422: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:40 smithi171 ceph-mon[37789]: pgmap v422: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 -- 172.21.15.44:0/603696097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a281055d0 msgr2=0x7f4a281059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 --2- 172.21.15.44:0/603696097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a281055d0 0x7f4a281059e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009860 tx=0x7f4a1802a340 comp rx=0 tx=0).stop 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 -- 172.21.15.44:0/603696097 shutdown_connections 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 --2- 172.21.15.44:0/603696097 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a2810af10 0x7f4a2810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 --2- 172.21.15.44:0/603696097 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a2810b860 0x7f4a28112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 --2- 172.21.15.44:0/603696097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a281055d0 0x7f4a281059e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.711+0000 7f4a27e00640 1 -- 172.21.15.44:0/603696097 >> 172.21.15.44:0/603696097 conn(0x7f4a28101040 msgr2=0x7f4a28103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:41.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.712+0000 7f4a27e00640 1 -- 172.21.15.44:0/603696097 shutdown_connections 2024-07-26T08:36:41.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.712+0000 7f4a27e00640 1 -- 172.21.15.44:0/603696097 wait complete. 2024-07-26T08:36:41.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.713+0000 7f4a27e00640 1 Processor -- start 2024-07-26T08:36:41.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 -- start start 2024-07-26T08:36:41.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 0x7f4a2819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:41.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2810af10 0x7f4a2819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:41.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a2810b860 0x7f4a281a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:41.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4a28115510 con 0x7f4a2810b860 2024-07-26T08:36:41.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a28115390 con 0x7f4a2810af10 2024-07-26T08:36:41.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.714+0000 7f4a27e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4a28115690 con 0x7f4a281055d0 2024-07-26T08:36:41.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.715+0000 7f4a26a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 0x7f4a2819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:41.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.715+0000 7f4a26a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 0x7f4a2819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46578/0 (socket says 172.21.15.44:46578) 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.715+0000 7f4a26a00640 1 -- 172.21.15.44:0/2798374186 learned_addr learned my addr 172.21.15.44:0/2798374186 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.715+0000 7f4a1fe00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2810af10 0x7f4a2819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.715+0000 7f4a27400640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a2810b860 0x7f4a281a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a26a00640 1 -- 172.21.15.44:0/2798374186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2810af10 msgr2=0x7f4a2819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a26a00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2810af10 0x7f4a2819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a26a00640 1 -- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a2810b860 msgr2=0x7f4a281a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:41.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a26a00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a2810b860 0x7f4a281a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a26a00640 1 -- 172.21.15.44:0/2798374186 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a281a1eb0 con 0x7f4a281055d0 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a1fe00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2810af10 0x7f4a2819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a26a00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 0x7f4a2819bf40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4a18035040 tx=0x7f4a18032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18037040 con 0x7f4a281055d0 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a281a2140 con 0x7f4a281055d0 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a18041070 con 0x7f4a281055d0 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.716+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18003850 con 0x7f4a281055d0 2024-07-26T08:36:41.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.717+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a281a8900 con 0x7f4a281055d0 2024-07-26T08:36:41.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.718+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a28106850 con 0x7f4a281055d0 2024-07-26T08:36:41.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.720+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f4a18004050 con 0x7f4a281055d0 2024-07-26T08:36:41.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.722+0000 7f4a1f400640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4a0c0b3ca0 0x7f4a0c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:41.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.722+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4a18104540 con 0x7f4a281055d0 2024-07-26T08:36:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.722+0000 7f4a1fe00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4a0c0b3ca0 0x7f4a0c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:41.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.723+0000 7f4a1fe00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4a0c0b3ca0 0x7f4a0c0b61b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a2810aa40 tx=0x7f4a10005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:41.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.724+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4a180cb680 con 0x7f4a281055d0 2024-07-26T08:36:41.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.924+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4a28063230 con 0x7f4a0c0b3ca0 2024-07-26T08:36:41.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.940+0000 7f4a1f400640 1 -- 172.21.15.44:0/2798374186 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340804 (secure 0 0 0) 0x7f4a28063230 con 0x7f4a0c0b3ca0 2024-07-26T08:36:41.949 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:36:41.951 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:36:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4a0c0b3ca0 msgr2=0x7f4a0c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4a0c0b3ca0 0x7f4a0c0b61b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a2810aa40 tx=0x7f4a10005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 msgr2=0x7f4a2819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 0x7f4a2819bf40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4a18035040 tx=0x7f4a18032d50 comp rx=0 tx=0).stop 2024-07-26T08:36:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 shutdown_connections 2024-07-26T08:36:41.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4a0c0b3ca0 0x7f4a0c0b61b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4a2810b860 0x7f4a281a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a2810af10 0x7f4a2819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 --2- 172.21.15.44:0/2798374186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a281055d0 0x7f4a2819bf40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.949+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 >> 172.21.15.44:0/2798374186 conn(0x7f4a28101040 msgr2=0x7f4a28103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.950+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 shutdown_connections 2024-07-26T08:36:41.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:41.950+0000 7f4a27e00640 1 -- 172.21.15.44:0/2798374186 wait complete. 2024-07-26T08:36:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:42 smithi044 ceph-mon[32404]: pgmap v423: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:42 smithi116 ceph-mon[37097]: pgmap v423: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:42 smithi171 ceph-mon[37789]: pgmap v423: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:42.964 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-07-26T08:36:40.853015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665371,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1384,"num_read_kb":801,"num_write":387,"num_write_kb":2308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":820,"ondisk_log_size":820,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365728,"kb_used_data":12416,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576096,"statfs":{"total":1151940427776,"available":1151565922304,"internally_reserved":0,"allocated":12713984,"data_stored":8440763,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007471"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":184,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:34.407675+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:36:34.407675+0000","last_peered":"2024-07-26T08:36:34.407675+0000","last_clean":"2024-07-26T08:36:34.407675+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:36:34.407675+0000","last_undegraded":"2024-07-26T08:36:34.407675+0000","last_fullsized":"2024-07-26T08:36:34.407675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":125,"num_read_kb":125,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746213+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:36:12.746213+0000","last_peered":"2024-07-26T08:36:12.746213+0000","last_clean":"2024-07-26T08:36:12.746213+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:36:12.746213+0000","last_undegraded":"2024-07-26T08:36:12.746213+0000","last_fullsized":"2024-07-26T08:36:12.746213+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965699+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:34:13.965699+0000","last_peered":"2024-07-26T08:34:13.965699+0000","last_clean":"2024-07-26T08:34:13.965699+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:34:13.965699+0000","last_undegraded":"2024-07-26T08:34:13.965699+0000","last_fullsized":"2024-07-26T08:34:13.965699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:29.071793+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:29.071793+0000","last_peered":"2024-07-26T08:34:29.071793+0000","last_clean":"2024-07-26T08:34:29.071793+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:29.071793+0000","last_undegraded":"2024-07-26T08:34:29.071793+0000","last_fullsized":"2024-07-26T08:34:29.071793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740925+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:36:12.740925+0000","last_peered":"2024-07-26T08:36:12.740925+0000","last_clean":"2024-07-26T08:36:12.740925+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:36:12.740925+0000","last_undegraded":"2024-07-26T08:36:12.740925+0000","last_fullsized":"2024-07-26T08:36:12.740925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.593535+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:34:14.593535+0000","last_peered":"2024-07-26T08:34:14.593535+0000","last_clean":"2024-07-26T08:34:14.593535+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:34:14.593535+0000","last_undegraded":"2024-07-26T08:34:14.593535+0000","last_fullsized":"2024-07-26T08:34:14.593535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:33.090130+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:34:33.090130+0000","last_peered":"2024-07-26T08:34:33.090130+0000","last_clean":"2024-07-26T08:34:33.090130+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:34:33.090130+0000","last_undegraded":"2024-07-26T08:34:33.090130+0000","last_fullsized":"2024-07-26T08:34:33.090130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741267+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:36:12.741267+0000","last_peered":"2024-07-26T08:36:12.741267+0000","last_clean":"2024-07-26T08:36:12.741267+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:36:12.741267+0000","last_undegraded":"2024-07-26T08:36:12.741267+0000","last_fullsized":"2024-07-26T08:36:12.741267+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955356+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:34:13.955356+0000","last_peered":"2024-07-26T08:34:13.955356+0000","last_clean":"2024-07-26T08:34:13.955356+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:34:13.955356+0000","last_undegraded":"2024-07-26T08:34:13.955356+0000","last_fullsized":"2024-07-26T08:34:13.955356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.113096+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:34:32.113096+0000","last_peered":"2024-07-26T08:34:32.113096+0000","last_clean":"2024-07-26T08:34:32.113096+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:34:32.113096+0000","last_undegraded":"2024-07-26T08:34:32.113096+0000","last_fullsized":"2024-07-26T08:34:32.113096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734672+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:36:12.734672+0000","last_peered":"2024-07-26T08:36:12.734672+0000","last_clean":"2024-07-26T08:36:12.734672+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:36:12.734672+0000","last_undegraded":"2024-07-26T08:36:12.734672+0000","last_fullsized":"2024-07-26T08:36:12.734672+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954873+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:34:13.954873+0000","last_peered":"2024-07-26T08:34:13.954873+0000","last_clean":"2024-07-26T08:34:13.954873+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:34:13.954873+0000","last_undegraded":"2024-07-26T08:34:13.954873+0000","last_fullsized":"2024-07-26T08:34:13.954873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742090+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:36:12.742090+0000","last_peered":"2024-07-26T08:36:12.742090+0000","last_clean":"2024-07-26T08:36:12.742090+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:36:12.742090+0000","last_undegraded":"2024-07-26T08:36:12.742090+0000","last_fullsized":"2024-07-26T08:36:12.742090+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106175+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:34:32.106175+0000","last_peered":"2024-07-26T08:34:32.106175+0000","last_clean":"2024-07-26T08:34:32.106175+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:34:32.106175+0000","last_undegraded":"2024-07-26T08:34:32.106175+0000","last_fullsized":"2024-07-26T08:34:32.106175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955043+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.955043+0000","last_peered":"2024-07-26T08:34:13.955043+0000","last_clean":"2024-07-26T08:34:13.955043+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:34:13.955043+0000","last_undegraded":"2024-07-26T08:34:13.955043+0000","last_fullsized":"2024-07-26T08:34:13.955043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743210+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:36:12.743210+0000","last_peered":"2024-07-26T08:36:12.743210+0000","last_clean":"2024-07-26T08:36:12.743210+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:36:12.743210+0000","last_undegraded":"2024-07-26T08:36:12.743210+0000","last_fullsized":"2024-07-26T08:36:12.743210+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.079193+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:34:31.079193+0000","last_peered":"2024-07-26T08:34:31.079193+0000","last_clean":"2024-07-26T08:34:31.079193+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:34:31.079193+0000","last_undegraded":"2024-07-26T08:34:31.079193+0000","last_fullsized":"2024-07-26T08:34:31.079193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:53.482447+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:34:53.482447+0000","last_peered":"2024-07-26T08:34:53.482447+0000","last_clean":"2024-07-26T08:34:53.482447+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:34:53.482447+0000","last_undegraded":"2024-07-26T08:34:53.482447+0000","last_fullsized":"2024-07-26T08:34:53.482447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740202+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:36:12.740202+0000","last_peered":"2024-07-26T08:36:12.740202+0000","last_clean":"2024-07-26T08:36:12.740202+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:36:12.740202+0000","last_undegraded":"2024-07-26T08:36:12.740202+0000","last_fullsized":"2024-07-26T08:36:12.740202+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097156+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:34:31.097156+0000","last_peered":"2024-07-26T08:34:31.097156+0000","last_clean":"2024-07-26T08:34:31.097156+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:34:31.097156+0000","last_undegraded":"2024-07-26T08:34:31.097156+0000","last_fullsized":"2024-07-26T08:34:31.097156+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.596626+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:34:14.596626+0000","last_peered":"2024-07-26T08:34:14.596626+0000","last_clean":"2024-07-26T08:34:14.596626+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:34:14.596626+0000","last_undegraded":"2024-07-26T08:34:14.596626+0000","last_fullsized":"2024-07-26T08:34:14.596626+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.693280+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:36:12.693280+0000","last_peered":"2024-07-26T08:36:12.693280+0000","last_clean":"2024-07-26T08:36:12.693280+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:36:12.693280+0000","last_undegraded":"2024-07-26T08:36:12.693280+0000","last_fullsized":"2024-07-26T08:36:12.693280+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:28.060952+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:34:28.060952+0000","last_peered":"2024-07-26T08:34:28.060952+0000","last_clean":"2024-07-26T08:34:28.060952+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:34:28.060952+0000","last_undegraded":"2024-07-26T08:34:28.060952+0000","last_fullsized":"2024-07-26T08:34:28.060952+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955242+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:34:13.955242+0000","last_peered":"2024-07-26T08:34:13.955242+0000","last_clean":"2024-07-26T08:34:13.955242+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:34:13.955242+0000","last_undegraded":"2024-07-26T08:34:13.955242+0000","last_fullsized":"2024-07-26T08:34:13.955242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744261+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:36:12.744261+0000","last_peered":"2024-07-26T08:36:12.744261+0000","last_clean":"2024-07-26T08:36:12.744261+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:36:12.744261+0000","last_undegraded":"2024-07-26T08:36:12.744261+0000","last_fullsized":"2024-07-26T08:36:12.744261+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108189+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:34:32.108189+0000","last_peered":"2024-07-26T08:34:32.108189+0000","last_clean":"2024-07-26T08:34:32.108189+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:34:32.108189+0000","last_undegraded":"2024-07-26T08:34:32.108189+0000","last_fullsized":"2024-07-26T08:34:32.108189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.966025+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:34:13.966025+0000","last_peered":"2024-07-26T08:34:13.966025+0000","last_clean":"2024-07-26T08:34:13.966025+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:34:13.966025+0000","last_undegraded":"2024-07-26T08:34:13.966025+0000","last_fullsized":"2024-07-26T08:34:13.966025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728411+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:36:12.728411+0000","last_peered":"2024-07-26T08:36:12.728411+0000","last_clean":"2024-07-26T08:36:12.728411+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:36:12.728411+0000","last_undegraded":"2024-07-26T08:36:12.728411+0000","last_fullsized":"2024-07-26T08:36:12.728411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'17","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:11.341058+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:35:11.341058+0000","last_peered":"2024-07-26T08:35:11.341058+0000","last_clean":"2024-07-26T08:35:11.341058+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:35:11.341058+0000","last_undegraded":"2024-07-26T08:35:11.341058+0000","last_fullsized":"2024-07-26T08:35:11.341058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591736+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:34:14.591736+0000","last_peered":"2024-07-26T08:34:14.591736+0000","last_clean":"2024-07-26T08:34:14.591736+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:34:14.591736+0000","last_undegraded":"2024-07-26T08:34:14.591736+0000","last_fullsized":"2024-07-26T08:34:14.591736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733504+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:36:12.733504+0000","last_peered":"2024-07-26T08:36:12.733504+0000","last_clean":"2024-07-26T08:36:12.733504+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:36:12.733504+0000","last_undegraded":"2024-07-26T08:36:12.733504+0000","last_fullsized":"2024-07-26T08:36:12.733504+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108436+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:34:32.108436+0000","last_peered":"2024-07-26T08:34:32.108436+0000","last_clean":"2024-07-26T08:34:32.108436+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:34:32.108436+0000","last_undegraded":"2024-07-26T08:34:32.108436+0000","last_fullsized":"2024-07-26T08:34:32.108436+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961966+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:34:13.961966+0000","last_peered":"2024-07-26T08:34:13.961966+0000","last_clean":"2024-07-26T08:34:13.961966+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:34:13.961966+0000","last_undegraded":"2024-07-26T08:34:13.961966+0000","last_fullsized":"2024-07-26T08:34:13.961966+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743910+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:36:12.743910+0000","last_peered":"2024-07-26T08:36:12.743910+0000","last_clean":"2024-07-26T08:36:12.743910+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:36:12.743910+0000","last_undegraded":"2024-07-26T08:36:12.743910+0000","last_fullsized":"2024-07-26T08:36:12.743910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.095298+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:34:30.095298+0000","last_peered":"2024-07-26T08:34:30.095298+0000","last_clean":"2024-07-26T08:34:30.095298+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:34:30.095298+0000","last_undegraded":"2024-07-26T08:34:30.095298+0000","last_fullsized":"2024-07-26T08:34:30.095298+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965933+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:34:13.965933+0000","last_peered":"2024-07-26T08:34:13.965933+0000","last_clean":"2024-07-26T08:34:13.965933+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:34:13.965933+0000","last_undegraded":"2024-07-26T08:34:13.965933+0000","last_fullsized":"2024-07-26T08:34:13.965933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739889+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:36:12.739889+0000","last_peered":"2024-07-26T08:36:12.739889+0000","last_clean":"2024-07-26T08:36:12.739889+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:36:12.739889+0000","last_undegraded":"2024-07-26T08:36:12.739889+0000","last_fullsized":"2024-07-26T08:36:12.739889+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":178,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:36.413133+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:36:36.413133+0000","last_peered":"2024-07-26T08:36:36.413133+0000","last_clean":"2024-07-26T08:36:36.413133+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:36:36.413133+0000","last_undegraded":"2024-07-26T08:36:36.413133+0000","last_fullsized":"2024-07-26T08:36:36.413133+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.328912+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:34:14.328912+0000","last_peered":"2024-07-26T08:34:14.328912+0000","last_clean":"2024-07-26T08:34:14.328912+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:34:14.328912+0000","last_undegraded":"2024-07-26T08:34:14.328912+0000","last_fullsized":"2024-07-26T08:34:14.328912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729525+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:36:12.729525+0000","last_peered":"2024-07-26T08:36:12.729525+0000","last_clean":"2024-07-26T08:36:12.729525+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:36:12.729525+0000","last_undegraded":"2024-07-26T08:36:12.729525+0000","last_fullsized":"2024-07-26T08:36:12.729525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":178,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:35.445196+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:36:35.445196+0000","last_peered":"2024-07-26T08:36:35.445196+0000","last_clean":"2024-07-26T08:36:35.445196+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:36:35.445196+0000","last_undegraded":"2024-07-26T08:36:35.445196+0000","last_fullsized":"2024-07-26T08:36:35.445196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329123+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:34:14.329123+0000","last_peered":"2024-07-26T08:34:14.329123+0000","last_clean":"2024-07-26T08:34:14.329123+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:34:14.329123+0000","last_undegraded":"2024-07-26T08:34:14.329123+0000","last_fullsized":"2024-07-26T08:34:14.329123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733921+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:36:12.733921+0000","last_peered":"2024-07-26T08:36:12.733921+0000","last_clean":"2024-07-26T08:36:12.733921+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:36:12.733921+0000","last_undegraded":"2024-07-26T08:36:12.733921+0000","last_fullsized":"2024-07-26T08:36:12.733921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096346+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:31.096346+0000","last_peered":"2024-07-26T08:34:31.096346+0000","last_clean":"2024-07-26T08:34:31.096346+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:31.096346+0000","last_undegraded":"2024-07-26T08:34:31.096346+0000","last_fullsized":"2024-07-26T08:34:31.096346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953794+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:34:13.953794+0000","last_peered":"2024-07-26T08:34:13.953794+0000","last_clean":"2024-07-26T08:34:13.953794+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:34:13.953794+0000","last_undegraded":"2024-07-26T08:34:13.953794+0000","last_fullsized":"2024-07-26T08:34:13.953794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745169+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:36:12.745169+0000","last_peered":"2024-07-26T08:36:12.745169+0000","last_clean":"2024-07-26T08:36:12.745169+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:36:12.745169+0000","last_undegraded":"2024-07-26T08:36:12.745169+0000","last_fullsized":"2024-07-26T08:36:12.745169+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.109130+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:34:32.109130+0000","last_peered":"2024-07-26T08:34:32.109130+0000","last_clean":"2024-07-26T08:34:32.109130+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:34:32.109130+0000","last_undegraded":"2024-07-26T08:34:32.109130+0000","last_fullsized":"2024-07-26T08:34:32.109130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.252777+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:34:14.252777+0000","last_peered":"2024-07-26T08:34:14.252777+0000","last_clean":"2024-07-26T08:34:14.252777+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:34:14.252777+0000","last_undegraded":"2024-07-26T08:34:14.252777+0000","last_fullsized":"2024-07-26T08:34:14.252777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:45.489565+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:34:45.489565+0000","last_peered":"2024-07-26T08:34:45.489565+0000","last_clean":"2024-07-26T08:34:45.489565+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:34:45.489565+0000","last_undegraded":"2024-07-26T08:34:45.489565+0000","last_fullsized":"2024-07-26T08:34:45.489565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741925+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:36:12.741925+0000","last_peered":"2024-07-26T08:36:12.741925+0000","last_clean":"2024-07-26T08:36:12.741925+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:36:12.741925+0000","last_undegraded":"2024-07-26T08:36:12.741925+0000","last_fullsized":"2024-07-26T08:36:12.741925+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":185,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:38.309870+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:36:38.309870+0000","last_peered":"2024-07-26T08:36:38.309870+0000","last_clean":"2024-07-26T08:36:38.309870+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:36:38.309870+0000","last_undegraded":"2024-07-26T08:36:38.309870+0000","last_fullsized":"2024-07-26T08:36:38.309870+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.952699+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:34:13.952699+0000","last_peered":"2024-07-26T08:34:13.952699+0000","last_clean":"2024-07-26T08:34:13.952699+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:34:13.952699+0000","last_undegraded":"2024-07-26T08:34:13.952699+0000","last_fullsized":"2024-07-26T08:34:13.952699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744004+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:36:12.744004+0000","last_peered":"2024-07-26T08:36:12.744004+0000","last_clean":"2024-07-26T08:36:12.744004+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:36:12.744004+0000","last_undegraded":"2024-07-26T08:36:12.744004+0000","last_fullsized":"2024-07-26T08:36:12.744004+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106683+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:34:32.106683+0000","last_peered":"2024-07-26T08:34:32.106683+0000","last_clean":"2024-07-26T08:34:32.106683+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:34:32.106683+0000","last_undegraded":"2024-07-26T08:34:32.106683+0000","last_fullsized":"2024-07-26T08:34:32.106683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967150+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:34:13.967150+0000","last_peered":"2024-07-26T08:34:13.967150+0000","last_clean":"2024-07-26T08:34:13.967150+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:34:13.967150+0000","last_undegraded":"2024-07-26T08:34:13.967150+0000","last_fullsized":"2024-07-26T08:34:13.967150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.726718+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:36:12.726718+0000","last_peered":"2024-07-26T08:36:12.726718+0000","last_clean":"2024-07-26T08:36:12.726718+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:36:12.726718+0000","last_undegraded":"2024-07-26T08:36:12.726718+0000","last_fullsized":"2024-07-26T08:36:12.726718+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096563+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:34:31.096563+0000","last_peered":"2024-07-26T08:34:31.096563+0000","last_clean":"2024-07-26T08:34:31.096563+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:34:31.096563+0000","last_undegraded":"2024-07-26T08:34:31.096563+0000","last_fullsized":"2024-07-26T08:34:31.096563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953156+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:34:13.953156+0000","last_peered":"2024-07-26T08:34:13.953156+0000","last_clean":"2024-07-26T08:34:13.953156+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:34:13.953156+0000","last_undegraded":"2024-07-26T08:34:13.953156+0000","last_fullsized":"2024-07-26T08:34:13.953156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.720115+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:36:12.720115+0000","last_peered":"2024-07-26T08:36:12.720115+0000","last_clean":"2024-07-26T08:36:12.720115+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:36:12.720115+0000","last_undegraded":"2024-07-26T08:36:12.720115+0000","last_fullsized":"2024-07-26T08:36:12.720115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106297+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:34:32.106297+0000","last_peered":"2024-07-26T08:34:32.106297+0000","last_clean":"2024-07-26T08:34:32.106297+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:34:32.106297+0000","last_undegraded":"2024-07-26T08:34:32.106297+0000","last_fullsized":"2024-07-26T08:34:32.106297+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953681+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:34:13.953681+0000","last_peered":"2024-07-26T08:34:13.953681+0000","last_clean":"2024-07-26T08:34:13.953681+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:34:13.953681+0000","last_undegraded":"2024-07-26T08:34:13.953681+0000","last_fullsized":"2024-07-26T08:34:13.953681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733613+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:36:12.733613+0000","last_peered":"2024-07-26T08:36:12.733613+0000","last_clean":"2024-07-26T08:36:12.733613+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:36:12.733613+0000","last_undegraded":"2024-07-26T08:36:12.733613+0000","last_fullsized":"2024-07-26T08:36:12.733613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108992+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:32.108992+0000","last_peered":"2024-07-26T08:34:32.108992+0000","last_clean":"2024-07-26T08:34:32.108992+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:32.108992+0000","last_undegraded":"2024-07-26T08:34:32.108992+0000","last_fullsized":"2024-07-26T08:34:32.108992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.329087+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:34:14.329087+0000","last_peered":"2024-07-26T08:34:14.329087+0000","last_clean":"2024-07-26T08:34:14.329087+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:34:14.329087+0000","last_undegraded":"2024-07-26T08:34:14.329087+0000","last_fullsized":"2024-07-26T08:34:14.329087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.731525+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:36:12.731525+0000","last_peered":"2024-07-26T08:36:12.731525+0000","last_clean":"2024-07-26T08:36:12.731525+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:36:12.731525+0000","last_undegraded":"2024-07-26T08:36:12.731525+0000","last_fullsized":"2024-07-26T08:36:12.731525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":184,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:37.452268+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:36:37.452268+0000","last_peered":"2024-07-26T08:36:37.452268+0000","last_clean":"2024-07-26T08:36:37.452268+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:36:37.452268+0000","last_undegraded":"2024-07-26T08:36:37.452268+0000","last_fullsized":"2024-07-26T08:36:37.452268+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.957436+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:34:13.957436+0000","last_peered":"2024-07-26T08:34:13.957436+0000","last_clean":"2024-07-26T08:34:13.957436+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:34:13.957436+0000","last_undegraded":"2024-07-26T08:34:13.957436+0000","last_fullsized":"2024-07-26T08:34:13.957436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733851+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:36:12.733851+0000","last_peered":"2024-07-26T08:36:12.733851+0000","last_clean":"2024-07-26T08:36:12.733851+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:36:12.733851+0000","last_undegraded":"2024-07-26T08:36:12.733851+0000","last_fullsized":"2024-07-26T08:36:12.733851+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.106170+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:34:32.106170+0000","last_peered":"2024-07-26T08:34:32.106170+0000","last_clean":"2024-07-26T08:34:32.106170+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:34:32.106170+0000","last_undegraded":"2024-07-26T08:34:32.106170+0000","last_fullsized":"2024-07-26T08:34:32.106170+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.961991+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:34:13.961991+0000","last_peered":"2024-07-26T08:34:13.961991+0000","last_clean":"2024-07-26T08:34:13.961991+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:34:13.961991+0000","last_undegraded":"2024-07-26T08:34:13.961991+0000","last_fullsized":"2024-07-26T08:34:13.961991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740754+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:36:12.740754+0000","last_peered":"2024-07-26T08:36:12.740754+0000","last_clean":"2024-07-26T08:36:12.740754+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:36:12.740754+0000","last_undegraded":"2024-07-26T08:36:12.740754+0000","last_fullsized":"2024-07-26T08:36:12.740754+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108272+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:34:32.108272+0000","last_peered":"2024-07-26T08:34:32.108272+0000","last_clean":"2024-07-26T08:34:32.108272+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:34:32.108272+0000","last_undegraded":"2024-07-26T08:34:32.108272+0000","last_fullsized":"2024-07-26T08:34:32.108272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.953564+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:34:13.953564+0000","last_peered":"2024-07-26T08:34:13.953564+0000","last_clean":"2024-07-26T08:34:13.953564+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:34:13.953564+0000","last_undegraded":"2024-07-26T08:34:13.953564+0000","last_fullsized":"2024-07-26T08:34:13.953564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.710145+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:36:12.710145+0000","last_peered":"2024-07-26T08:36:12.710145+0000","last_clean":"2024-07-26T08:36:12.710145+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:36:12.710145+0000","last_undegraded":"2024-07-26T08:36:12.710145+0000","last_fullsized":"2024-07-26T08:36:12.710145+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'41","reported_seq":349,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:37.308270+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:36:37.308270+0000","last_peered":"2024-07-26T08:36:37.308270+0000","last_clean":"2024-07-26T08:36:37.308270+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:36:37.308270+0000","last_undegraded":"2024-07-26T08:36:37.308270+0000","last_fullsized":"2024-07-26T08:36:37.308270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67283,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":262,"num_read_kb":197,"num_write":43,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.955409+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:34:13.955409+0000","last_peered":"2024-07-26T08:34:13.955409+0000","last_clean":"2024-07-26T08:34:13.955409+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:34:13.955409+0000","last_undegraded":"2024-07-26T08:34:13.955409+0000","last_fullsized":"2024-07-26T08:34:13.955409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738435+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:36:12.738435+0000","last_peered":"2024-07-26T08:36:12.738435+0000","last_clean":"2024-07-26T08:36:12.738435+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:36:12.738435+0000","last_undegraded":"2024-07-26T08:36:12.738435+0000","last_fullsized":"2024-07-26T08:36:12.738435+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.111302+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:34:32.111302+0000","last_peered":"2024-07-26T08:34:32.111302+0000","last_clean":"2024-07-26T08:34:32.111302+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:34:32.111302+0000","last_undegraded":"2024-07-26T08:34:32.111302+0000","last_fullsized":"2024-07-26T08:34:32.111302+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.969909+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:34:13.969909+0000","last_peered":"2024-07-26T08:34:13.969909+0000","last_clean":"2024-07-26T08:34:13.969909+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:34:13.969909+0000","last_undegraded":"2024-07-26T08:34:13.969909+0000","last_fullsized":"2024-07-26T08:34:13.969909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735248+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:36:12.735248+0000","last_peered":"2024-07-26T08:36:12.735248+0000","last_clean":"2024-07-26T08:36:12.735248+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:36:12.735248+0000","last_undegraded":"2024-07-26T08:36:12.735248+0000","last_fullsized":"2024-07-26T08:36:12.735248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.097187+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:34:31.097187+0000","last_peered":"2024-07-26T08:34:31.097187+0000","last_clean":"2024-07-26T08:34:31.097187+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:34:31.097187+0000","last_undegraded":"2024-07-26T08:34:31.097187+0000","last_fullsized":"2024-07-26T08:34:31.097187+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.954260+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:34:13.954260+0000","last_peered":"2024-07-26T08:34:13.954260+0000","last_clean":"2024-07-26T08:34:13.954260+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:34:13.954260+0000","last_undegraded":"2024-07-26T08:34:13.954260+0000","last_fullsized":"2024-07-26T08:34:13.954260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741011+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:36:12.741011+0000","last_peered":"2024-07-26T08:36:12.741011+0000","last_clean":"2024-07-26T08:36:12.741011+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:36:12.741011+0000","last_undegraded":"2024-07-26T08:36:12.741011+0000","last_fullsized":"2024-07-26T08:36:12.741011+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":303,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:37.307388+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:36:37.307388+0000","last_peered":"2024-07-26T08:36:37.307388+0000","last_clean":"2024-07-26T08:36:37.307388+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:36:37.307388+0000","last_undegraded":"2024-07-26T08:36:37.307388+0000","last_fullsized":"2024-07-26T08:36:37.307388+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":252,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:14.591838+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:34:14.591838+0000","last_peered":"2024-07-26T08:34:14.591838+0000","last_clean":"2024-07-26T08:34:14.591838+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:34:14.591838+0000","last_undegraded":"2024-07-26T08:34:14.591838+0000","last_fullsized":"2024-07-26T08:34:14.591838+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.967620+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:34:13.967620+0000","last_peered":"2024-07-26T08:34:13.967620+0000","last_clean":"2024-07-26T08:34:13.967620+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:34:13.967620+0000","last_undegraded":"2024-07-26T08:34:13.967620+0000","last_fullsized":"2024-07-26T08:34:13.967620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746247+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:36:12.746247+0000","last_peered":"2024-07-26T08:36:12.746247+0000","last_clean":"2024-07-26T08:36:12.746247+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:36:12.746247+0000","last_undegraded":"2024-07-26T08:36:12.746247+0000","last_fullsized":"2024-07-26T08:36:12.746247+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.111667+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:34:32.111667+0000","last_peered":"2024-07-26T08:34:32.111667+0000","last_clean":"2024-07-26T08:34:32.111667+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:34:32.111667+0000","last_undegraded":"2024-07-26T08:34:32.111667+0000","last_fullsized":"2024-07-26T08:34:32.111667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.962233+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:34:13.962233+0000","last_peered":"2024-07-26T08:34:13.962233+0000","last_clean":"2024-07-26T08:34:13.962233+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:34:13.962233+0000","last_undegraded":"2024-07-26T08:34:13.962233+0000","last_fullsized":"2024-07-26T08:34:13.962233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741760+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:36:12.741760+0000","last_peered":"2024-07-26T08:36:12.741760+0000","last_clean":"2024-07-26T08:36:12.741760+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:36:12.741760+0000","last_undegraded":"2024-07-26T08:36:12.741760+0000","last_fullsized":"2024-07-26T08:36:12.741760+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:31.096093+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:34:31.096093+0000","last_peered":"2024-07-26T08:34:31.096093+0000","last_clean":"2024-07-26T08:34:31.096093+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:34:31.096093+0000","last_undegraded":"2024-07-26T08:34:31.096093+0000","last_fullsized":"2024-07-26T08:34:31.096093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:30.091831+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:34:30.091831+0000","last_peered":"2024-07-26T08:34:30.091831+0000","last_clean":"2024-07-26T08:34:30.091831+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:34:30.091831+0000","last_undegraded":"2024-07-26T08:34:30.091831+0000","last_fullsized":"2024-07-26T08:34:30.091831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740795+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:36:12.740795+0000","last_peered":"2024-07-26T08:36:12.740795+0000","last_clean":"2024-07-26T08:36:12.740795+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:36:12.740795+0000","last_undegraded":"2024-07-26T08:36:12.740795+0000","last_fullsized":"2024-07-26T08:36:12.740795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.964602+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:34:13.964602+0000","last_peered":"2024-07-26T08:34:13.964602+0000","last_clean":"2024-07-26T08:34:13.964602+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:34:13.964602+0000","last_undegraded":"2024-07-26T08:34:13.964602+0000","last_fullsized":"2024-07-26T08:34:13.964602+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":178,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:35.304900+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:36:35.304900+0000","last_peered":"2024-07-26T08:36:35.304900+0000","last_clean":"2024-07-26T08:36:35.304900+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:36:35.304900+0000","last_undegraded":"2024-07-26T08:36:35.304900+0000","last_fullsized":"2024-07-26T08:36:35.304900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727268+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:36:12.727268+0000","last_peered":"2024-07-26T08:36:12.727268+0000","last_clean":"2024-07-26T08:36:12.727268+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:36:12.727268+0000","last_undegraded":"2024-07-26T08:36:12.727268+0000","last_fullsized":"2024-07-26T08:36:12.727268+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:13.965555+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:34:13.965555+0000","last_peered":"2024-07-26T08:34:13.965555+0000","last_clean":"2024-07-26T08:34:13.965555+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:34:13.965555+0000","last_undegraded":"2024-07-26T08:34:13.965555+0000","last_fullsized":"2024-07-26T08:34:13.965555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723604+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:36:12.723604+0000","last_peered":"2024-07-26T08:36:12.723604+0000","last_clean":"2024-07-26T08:36:12.723604+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:36:12.723604+0000","last_undegraded":"2024-07-26T08:36:12.723604+0000","last_fullsized":"2024-07-26T08:36:12.723604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721726+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:36:12.721726+0000","last_peered":"2024-07-26T08:36:12.721726+0000","last_clean":"2024-07-26T08:36:12.721726+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:36:12.721726+0000","last_undegraded":"2024-07-26T08:36:12.721726+0000","last_fullsized":"2024-07-26T08:36:12.721726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737335+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:36:12.737335+0000","last_peered":"2024-07-26T08:36:12.737335+0000","last_clean":"2024-07-26T08:36:12.737335+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:36:12.737335+0000","last_undegraded":"2024-07-26T08:36:12.737335+0000","last_fullsized":"2024-07-26T08:36:12.737335+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729035+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:36:12.729035+0000","last_peered":"2024-07-26T08:36:12.729035+0000","last_clean":"2024-07-26T08:36:12.729035+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:36:12.729035+0000","last_undegraded":"2024-07-26T08:36:12.729035+0000","last_fullsized":"2024-07-26T08:36:12.729035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737901+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:36:12.737901+0000","last_peered":"2024-07-26T08:36:12.737901+0000","last_clean":"2024-07-26T08:36:12.737901+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:36:12.737901+0000","last_undegraded":"2024-07-26T08:36:12.737901+0000","last_fullsized":"2024-07-26T08:36:12.737901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718361+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:36:12.718361+0000","last_peered":"2024-07-26T08:36:12.718361+0000","last_clean":"2024-07-26T08:36:12.718361+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:36:12.718361+0000","last_undegraded":"2024-07-26T08:36:12.718361+0000","last_fullsized":"2024-07-26T08:36:12.718361+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.732615+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:36:12.732615+0000","last_peered":"2024-07-26T08:36:12.732615+0000","last_clean":"2024-07-26T08:36:12.732615+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:36:12.732615+0000","last_undegraded":"2024-07-26T08:36:12.732615+0000","last_fullsized":"2024-07-26T08:36:12.732615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743216+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:36:12.743216+0000","last_peered":"2024-07-26T08:36:12.743216+0000","last_clean":"2024-07-26T08:36:12.743216+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:36:12.743216+0000","last_undegraded":"2024-07-26T08:36:12.743216+0000","last_fullsized":"2024-07-26T08:36:12.743216+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745514+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:36:12.745514+0000","last_peered":"2024-07-26T08:36:12.745514+0000","last_clean":"2024-07-26T08:36:12.745514+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:36:12.745514+0000","last_undegraded":"2024-07-26T08:36:12.745514+0000","last_fullsized":"2024-07-26T08:36:12.745514+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742101+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:36:12.742101+0000","last_peered":"2024-07-26T08:36:12.742101+0000","last_clean":"2024-07-26T08:36:12.742101+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:36:12.742101+0000","last_undegraded":"2024-07-26T08:36:12.742101+0000","last_fullsized":"2024-07-26T08:36:12.742101+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746400+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:36:12.746400+0000","last_peered":"2024-07-26T08:36:12.746400+0000","last_clean":"2024-07-26T08:36:12.746400+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:36:12.746400+0000","last_undegraded":"2024-07-26T08:36:12.746400+0000","last_fullsized":"2024-07-26T08:36:12.746400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742305+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:36:12.742305+0000","last_peered":"2024-07-26T08:36:12.742305+0000","last_clean":"2024-07-26T08:36:12.742305+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:36:12.742305+0000","last_undegraded":"2024-07-26T08:36:12.742305+0000","last_fullsized":"2024-07-26T08:36:12.742305+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723067+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:36:12.723067+0000","last_peered":"2024-07-26T08:36:12.723067+0000","last_clean":"2024-07-26T08:36:12.723067+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:36:12.723067+0000","last_undegraded":"2024-07-26T08:36:12.723067+0000","last_fullsized":"2024-07-26T08:36:12.723067+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735951+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:36:12.735951+0000","last_peered":"2024-07-26T08:36:12.735951+0000","last_clean":"2024-07-26T08:36:12.735951+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:36:12.735951+0000","last_undegraded":"2024-07-26T08:36:12.735951+0000","last_fullsized":"2024-07-26T08:36:12.735951+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733519+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:36:12.733519+0000","last_peered":"2024-07-26T08:36:12.733519+0000","last_clean":"2024-07-26T08:36:12.733519+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:36:12.733519+0000","last_undegraded":"2024-07-26T08:36:12.733519+0000","last_fullsized":"2024-07-26T08:36:12.733519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740709+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:36:12.740709+0000","last_peered":"2024-07-26T08:36:12.740709+0000","last_clean":"2024-07-26T08:36:12.740709+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:36:12.740709+0000","last_undegraded":"2024-07-26T08:36:12.740709+0000","last_fullsized":"2024-07-26T08:36:12.740709+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743025+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:36:12.743025+0000","last_peered":"2024-07-26T08:36:12.743025+0000","last_clean":"2024-07-26T08:36:12.743025+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:36:12.743025+0000","last_undegraded":"2024-07-26T08:36:12.743025+0000","last_fullsized":"2024-07-26T08:36:12.743025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735949+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:36:12.735949+0000","last_peered":"2024-07-26T08:36:12.735949+0000","last_clean":"2024-07-26T08:36:12.735949+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:36:12.735949+0000","last_undegraded":"2024-07-26T08:36:12.735949+0000","last_fullsized":"2024-07-26T08:36:12.735949+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736238+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:36:12.736238+0000","last_peered":"2024-07-26T08:36:12.736238+0000","last_clean":"2024-07-26T08:36:12.736238+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:36:12.736238+0000","last_undegraded":"2024-07-26T08:36:12.736238+0000","last_fullsized":"2024-07-26T08:36:12.736238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728244+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:36:12.728244+0000","last_peered":"2024-07-26T08:36:12.728244+0000","last_clean":"2024-07-26T08:36:12.728244+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:36:12.728244+0000","last_undegraded":"2024-07-26T08:36:12.728244+0000","last_fullsized":"2024-07-26T08:36:12.728244+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718869+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:36:12.718869+0000","last_peered":"2024-07-26T08:36:12.718869+0000","last_clean":"2024-07-26T08:36:12.718869+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:36:12.718869+0000","last_undegraded":"2024-07-26T08:36:12.718869+0000","last_fullsized":"2024-07-26T08:36:12.718869+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742060+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:36:12.742060+0000","last_peered":"2024-07-26T08:36:12.742060+0000","last_clean":"2024-07-26T08:36:12.742060+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:36:12.742060+0000","last_undegraded":"2024-07-26T08:36:12.742060+0000","last_fullsized":"2024-07-26T08:36:12.742060+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721655+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:36:12.721655+0000","last_peered":"2024-07-26T08:36:12.721655+0000","last_clean":"2024-07-26T08:36:12.721655+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:36:12.721655+0000","last_undegraded":"2024-07-26T08:36:12.721655+0000","last_fullsized":"2024-07-26T08:36:12.721655+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740155+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:36:12.740155+0000","last_peered":"2024-07-26T08:36:12.740155+0000","last_clean":"2024-07-26T08:36:12.740155+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:36:12.740155+0000","last_undegraded":"2024-07-26T08:36:12.740155+0000","last_fullsized":"2024-07-26T08:36:12.740155+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744881+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:36:12.744881+0000","last_peered":"2024-07-26T08:36:12.744881+0000","last_clean":"2024-07-26T08:36:12.744881+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:36:12.744881+0000","last_undegraded":"2024-07-26T08:36:12.744881+0000","last_fullsized":"2024-07-26T08:36:12.744881+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739021+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:36:12.739021+0000","last_peered":"2024-07-26T08:36:12.739021+0000","last_clean":"2024-07-26T08:36:12.739021+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:36:12.739021+0000","last_undegraded":"2024-07-26T08:36:12.739021+0000","last_fullsized":"2024-07-26T08:36:12.739021+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728498+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:36:12.728498+0000","last_peered":"2024-07-26T08:36:12.728498+0000","last_clean":"2024-07-26T08:36:12.728498+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:36:12.728498+0000","last_undegraded":"2024-07-26T08:36:12.728498+0000","last_fullsized":"2024-07-26T08:36:12.728498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743776+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:36:12.743776+0000","last_peered":"2024-07-26T08:36:12.743776+0000","last_clean":"2024-07-26T08:36:12.743776+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:36:12.743776+0000","last_undegraded":"2024-07-26T08:36:12.743776+0000","last_fullsized":"2024-07-26T08:36:12.743776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719531+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:36:12.719531+0000","last_peered":"2024-07-26T08:36:12.719531+0000","last_clean":"2024-07-26T08:36:12.719531+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:36:12.719531+0000","last_undegraded":"2024-07-26T08:36:12.719531+0000","last_fullsized":"2024-07-26T08:36:12.719531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.735860+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:36:12.735860+0000","last_peered":"2024-07-26T08:36:12.735860+0000","last_clean":"2024-07-26T08:36:12.735860+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:36:12.735860+0000","last_undegraded":"2024-07-26T08:36:12.735860+0000","last_fullsized":"2024-07-26T08:36:12.735860+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.725277+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:36:12.725277+0000","last_peered":"2024-07-26T08:36:12.725277+0000","last_clean":"2024-07-26T08:36:12.725277+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:36:12.725277+0000","last_undegraded":"2024-07-26T08:36:12.725277+0000","last_fullsized":"2024-07-26T08:36:12.725277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740147+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:36:12.740147+0000","last_peered":"2024-07-26T08:36:12.740147+0000","last_clean":"2024-07-26T08:36:12.740147+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:36:12.740147+0000","last_undegraded":"2024-07-26T08:36:12.740147+0000","last_fullsized":"2024-07-26T08:36:12.740147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74769,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1303,"num_read_kb":732,"num_write":262,"num_write_kb":156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":224307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645373,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540314,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841601,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1198397,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005126,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201358,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397587,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136616,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561110,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757341,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920866,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539710,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117095,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378736,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":604169,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542259,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1085440,"data_stored":603417,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":537092,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:36:42.969 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T08:36:42.970 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T08:36:42.970 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-26T08:36:42.970 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph health --format=json 2024-07-26T08:36:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:43 smithi044 ceph-mon[32404]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:36:43.501 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:43 smithi116 ceph-mon[37097]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:36:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:43 smithi171 ceph-mon[37789]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:36:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:44 smithi044 ceph-mon[32404]: pgmap v424: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:44 smithi116 ceph-mon[37097]: pgmap v424: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:44 smithi171 ceph-mon[37789]: pgmap v424: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:45.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 -- 172.21.15.44:0/2007095289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b10108d20 msgr2=0x7f1b10109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2007095289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b10108d20 0x7f1b10109130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009900 tx=0x7f1af802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 -- 172.21.15.44:0/2007095289 shutdown_connections 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2007095289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 0x7f1b1010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2007095289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b10108d20 0x7f1b10109130 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2007095289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b101033e0 0x7f1b101037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 -- 172.21.15.44:0/2007095289 >> 172.21.15.44:0/2007095289 conn(0x7f1b100fee30 msgr2=0x7f1b10101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.809+0000 7f1b0f200640 1 -- 172.21.15.44:0/2007095289 shutdown_connections 2024-07-26T08:36:45.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.810+0000 7f1b0f200640 1 -- 172.21.15.44:0/2007095289 wait complete. 2024-07-26T08:36:45.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.811+0000 7f1b0f200640 1 Processor -- start 2024-07-26T08:36:45.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.811+0000 7f1b0f200640 1 -- start start 2024-07-26T08:36:45.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 0x7f1b10075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0f200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b10108d20 0x7f1b10075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 0x7f1b10075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b101132e0 con 0x7f1b10109670 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10113160 con 0x7f1b101033e0 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0f200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10113460 con 0x7f1b10108d20 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 0x7f1b10075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 0x7f1b10075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:41410/0 (socket says 172.21.15.44:41410) 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0de00640 1 -- 172.21.15.44:0/2693293180 learned_addr learned my addr 172.21.15.44:0/2693293180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0d400640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b10108d20 0x7f1b10075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.812+0000 7f1b0e800640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 0x7f1b10075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:45.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0de00640 1 -- 172.21.15.44:0/2693293180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b10108d20 msgr2=0x7f1b10075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0de00640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b10108d20 0x7f1b10075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0de00640 1 -- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 msgr2=0x7f1b10075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0de00640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 0x7f1b10075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0de00640 1 -- 172.21.15.44:0/2693293180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b101a7fb0 con 0x7f1b101033e0 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0e800640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 0x7f1b10075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0de00640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 0x7f1b10075140 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000eab0 tx=0x7f1b0000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0000ce40 con 0x7f1b101033e0 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b00011920 con 0x7f1b101033e0 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b101a82a0 con 0x7f1b101033e0 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.813+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00010a30 con 0x7f1b101033e0 2024-07-26T08:36:45.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.814+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b101a87b0 con 0x7f1b101033e0 2024-07-26T08:36:45.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.815+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b101037f0 con 0x7f1b101033e0 2024-07-26T08:36:45.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.817+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f1b00010bd0 con 0x7f1b101033e0 2024-07-26T08:36:45.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.819+0000 7f1b06a00640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1af40b3c20 0x7f1af40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:45.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.819+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f1b000e4760 con 0x7f1b101033e0 2024-07-26T08:36:45.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.819+0000 7f1b0d400640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1af40b3c20 0x7f1af40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:45.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.820+0000 7f1b0d400640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1af40b3c20 0x7f1af40b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009230 tx=0x7f1af8031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:45.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:45.820+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1b00016030 con 0x7f1b101033e0 2024-07-26T08:36:46.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.072+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1b10076440 con 0x7f1b101033e0 2024-07-26T08:36:46.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.073+0000 7f1b06a00640 1 -- 172.21.15.44:0/2693293180 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f1b000ab8a0 con 0x7f1b101033e0 2024-07-26T08:36:46.075 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:36:46.075 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T08:36:46.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1af40b3c20 msgr2=0x7f1af40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:46.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1af40b3c20 0x7f1af40b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009230 tx=0x7f1af8031400 comp rx=0 tx=0).stop 2024-07-26T08:36:46.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 msgr2=0x7f1b10075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:46.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 0x7f1b10075140 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000eab0 tx=0x7f1b0000ef70 comp rx=0 tx=0).stop 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 shutdown_connections 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1af40b3c20 0x7f1af40b6130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b10108d20 0x7f1b10075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b10109670 0x7f1b10075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 --2- 172.21.15.44:0/2693293180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b101033e0 0x7f1b10075140 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 >> 172.21.15.44:0/2693293180 conn(0x7f1b100fee30 msgr2=0x7f1b10101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 shutdown_connections 2024-07-26T08:36:46.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:46.076+0000 7f1b0f200640 1 -- 172.21.15.44:0/2693293180 wait complete. 2024-07-26T08:36:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:46 smithi044 ceph-mon[32404]: pgmap v425: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:46.474 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:46 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2693293180' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T08:36:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:46 smithi116 ceph-mon[37097]: pgmap v425: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:46 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2693293180' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T08:36:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:46 smithi171 ceph-mon[37789]: pgmap v425: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:46 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2693293180' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T08:36:46.927 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-26T08:36:46.928 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:36:47.417 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:48.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:48 smithi044 ceph-mon[32404]: pgmap v426: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:48 smithi116 ceph-mon[37097]: pgmap v426: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:48 smithi171 ceph-mon[37789]: pgmap v426: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 -- 172.21.15.44:0/3867655993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e8075b30 msgr2=0x7fa8e810f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 --2- 172.21.15.44:0/3867655993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e8075b30 0x7fa8e810f400 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009860 tx=0x7fa8d402a340 comp rx=0 tx=0).stop 2024-07-26T08:36:49.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 -- 172.21.15.44:0/3867655993 shutdown_connections 2024-07-26T08:36:49.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 --2- 172.21.15.44:0/3867655993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e8075b30 0x7fa8e810f400 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 --2- 172.21.15.44:0/3867655993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e810f940 0x7fa8e8111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 --2- 172.21.15.44:0/3867655993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e80751e0 0x7fa8e80755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 -- 172.21.15.44:0/3867655993 >> 172.21.15.44:0/3867655993 conn(0x7fa8e80fee70 msgr2=0x7fa8e81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:49.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 -- 172.21.15.44:0/3867655993 shutdown_connections 2024-07-26T08:36:49.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.681+0000 7fa8ee800640 1 -- 172.21.15.44:0/3867655993 wait complete. 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 Processor -- start 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 -- start start 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e80751e0 0x7fa8e819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e8075b30 0x7fa8e819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e810f940 0x7fa8e81a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e81154e0 con 0x7fa8e80751e0 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8115360 con 0x7fa8e8075b30 2024-07-26T08:36:49.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.683+0000 7fa8ee800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8115660 con 0x7fa8e810f940 2024-07-26T08:36:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8ed400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e80751e0 0x7fa8e819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8ede00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e810f940 0x7fa8e81a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8ed400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e80751e0 0x7fa8e819bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48212/0 (socket says 172.21.15.44:48212) 2024-07-26T08:36:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8eca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e8075b30 0x7fa8e819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:49.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8ed400640 1 -- 172.21.15.44:0/2473561146 learned_addr learned my addr 172.21.15.44:0/2473561146 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:49.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8eca00640 1 -- 172.21.15.44:0/2473561146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e810f940 msgr2=0x7fa8e81a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:49.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8eca00640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e810f940 0x7fa8e81a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8eca00640 1 -- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e80751e0 msgr2=0x7fa8e819bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:49.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8eca00640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e80751e0 0x7fa8e819bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8eca00640 1 -- 172.21.15.44:0/2473561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e81a1ec0 con 0x7fa8e8075b30 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.684+0000 7fa8ede00640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e810f940 0x7fa8e81a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.685+0000 7fa8eca00640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e8075b30 0x7fa8e819c4f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8d402a860 tx=0x7fa8d4032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.685+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4037040 con 0x7fa8e8075b30 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.685+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e81a2150 con 0x7fa8e8075b30 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.685+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d4030c10 con 0x7fa8e8075b30 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.685+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d40035f0 con 0x7fa8e8075b30 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.685+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e81a8910 con 0x7fa8e8075b30 2024-07-26T08:36:49.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.686+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e80755f0 con 0x7fa8e8075b30 2024-07-26T08:36:49.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.689+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fa8d4043020 con 0x7fa8e8075b30 2024-07-26T08:36:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.692+0000 7fa8e6000640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8d00b3c20 0x7fa8d00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.692+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fa8d4103850 con 0x7fa8e8075b30 2024-07-26T08:36:49.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.692+0000 7fa8ed400640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8d00b3c20 0x7fa8d00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:49.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.693+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa8d40caa10 con 0x7fa8e8075b30 2024-07-26T08:36:49.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.693+0000 7fa8ed400640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8d00b3c20 0x7fa8d00b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009cd0 tx=0x7fa8dc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:49.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.935+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8e81a8ae0 con 0x7fa8e8075b30 2024-07-26T08:36:49.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.936+0000 7fa8e6000640 1 -- 172.21.15.44:0/2473561146 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7fa8d40328c0 con 0x7fa8e8075b30 2024-07-26T08:36:49.938 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:36:49.938 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"btime":"2024-07-26T08:34:37:118441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:34:36.302968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24661,24668,24679,24686,24701]},"id":1}]} 2024-07-26T08:36:49.938 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:36:49.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8d00b3c20 msgr2=0x7fa8d00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:49.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8d00b3c20 0x7fa8d00b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009cd0 tx=0x7fa8dc005f90 comp rx=0 tx=0).stop 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e8075b30 msgr2=0x7fa8e819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e8075b30 0x7fa8e819c4f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8d402a860 tx=0x7fa8d4032d50 comp rx=0 tx=0).stop 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 shutdown_connections 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fa8d00b3c20 0x7fa8d00b6130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8e810f940 0x7fa8e81a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e8075b30 0x7fa8e819c4f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 --2- 172.21.15.44:0/2473561146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8e80751e0 0x7fa8e819bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:49.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 >> 172.21.15.44:0/2473561146 conn(0x7fa8e80fee70 msgr2=0x7fa8e8109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:49.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 shutdown_connections 2024-07-26T08:36:49.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:49.939+0000 7fa8ee800640 1 -- 172.21.15.44:0/2473561146 wait complete. 2024-07-26T08:36:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:50 smithi044 ceph-mon[32404]: pgmap v427: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:50 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2473561146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:36:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:50 smithi116 ceph-mon[37097]: pgmap v427: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:50 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2473561146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:36:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:50 smithi171 ceph-mon[37789]: pgmap v427: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:50 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2473561146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:36:50.981 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:36:51.482 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:52 smithi044 ceph-mon[32404]: pgmap v428: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:52 smithi116 ceph-mon[37097]: pgmap v428: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:52 smithi171 ceph-mon[37789]: pgmap v428: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:36:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 -- 172.21.15.44:0/223282589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19301017a0 msgr2=0x7f1930101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:53.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 --2- 172.21.15.44:0/223282589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19301017a0 0x7f1930101bb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1918009860 tx=0x7f191802a340 comp rx=0 tx=0).stop 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 -- 172.21.15.44:0/223282589 shutdown_connections 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 --2- 172.21.15.44:0/223282589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f19301017a0 0x7f1930101bb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 --2- 172.21.15.44:0/223282589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 0x7f1930111f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 --2- 172.21.15.44:0/223282589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1930104090 0x7f19301044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 -- 172.21.15.44:0/223282589 >> 172.21.15.44:0/223282589 conn(0x7f1930077690 msgr2=0x7f1930077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 -- 172.21.15.44:0/223282589 shutdown_connections 2024-07-26T08:36:53.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.952+0000 7f1936200640 1 -- 172.21.15.44:0/223282589 wait complete. 2024-07-26T08:36:53.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.953+0000 7f1936200640 1 Processor -- start 2024-07-26T08:36:53.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 -- start start 2024-07-26T08:36:53.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 0x7f193019c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:53.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 0x7f193019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1930104090 0x7f19301a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f19301156b0 con 0x7f19301017a0 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1930115530 con 0x7f19301022c0 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f1936200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1930115830 con 0x7f1930104090 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f192f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 0x7f193019c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f192f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 0x7f193019c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54646/0 (socket says 172.21.15.44:54646) 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f192f400640 1 -- 172.21.15.44:0/1769224966 learned_addr learned my addr 172.21.15.44:0/1769224966 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.954+0000 7f192ea00640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 0x7f193019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192fe00640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1930104090 0x7f19301a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:53.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192f400640 1 -- 172.21.15.44:0/1769224966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1930104090 msgr2=0x7f19301a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192f400640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1930104090 0x7f19301a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192f400640 1 -- 172.21.15.44:0/1769224966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 msgr2=0x7f193019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192f400640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 0x7f193019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192f400640 1 -- 172.21.15.44:0/1769224966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19301a1ff0 con 0x7f19301017a0 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192ea00640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 0x7f193019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.955+0000 7f192f400640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 0x7f193019c080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f192000e9c0 tx=0x7f192000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.956+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192000ccc0 con 0x7f19301017a0 2024-07-26T08:36:53.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.956+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19301a22e0 con 0x7f19301017a0 2024-07-26T08:36:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.956+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19200054d0 con 0x7f19301017a0 2024-07-26T08:36:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.956+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19301a8a40 con 0x7f19301017a0 2024-07-26T08:36:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.956+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920018430 con 0x7f19301017a0 2024-07-26T08:36:53.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.957+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1930101bb0 con 0x7f19301017a0 2024-07-26T08:36:53.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.960+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f1920004f20 con 0x7f19301017a0 2024-07-26T08:36:53.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.961+0000 7f1927e00640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19100b3c20 0x7f19100b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:53.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.962+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f19200e44f0 con 0x7f19301017a0 2024-07-26T08:36:53.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.962+0000 7f192ea00640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19100b3c20 0x7f19100b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:53.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.962+0000 7f192ea00640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19100b3c20 0x7f19100b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1918035040 tx=0x7f1918030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:53.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:53.966+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f19200ab630 con 0x7f19301017a0 2024-07-26T08:36:54.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.208+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f193019d1b0 con 0x7f19301017a0 2024-07-26T08:36:54.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.210+0000 7f1927e00640 1 -- 172.21.15.44:0/1769224966 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f19200aaf30 con 0x7f19301017a0 2024-07-26T08:36:54.212 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:36:54.212 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"btime":"2024-07-26T08:34:37:118441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:34:36.302968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24661,24668,24679,24686,24701]},"id":1}]} 2024-07-26T08:36:54.212 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:36:54.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19100b3c20 msgr2=0x7f19100b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:54.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19100b3c20 0x7f19100b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1918035040 tx=0x7f1918030340 comp rx=0 tx=0).stop 2024-07-26T08:36:54.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 msgr2=0x7f193019c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:54.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 0x7f193019c080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f192000e9c0 tx=0x7f192000ee80 comp rx=0 tx=0).stop 2024-07-26T08:36:54.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 shutdown_connections 2024-07-26T08:36:54.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f19100b3c20 0x7f19100b6130 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:54.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1930104090 0x7f19301a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:54.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19301022c0 0x7f193019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:54.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 --2- 172.21.15.44:0/1769224966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19301017a0 0x7f193019c080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:54.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.213+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 >> 172.21.15.44:0/1769224966 conn(0x7f1930077690 msgr2=0x7f1930100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:54.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.214+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 shutdown_connections 2024-07-26T08:36:54.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:54.214+0000 7f1936200640 1 -- 172.21.15.44:0/1769224966 wait complete. 2024-07-26T08:36:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:54 smithi044 ceph-mon[32404]: pgmap v429: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:36:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:54 smithi116 ceph-mon[37097]: pgmap v429: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:36:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:54 smithi171 ceph-mon[37789]: pgmap v429: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:36:55.198 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:36:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:55 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1769224966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:36:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:55 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1769224966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:36:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:55 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1769224966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:36:55.670 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:36:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:56 smithi044 ceph-mon[32404]: pgmap v430: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:56 smithi116 ceph-mon[37097]: pgmap v430: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:56 smithi171 ceph-mon[37789]: pgmap v430: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:36:57.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 -- 172.21.15.44:0/365360070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4109650 msgr2=0x7feec410fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 --2- 172.21.15.44:0/365360070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4109650 0x7feec410fe80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feebc009900 tx=0x7feebc02a410 comp rx=0 tx=0).stop 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 -- 172.21.15.44:0/365360070 shutdown_connections 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 --2- 172.21.15.44:0/365360070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4108d00 0x7feec4109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 --2- 172.21.15.44:0/365360070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4109650 0x7feec410fe80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 --2- 172.21.15.44:0/365360070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 0x7feec41037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.994+0000 7feecae00640 1 -- 172.21.15.44:0/365360070 >> 172.21.15.44:0/365360070 conn(0x7feec40fee10 msgr2=0x7feec4101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.995+0000 7feecae00640 1 -- 172.21.15.44:0/365360070 shutdown_connections 2024-07-26T08:36:57.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.995+0000 7feecae00640 1 -- 172.21.15.44:0/365360070 wait complete. 2024-07-26T08:36:57.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.996+0000 7feecae00640 1 Processor -- start 2024-07-26T08:36:57.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.996+0000 7feecae00640 1 -- start start 2024-07-26T08:36:57.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.997+0000 7feecae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 0x7feec419bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:57.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.997+0000 7feecae00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4108d00 0x7feec419c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:57.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.997+0000 7feecae00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 0x7feec41a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:57.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.997+0000 7feecae00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feec4113300 con 0x7feec4108d00 2024-07-26T08:36:57.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.997+0000 7feecae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feec4113180 con 0x7feec41033c0 2024-07-26T08:36:57.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.997+0000 7feecae00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feec4113480 con 0x7feec4109650 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 0x7feec41a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feec9a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 0x7feec419bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 0x7feec41a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48064/0 (socket says 172.21.15.44:48064) 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 -- 172.21.15.44:0/1404335275 learned_addr learned my addr 172.21.15.44:0/1404335275 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feec9a00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 0x7feec419bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:38714/0 (socket says 172.21.15.44:38714) 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feec9000640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4108d00 0x7feec419c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 -- 172.21.15.44:0/1404335275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 msgr2=0x7feec419bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 0x7feec419bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 -- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4108d00 msgr2=0x7feec419c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:58.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4108d00 0x7feec419c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:58.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.998+0000 7feeca400640 1 -- 172.21.15.44:0/1404335275 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec41a1e60 con 0x7feec4109650 2024-07-26T08:36:58.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.999+0000 7feeca400640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 0x7feec41a1730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feebc004770 tx=0x7feebc004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:58.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.999+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc03f030 con 0x7feec4109650 2024-07-26T08:36:58.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.999+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feebc004440 con 0x7feec4109650 2024-07-26T08:36:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.999+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec41a20f0 con 0x7feec4109650 2024-07-26T08:36:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.999+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc035070 con 0x7feec4109650 2024-07-26T08:36:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:57.999+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec41a88b0 con 0x7feec4109650 2024-07-26T08:36:58.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.000+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec41037d0 con 0x7feec4109650 2024-07-26T08:36:58.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.003+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7feebc042030 con 0x7feec4109650 2024-07-26T08:36:58.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.005+0000 7feec2000640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feeac0b3ca0 0x7feeac0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:36:58.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.005+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7feebc104640 con 0x7feec4109650 2024-07-26T08:36:58.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.005+0000 7feec9a00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feeac0b3ca0 0x7feeac0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:36:58.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.006+0000 7feec9a00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feeac0b3ca0 0x7feeac0b61b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feeb8005fd0 tx=0x7feeb8009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:36:58.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.006+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feebc0cb130 con 0x7feec4109650 2024-07-26T08:36:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.215+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feec41a8b10 con 0x7feec4109650 2024-07-26T08:36:58.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.217+0000 7feec2000640 1 -- 172.21.15.44:0/1404335275 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7feebc0cf680 con 0x7feec4109650 2024-07-26T08:36:58.219 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:36:58.219 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:36:58.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feeac0b3ca0 msgr2=0x7feeac0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:58.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feeac0b3ca0 0x7feeac0b61b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feeb8005fd0 tx=0x7feeb8009500 comp rx=0 tx=0).stop 2024-07-26T08:36:58.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 msgr2=0x7feec41a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:36:58.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 0x7feec41a1730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feebc004770 tx=0x7feebc004040 comp rx=0 tx=0).stop 2024-07-26T08:36:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 shutdown_connections 2024-07-26T08:36:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7feeac0b3ca0 0x7feeac0b61b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feec4109650 0x7feec41a1730 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feec4108d00 0x7feec419c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 --2- 172.21.15.44:0/1404335275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feec41033c0 0x7feec419bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:36:58.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.220+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 >> 172.21.15.44:0/1404335275 conn(0x7feec40fee10 msgr2=0x7feec4108390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:36:58.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.221+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 shutdown_connections 2024-07-26T08:36:58.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:36:58.221+0000 7feecae00640 1 -- 172.21.15.44:0/1404335275 wait complete. 2024-07-26T08:36:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:58 smithi044 ceph-mon[32404]: pgmap v431: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:58 smithi116 ceph-mon[37097]: pgmap v431: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:58 smithi171 ceph-mon[37789]: pgmap v431: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:36:59.181 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:36:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:36:59 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1404335275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:36:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:36:59 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1404335275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:36:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:36:59 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1404335275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:36:59.682 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:00 smithi044 ceph-mon[32404]: pgmap v432: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:00 smithi116 ceph-mon[37097]: pgmap v432: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:00 smithi171 ceph-mon[37789]: pgmap v432: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 -- 172.21.15.44:0/3902804782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 msgr2=0x7fdb00109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 --2- 172.21.15.44:0/3902804782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb00109e00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009900 tx=0x7fdaf402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 -- 172.21.15.44:0/3902804782 shutdown_connections 2024-07-26T08:37:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 --2- 172.21.15.44:0/3902804782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010fc80 0x7fdb001164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 --2- 172.21.15.44:0/3902804782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb0010f330 0x7fdb0010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 --2- 172.21.15.44:0/3902804782 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb00109e00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 -- 172.21.15.44:0/3902804782 >> 172.21.15.44:0/3902804782 conn(0x7fdb00078ff0 msgr2=0x7fdb00107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 -- 172.21.15.44:0/3902804782 shutdown_connections 2024-07-26T08:37:02.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.056+0000 7fdb06400640 1 -- 172.21.15.44:0/3902804782 wait complete. 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 Processor -- start 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 -- start start 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb001a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010f330 0x7fdb001a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb0010fc80 0x7fdb001a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb00119940 con 0x7fdb001099f0 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb001197c0 con 0x7fdb0010fc80 2024-07-26T08:37:02.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.058+0000 7fdb06400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdb00119ac0 con 0x7fdb0010f330 2024-07-26T08:37:02.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb001a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:02.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb001a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52142/0 (socket says 172.21.15.44:52142) 2024-07-26T08:37:02.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 -- 172.21.15.44:0/1294944323 learned_addr learned my addr 172.21.15.44:0/1294944323 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:02.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05a00640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb0010fc80 0x7fdb001a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:02.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdaffe00640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010f330 0x7fdb001a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 -- 172.21.15.44:0/1294944323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010f330 msgr2=0x7fdb001a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:02.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010f330 0x7fdb001a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 -- 172.21.15.44:0/1294944323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb0010fc80 msgr2=0x7fdb001a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb0010fc80 0x7fdb001a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdb05000640 1 -- 172.21.15.44:0/1294944323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb001a62e0 con 0x7fdb001099f0 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.059+0000 7fdaffe00640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010f330 0x7fdb001a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.060+0000 7fdb05000640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb001a0370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4005940 tx=0x7fdaf4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.060+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf403f030 con 0x7fdb001099f0 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.060+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb001a6570 con 0x7fdb001099f0 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.060+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaf4003aa0 con 0x7fdb001099f0 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.060+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4035070 con 0x7fdb001099f0 2024-07-26T08:37:02.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.060+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb001acd30 con 0x7fdb001099f0 2024-07-26T08:37:02.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.062+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb00109e00 con 0x7fdb001099f0 2024-07-26T08:37:02.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.066+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fdaf4042030 con 0x7fdb001099f0 2024-07-26T08:37:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.068+0000 7fdafd600640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdae80b3c20 0x7fdae80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.068+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fdaf4104990 con 0x7fdb001099f0 2024-07-26T08:37:02.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.068+0000 7fdaffe00640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdae80b3c20 0x7fdae80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:02.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.069+0000 7fdaffe00640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdae80b3c20 0x7fdae80b6130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdb0010fb30 tx=0x7fdaf000b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:02.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.070+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdaf40cb480 con 0x7fdb001099f0 2024-07-26T08:37:02.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.313+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb001a14a0 con 0x7fdb001099f0 2024-07-26T08:37:02.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.315+0000 7fdafd600640 1 -- 172.21.15.44:0/1294944323 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7fdb001a14a0 con 0x7fdb001099f0 2024-07-26T08:37:02.317 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:02.317 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"btime":"2024-07-26T08:34:37:118441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:34:36.302968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24661,24668,24679,24686,24701]},"id":1}]} 2024-07-26T08:37:02.317 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdae80b3c20 msgr2=0x7fdae80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdae80b3c20 0x7fdae80b6130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdb0010fb30 tx=0x7fdaf000b390 comp rx=0 tx=0).stop 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 msgr2=0x7fdb001a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb001a0370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4005940 tx=0x7fdaf4004270 comp rx=0 tx=0).stop 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 shutdown_connections 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdae80b3c20 0x7fdae80b6130 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb0010f330 0x7fdb001a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb0010fc80 0x7fdb001a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 --2- 172.21.15.44:0/1294944323 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb001099f0 0x7fdb001a0370 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 >> 172.21.15.44:0/1294944323 conn(0x7fdb00078ff0 msgr2=0x7fdb00107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:02.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 shutdown_connections 2024-07-26T08:37:02.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:02.318+0000 7fdb06400640 1 -- 172.21.15.44:0/1294944323 wait complete. 2024-07-26T08:37:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:02 smithi044 ceph-mon[32404]: pgmap v433: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:02 smithi116 ceph-mon[37097]: pgmap v433: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:02 smithi171 ceph-mon[37789]: pgmap v433: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:03.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T08:33:06.110522+0000', 'modified': '2024-07-26T08:34:36.302968+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24701, 'mds_1': 24686, 'mds_2': 24679, 'mds_3': 24668, 'mds_4': 24661}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24613': {'gid': 24613, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6833/4260884949', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6832', 'nonce': 4260884949}, {'type': 'v1', 'addr': '172.21.15.171:6833', 'nonce': 4260884949}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24620': {'gid': 24620, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6833/2097534847', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6832', 'nonce': 2097534847}, {'type': 'v1', 'addr': '172.21.15.116:6833', 'nonce': 2097534847}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24638': {'gid': 24638, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.44:6837/3938332919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6836', 'nonce': 3938332919}, {'type': 'v1', 'addr': '172.21.15.44:6837', 'nonce': 3938332919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24650': {'gid': 24650, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/1052813979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 1052813979}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 1052813979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24655': {'gid': 24655, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6835/372796843', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6834', 'nonce': 372796843}, {'type': 'v1', 'addr': '172.21.15.171:6835', 'nonce': 372796843}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24661': {'gid': 24661, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.171:6837/3462225872', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6836', 'nonce': 3462225872}, {'type': 'v1', 'addr': '172.21.15.171:6837', 'nonce': 3462225872}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24668': {'gid': 24668, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.116:6837/2715773427', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 2715773427}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2715773427}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24679': {'gid': 24679, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.171:6839/2065417578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6838', 'nonce': 2065417578}, {'type': 'v1', 'addr': '172.21.15.171:6839', 'nonce': 2065417578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24686': {'gid': 24686, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 30, 'addr': '172.21.15.44:6839/1144072288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6838', 'nonce': 1144072288}, {'type': 'v1', 'addr': '172.21.15.44:6839', 'nonce': 1144072288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24701': {'gid': 24701, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6839/3153223603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 3153223603}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 3153223603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24701, 'qdb_cluster': [24661, 24668, 24679, 24686, 24701]} 2024-07-26T08:37:03.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T08:37:03.229 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.c status 2024-07-26T08:37:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:03 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1294944323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:03 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1294944323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:03 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1294944323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:03.724 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:04 smithi044 ceph-mon[32404]: pgmap v434: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:37:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:04 smithi116 ceph-mon[37097]: pgmap v434: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:37:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:04 smithi171 ceph-mon[37789]: pgmap v434: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:37:06.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 -- 172.21.15.44:0/3665339871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a701040c0 msgr2=0x7f2a701044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 --2- 172.21.15.44:0/3665339871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a701040c0 0x7f2a701044d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009860 tx=0x7f2a6002a340 comp rx=0 tx=0).stop 2024-07-26T08:37:06.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 -- 172.21.15.44:0/3665339871 shutdown_connections 2024-07-26T08:37:06.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 --2- 172.21.15.44:0/3665339871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a7010d670 0x7f2a7010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 --2- 172.21.15.44:0/3665339871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a70104af0 0x7f2a7010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 --2- 172.21.15.44:0/3665339871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a701040c0 0x7f2a701044d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 -- 172.21.15.44:0/3665339871 >> 172.21.15.44:0/3665339871 conn(0x7f2a700fdcf0 msgr2=0x7f2a70100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:06.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 -- 172.21.15.44:0/3665339871 shutdown_connections 2024-07-26T08:37:06.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.207+0000 7f2a75c00640 1 -- 172.21.15.44:0/3665339871 wait complete. 2024-07-26T08:37:06.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.208+0000 7f2a75c00640 1 Processor -- start 2024-07-26T08:37:06.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 -- start start 2024-07-26T08:37:06.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a701040c0 0x7f2a7019bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70104af0 0x7f2a7019c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a7010d670 0x7f2a701a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2a701132a0 con 0x7f2a701040c0 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a70113120 con 0x7f2a70104af0 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.209+0000 7f2a75c00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2a70113420 con 0x7f2a7010d670 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a701040c0 0x7f2a7019bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70104af0 0x7f2a7019c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70104af0 0x7f2a7019c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34224/0 (socket says 172.21.15.44:34224) 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6f400640 1 -- 172.21.15.44:0/253999555 learned_addr learned my addr 172.21.15.44:0/253999555 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:06.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a75200640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a7010d670 0x7f2a701a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6fe00640 1 -- 172.21.15.44:0/253999555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a7010d670 msgr2=0x7f2a701a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a7010d670 0x7f2a701a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6fe00640 1 -- 172.21.15.44:0/253999555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70104af0 msgr2=0x7f2a7019c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70104af0 0x7f2a7019c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.210+0000 7f2a6fe00640 1 -- 172.21.15.44:0/253999555 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a701a1e40 con 0x7f2a701040c0 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.211+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a701040c0 0x7f2a7019bed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a60035040 tx=0x7f2a60032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.211+0000 7f2a6cc00640 1 -- 172.21.15.44:0/253999555 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60040020 con 0x7f2a701040c0 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.211+0000 7f2a6cc00640 1 -- 172.21.15.44:0/253999555 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a60030d50 con 0x7f2a701040c0 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.211+0000 7f2a6cc00640 1 -- 172.21.15.44:0/253999555 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60037040 con 0x7f2a701040c0 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.211+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a701a20d0 con 0x7f2a701040c0 2024-07-26T08:37:06.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.211+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a701a8890 con 0x7f2a701040c0 2024-07-26T08:37:06.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.215+0000 7f2a7ddca740 1 Processor -- start 2024-07-26T08:37:06.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.215+0000 7f2a7ddca740 1 -- start start 2024-07-26T08:37:06.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589780ff780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x5589780ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d39280 0x558978100200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558978075970 con 0x558977d310a0 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558978064e70 con 0x558977d59f60 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558978068eb0 con 0x558977d39280 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a6cc00640 1 -- 172.21.15.44:0/253999555 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f2a60043020 con 0x7f2a701040c0 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a75200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589780ff780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a75200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589780ff780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34234/0 (socket says 172.21.15.44:34234) 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a75200640 1 -- 172.21.15.44:0/1356319924 learned_addr learned my addr 172.21.15.44:0/1356319924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:06.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x5589780ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.216+0000 7f2a6f400640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d39280 0x558978100200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a75200640 1 -- 172.21.15.44:0/1356319924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d39280 msgr2=0x558978100200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a75200640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d39280 0x558978100200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a75200640 1 -- 172.21.15.44:0/1356319924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 msgr2=0x5589780ffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a75200640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x5589780ffcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a75200640 1 -- 172.21.15.44:0/1356319924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897800c110 con 0x558977d59f60 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a6f400640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d39280 0x558978100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:06.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x5589780ffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:06.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.217+0000 7f2a75200640 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589780ff780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009fd0 tx=0x7f2a680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9988a8634feb08a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:06.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.218+0000 7f2a6cc00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2a540b3ca0 0x7f2a540b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.218+0000 7f2a6cc00640 1 -- 172.21.15.44:0/253999555 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2a60103d00 con 0x7f2a701040c0 2024-07-26T08:37:06.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.219+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2a540b3ca0 0x7f2a540b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.219+0000 7f2a6fe00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2a540b3ca0 0x7f2a540b61b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2a600ca440 tx=0x7f2a60005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:06.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.222+0000 7f2a65600640 1 -- 172.21.15.44:0/1356319924 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68004d20 con 0x558977d59f60 2024-07-26T08:37:06.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.222+0000 7f2a65600640 1 -- 172.21.15.44:0/1356319924 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a68004050 con 0x558977d59f60 2024-07-26T08:37:06.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.222+0000 7f2a65600640 1 -- 172.21.15.44:0/1356319924 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68005280 con 0x558977d59f60 2024-07-26T08:37:06.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 -- 172.21.15.44:0/1356319924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 msgr2=0x5589780ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589780ff780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009fd0 tx=0x7f2a680118d0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 -- 172.21.15.44:0/1356319924 shutdown_connections 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d39280 0x558978100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x5589780ffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/1356319924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589780ff780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 -- 172.21.15.44:0/1356319924 >> 172.21.15.44:0/1356319924 conn(0x558977d4e350 msgr2=0x5589780fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 -- 172.21.15.44:0/1356319924 shutdown_connections 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.223+0000 7f2a7ddca740 1 -- 172.21.15.44:0/1356319924 wait complete. 2024-07-26T08:37:06.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.224+0000 7f2a7ddca740 1 -- start start 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.224+0000 7f2a7ddca740 1 Processor -- start 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.224+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d310a0 0x55897810b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897810b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589781172b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558977ec85f0 con 0x558977d39280 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558978001540 con 0x558977d59f60 2024-07-26T08:37:06.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55897810df60 con 0x558977d310a0 2024-07-26T08:37:06.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a7ddca740 10 client.0 _ll_register_callbacks cb 0x558978010f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:06.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897810b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a75200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d310a0 0x55897810b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897810b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52158/0 (socket says 172.21.15.44:52158) 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 -- 172.21.15.44:0/3500433496 learned_addr learned my addr 172.21.15.44:0/3500433496 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a75200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d310a0 0x55897810b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:51164/0 (socket says 172.21.15.44:51164) 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a75200640 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589781172b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 -- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d310a0 msgr2=0x55897810b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d310a0 0x55897810b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 -- 172.21.15.44:0/3500433496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 msgr2=0x5589781172b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589781172b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.225+0000 7f2a6f400640 1 -- 172.21.15.44:0/3500433496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558977dc5540 con 0x558977d39280 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.226+0000 7f2a6f400640 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897810b5e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c0116a0 tx=0x7f2a5c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872d49ec3fed786b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.226+0000 7f2a5a000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:37:06.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.226+0000 7f2a5a000640 1 -- 172.21.15.44:0/3500433496 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c01e020 con 0x558977d39280 2024-07-26T08:37:06.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.226+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a68004050 con 0x558977d39280 2024-07-26T08:37:06.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.226+0000 7f2a5a000640 1 -- 172.21.15.44:0/3500433496 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2a5c00fc70 con 0x558977d39280 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a5a000640 1 -- 172.21.15.44:0/3500433496 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a5c026050 con 0x558977d39280 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 10 client.15024 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558978119f10 con 0x558977d39280 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a5a000640 1 -- 172.21.15.44:0/3500433496 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a68004050 con 0x558977d39280 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a5a000640 1 -- 172.21.15.44:0/3500433496 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2a5c019070 con 0x558977d39280 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 10 client.15024 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 10 client.15024 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T08:37:06.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 20 client.15024 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:06.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977d188e0 0x55897811d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 4 client.15024 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:37:06.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.227+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589780778c0 con 0x558977d188e0 2024-07-26T08:37:06.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.228+0000 7f2a75200640 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977d188e0 0x55897811d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.228+0000 7f2a75200640 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977d188e0 0x55897811d9e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=556d6c49b00c899e server_cookie=6846f981f63358c9 in_seq=0 out_seq=0 2024-07-26T08:37:06.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.228+0000 7f2a5a000640 10 client.15024 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:37:06.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.229+0000 7f2a5a000640 1 -- 172.21.15.44:0/3500433496 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5589780778c0 con 0x558977d188e0 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.229+0000 7f2a5a000640 10 client.15024 handle_command_reply: tid=0 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.229+0000 7f2a7ddca740 1 client.15024 shutdown 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 msgr2=0x55897810b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897810b5e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c0116a0 tx=0x7f2a5c011b60 comp rx=0 tx=0).stop 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 shutdown_connections 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977d188e0 0x55897811d9e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d59f60 0x5589781172b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897810b5e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/3500433496 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d310a0 0x55897810b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 >> 172.21.15.44:0/3500433496 conn(0x558977d4e350 msgr2=0x558978102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a5a000640 0 client.15024 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:37:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 shutdown_connections 2024-07-26T08:37:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 1 -- 172.21.15.44:0/3500433496 wait complete. 2024-07-26T08:37:06.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.230+0000 7f2a7ddca740 20 client.15024 trim_cache size 0 max 16384 2024-07-26T08:37:06.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.251+0000 7f2a7ddca740 1 Processor -- start 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.251+0000 7f2a7ddca740 1 -- start start 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x55897811d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 0x5589780fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x5589781058a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55897811bd20 con 0x558977d310a0 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55897811c3f0 con 0x558977d39280 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55897811cac0 con 0x558977d59f60 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 0x5589780fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a75200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x55897811d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 0x5589780fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34254/0 (socket says 172.21.15.44:34254) 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.252+0000 7f2a6f400640 1 -- 172.21.15.44:0/2181273555 learned_addr learned my addr 172.21.15.44:0/2181273555 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x5589781058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 -- 172.21.15.44:0/2181273555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 msgr2=0x5589781058a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x5589781058a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 -- 172.21.15.44:0/2181273555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 msgr2=0x5589780fd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 0x5589780fd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 -- 172.21.15.44:0/2181273555 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558978109f80 con 0x558977d310a0 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a6f400640 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 0x5589780fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.253+0000 7f2a75200640 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x55897811d310 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009fa0 tx=0x7f2a68000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d831d62eff0564d6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.254+0000 7f2a64c00640 1 -- 172.21.15.44:0/2181273555 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68004970 con 0x558977d310a0 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.254+0000 7f2a64c00640 1 -- 172.21.15.44:0/2181273555 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a6802c030 con 0x558977d310a0 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.254+0000 7f2a64c00640 1 -- 172.21.15.44:0/2181273555 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68005330 con 0x558977d310a0 2024-07-26T08:37:06.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 -- 172.21.15.44:0/2181273555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 msgr2=0x55897811d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x55897811d310 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009fa0 tx=0x7f2a68000f40 comp rx=0 tx=0).stop 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 -- 172.21.15.44:0/2181273555 shutdown_connections 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x5589781058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d39280 0x5589780fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/2181273555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d310a0 0x55897811d310 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 -- 172.21.15.44:0/2181273555 >> 172.21.15.44:0/2181273555 conn(0x558977d4e350 msgr2=0x5589780fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 -- 172.21.15.44:0/2181273555 shutdown_connections 2024-07-26T08:37:06.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.255+0000 7f2a7ddca740 1 -- 172.21.15.44:0/2181273555 wait complete. 2024-07-26T08:37:06.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.256+0000 7f2a7ddca740 1 -- start start 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.256+0000 7f2a7ddca740 1 Processor -- start 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d310a0 0x558978126320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897811ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x55897811f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55897811d860 con 0x558977d39280 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55897811d190 con 0x558977d310a0 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55897811df30 con 0x558977d59f60 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a7ddca740 10 client.0 _ll_register_callbacks cb 0x7f2a5c009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a75200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d310a0 0x558978126320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a75200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d310a0 0x558978126320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34268/0 (socket says 172.21.15.44:34268) 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a75200640 1 -- 172.21.15.44:0/68812420 learned_addr learned my addr 172.21.15.44:0/68812420 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:06.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a75200640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x55897811f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a6f400640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897811ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a75200640 1 -- 172.21.15.44:0/68812420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d310a0 msgr2=0x558978126320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.257+0000 7f2a75200640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d310a0 0x558978126320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a75200640 1 -- 172.21.15.44:0/68812420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 msgr2=0x55897811ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a75200640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897811ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a75200640 1 -- 172.21.15.44:0/68812420 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897811f860 con 0x558977d59f60 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a75200640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x55897811f120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005fd0 tx=0x7f2a68003f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48db455b43e0eea2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a5a000640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a5a000640 1 -- 172.21.15.44:0/68812420 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68004a50 con 0x558977d59f60 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a6802c030 con 0x558977d59f60 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.258+0000 7f2a5a000640 1 -- 172.21.15.44:0/68812420 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2a68004bf0 con 0x558977d59f60 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a5a000640 1 -- 172.21.15.44:0/68812420 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a68049050 con 0x558977d59f60 2024-07-26T08:37:06.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 10 client.25159 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558978185b60 con 0x558977d59f60 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a5a000640 1 -- 172.21.15.44:0/68812420 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a6802c030 con 0x558977d59f60 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a5a000640 1 -- 172.21.15.44:0/68812420 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2a6800f5e0 con 0x558977d59f60 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 10 client.25159 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 10 client.25159 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 20 client.25159 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977cedc90 0x558978188e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 4 client.25159 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.259+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "status"}) -- 0x558978073360 con 0x558977cedc90 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.260+0000 7f2a6f400640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977cedc90 0x558978188e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.260+0000 7f2a6f400640 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977cedc90 0x558978188e80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f09711b859c051c1 server_cookie=f050e4b5dc6f0740 in_seq=0 out_seq=0 2024-07-26T08:37:06.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.260+0000 7f2a5a000640 10 client.25159 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a5a000640 1 -- 172.21.15.44:0/68812420 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558978073360 con 0x558977cedc90 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a5a000640 10 client.25159 handle_command_reply: tid=0 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 client.25159 shutdown 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 msgr2=0x55897811f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x55897811f120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a68005fd0 tx=0x7f2a68003f50 comp rx=0 tx=0).stop 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 shutdown_connections 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558977d59f60 0x55897811f120 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x558977cedc90 0x558978188e80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558977d39280 0x55897811ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 --2- 172.21.15.44:0/68812420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558977d310a0 0x558978126320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 >> 172.21.15.44:0/68812420 conn(0x558977d4e350 msgr2=0x55897810add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:06.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a5a000640 0 client.25159 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 shutdown_connections 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 1 -- 172.21.15.44:0/68812420 wait complete. 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.261+0000 7f2a7ddca740 20 client.25159 trim_cache size 0 max 16384 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 4, 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24661, 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:37:06.267 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 155.17655085000001, 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 280.96618403600002 2024-07-26T08:37:06.268 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2a540b3ca0 msgr2=0x7f2a540b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2a540b3ca0 0x7f2a540b61b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2a600ca440 tx=0x7f2a60005f90 comp rx=0 tx=0).stop 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a701040c0 msgr2=0x7f2a7019bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a701040c0 0x7f2a7019bed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2a60035040 tx=0x7f2a60032d50 comp rx=0 tx=0).stop 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 shutdown_connections 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2a540b3ca0 0x7f2a540b61b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a7010d670 0x7f2a701a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a70104af0 0x7f2a7019c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 --2- 172.21.15.44:0/253999555 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2a701040c0 0x7f2a7019bed0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.262+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 >> 172.21.15.44:0/253999555 conn(0x7f2a700fdcf0 msgr2=0x7f2a70101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.263+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 shutdown_connections 2024-07-26T08:37:06.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:06.263+0000 7f2a75c00640 1 -- 172.21.15.44:0/253999555 wait complete. 2024-07-26T08:37:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:06 smithi044 ceph-mon[32404]: pgmap v435: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:06 smithi116 ceph-mon[37097]: pgmap v435: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:06 smithi171 ceph-mon[37789]: pgmap v435: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:07.208 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.h status 2024-07-26T08:37:07.729 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:08 smithi044 ceph-mon[32404]: pgmap v436: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:08 smithi116 ceph-mon[37097]: pgmap v436: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:08 smithi171 ceph-mon[37789]: pgmap v436: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:10.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.112+0000 7fea15e00640 1 -- 172.21.15.44:0/16574518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 msgr2=0x7fea100755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.112+0000 7fea15e00640 1 --2- 172.21.15.44:0/16574518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea100755f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009900 tx=0x7fe9f802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.112+0000 7fea15e00640 1 -- 172.21.15.44:0/16574518 shutdown_connections 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.112+0000 7fea15e00640 1 --2- 172.21.15.44:0/16574518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea1010faf0 0x7fea10111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.112+0000 7fea15e00640 1 --2- 172.21.15.44:0/16574518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea10075b30 0x7fea1010f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.112+0000 7fea15e00640 1 --2- 172.21.15.44:0/16574518 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea100755f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.113+0000 7fea15e00640 1 -- 172.21.15.44:0/16574518 >> 172.21.15.44:0/16574518 conn(0x7fea1007b280 msgr2=0x7fea1007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.113+0000 7fea15e00640 1 -- 172.21.15.44:0/16574518 shutdown_connections 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.113+0000 7fea15e00640 1 -- 172.21.15.44:0/16574518 wait complete. 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.114+0000 7fea15e00640 1 Processor -- start 2024-07-26T08:37:10.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 -- start start 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea1019c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea10075b30 0x7fea1019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea1010faf0 0x7fea101a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea101156f0 con 0x7fea1010faf0 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea10115570 con 0x7fea10075b30 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea15e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fea10115870 con 0x7fea100751e0 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.115+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea1019c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea1019c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:51206/0 (socket says 172.21.15.44:51206) 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 -- 172.21.15.44:0/2082403830 learned_addr learned my addr 172.21.15.44:0/2082403830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0f400640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea1010faf0 0x7fea101a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0e000640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea10075b30 0x7fea1019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 -- 172.21.15.44:0/2082403830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea10075b30 msgr2=0x7fea1019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea10075b30 0x7fea1019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 -- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea1010faf0 msgr2=0x7fea101a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea1010faf0 0x7fea101a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 -- 172.21.15.44:0/2082403830 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea101a1ff0 con 0x7fea100751e0 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0e000640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea10075b30 0x7fea1019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0f400640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea1010faf0 0x7fea101a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:10.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.116+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea1019c080 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8005ab0 tx=0x7fe9f8003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:10.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.117+0000 7fea14a00640 1 -- 172.21.15.44:0/2082403830 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f803f070 con 0x7fea100751e0 2024-07-26T08:37:10.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.117+0000 7fea14a00640 1 -- 172.21.15.44:0/2082403830 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f8031d60 con 0x7fea100751e0 2024-07-26T08:37:10.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.117+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea101a2280 con 0x7fea100751e0 2024-07-26T08:37:10.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.117+0000 7fea14a00640 1 -- 172.21.15.44:0/2082403830 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8035030 con 0x7fea100751e0 2024-07-26T08:37:10.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.117+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea101a8a40 con 0x7fea100751e0 2024-07-26T08:37:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.121+0000 7fea1cc21740 1 Processor -- start 2024-07-26T08:37:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.121+0000 7fea1cc21740 1 -- start start 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.121+0000 7fea14a00640 1 -- 172.21.15.44:0/2082403830 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fe9f8042070 con 0x7fea100751e0 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.121+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9d82570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d82ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99bc280 0x55fed9d82ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea1cc21740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fed9cf8970 con 0x55fed99bc280 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea1cc21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fed9ce7e70 con 0x55fed99dcf60 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea1cc21740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fed9cebeb0 con 0x55fed99b40a0 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d82ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea0f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9d82570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d82ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:51218/0 (socket says 172.21.15.44:51218) 2024-07-26T08:37:10.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea0e000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99bc280 0x55fed9d82ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.122+0000 7fea0ea00640 1 -- 172.21.15.44:0/1042341917 learned_addr learned my addr 172.21.15.44:0/1042341917 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0ea00640 1 -- 172.21.15.44:0/1042341917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 msgr2=0x55fed9d82570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0ea00640 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9d82570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0ea00640 1 -- 172.21.15.44:0/1042341917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99bc280 msgr2=0x55fed9d82ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0ea00640 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99bc280 0x55fed9d82ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0ea00640 1 -- 172.21.15.44:0/1042341917 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fed9c8f110 con 0x55fed99b40a0 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0f400640 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9d82570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0e000640 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99bc280 0x55fed9d82ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea0ea00640 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d82ab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9f810ab90 tx=0x7fe9f810ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=435cbdcbedf0935 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fea14a00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9f40b3ca0 0x7fe9f40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fe9ffe00640 1 -- 172.21.15.44:0/1042341917 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8113070 con 0x55fed99b40a0 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.123+0000 7fe9ffe00640 1 -- 172.21.15.44:0/1042341917 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f80caf40 con 0x55fed99b40a0 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.124+0000 7fe9ffe00640 1 -- 172.21.15.44:0/1042341917 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f80cb430 con 0x55fed99b40a0 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.124+0000 7fea14a00640 1 -- 172.21.15.44:0/2082403830 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fe9f8104990 con 0x7fea100751e0 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.124+0000 7fea0f400640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9f40b3ca0 0x7fe9f40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 -- 172.21.15.44:0/1042341917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 msgr2=0x55fed9d82ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d82ab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9f810ab90 tx=0x7fe9f810ac80 comp rx=0 tx=0).stop 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea0f400640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9f40b3ca0 0x7fe9f40b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fea08009e80 tx=0x7fea08005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 -- 172.21.15.44:0/1042341917 shutdown_connections 2024-07-26T08:37:10.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d82ab0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99bc280 0x55fed9d82ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 --2- 172.21.15.44:0/1042341917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9d82570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 -- 172.21.15.44:0/1042341917 >> 172.21.15.44:0/1042341917 conn(0x55fed99d1350 msgr2=0x55fed9d80460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:10.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 -- 172.21.15.44:0/1042341917 shutdown_connections 2024-07-26T08:37:10.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.125+0000 7fea1cc21740 1 -- 172.21.15.44:0/1042341917 wait complete. 2024-07-26T08:37:10.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.126+0000 7fea1cc21740 1 -- start start 2024-07-26T08:37:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.126+0000 7fea1cc21740 1 Processor -- start 2024-07-26T08:37:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.126+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9d8dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d8e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 0x55fed9d99d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea1cc21740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fed9b4b5f0 con 0x55fed99b40a0 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea1cc21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fed9c84540 con 0x55fed99bc280 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea1cc21740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fed9d90b10 con 0x55fed99dcf60 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea1cc21740 10 client.0 _ll_register_callbacks cb 0x55fed9c93f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 0x55fed9d99d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9d8dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 0x55fed9d99d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:51226/0 (socket says 172.21.15.44:51226) 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d8e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9d8dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52214/0 (socket says 172.21.15.44:52214) 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 -- 172.21.15.44:0/2925712751 learned_addr learned my addr 172.21.15.44:0/2925712751 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 -- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 msgr2=0x55fed9d8e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d8e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 -- 172.21.15.44:0/2925712751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 msgr2=0x55fed9d8dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.127+0000 7fea0f400640 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9d8dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fea0f400640 1 -- 172.21.15.44:0/2925712751 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fed9a48540 con 0x55fed99dcf60 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9d8dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fea0f400640 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 0x55fed9d99d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fea080209e0 tx=0x7fea08004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96b14107c3864a6e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fe9fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2925712751 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0802d060 con 0x55fed99dcf60 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f80caf40 con 0x55fed99dcf60 2024-07-26T08:37:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.128+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2925712751 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fea08018030 con 0x55fed99dcf60 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.129+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2925712751 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea08034050 con 0x55fed99dcf60 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.129+0000 7fea1cc21740 10 client.25171 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.129+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fed9d9c9c0 con 0x55fed99dcf60 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.129+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2925712751 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f80caf40 con 0x55fed99dcf60 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.129+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2925712751 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fea08028a00 con 0x55fed99dcf60 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.129+0000 7fea1cc21740 10 client.25171 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.130+0000 7fea1cc21740 10 client.25171 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.130+0000 7fea1cc21740 20 client.25171 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.130+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed999b8e0 0x55fed9da0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.130+0000 7fea1cc21740 4 client.25171 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:37:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.130+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fed9cfa8c0 con 0x55fed999b8e0 2024-07-26T08:37:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.130+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed999b8e0 0x55fed9da0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.131+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed999b8e0 0x55fed9da0450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4dbb74f5650ac5cc server_cookie=9602c2464638a14c in_seq=0 out_seq=0 2024-07-26T08:37:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.131+0000 7fe9fcc00640 10 client.25171 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:37:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.131+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2925712751 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fed9cfa8c0 con 0x55fed999b8e0 2024-07-26T08:37:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.131+0000 7fe9fcc00640 10 client.25171 handle_command_reply: tid=0 2024-07-26T08:37:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 client.25171 shutdown 2024-07-26T08:37:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 msgr2=0x55fed9d99d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 0x55fed9d99d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fea080209e0 tx=0x7fea08004730 comp rx=0 tx=0).stop 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 shutdown_connections 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed999b8e0 0x55fed9da0450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99dcf60 0x55fed9d99d70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d8e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fe9fcc00640 0 client.25171 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2925712751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9d8dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 >> 172.21.15.44:0/2925712751 conn(0x55fed99d1350 msgr2=0x55fed9d85750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 shutdown_connections 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 1 -- 172.21.15.44:0/2925712751 wait complete. 2024-07-26T08:37:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.132+0000 7fea1cc21740 20 client.25171 trim_cache size 0 max 16384 2024-07-26T08:37:10.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.152+0000 7fea1cc21740 1 Processor -- start 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.152+0000 7fea1cc21740 1 -- start start 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d9f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d88690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99dcf60 0x55fed9d88f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea1cc21740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fed9d9e790 con 0x55fed99dcf60 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea1cc21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fed9d9ee60 con 0x55fed99bc280 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea1cc21740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fed9d9f530 con 0x55fed99b40a0 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d9f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d9f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:51238/0 (socket says 172.21.15.44:51238) 2024-07-26T08:37:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea0ea00640 1 -- 172.21.15.44:0/624912410 learned_addr learned my addr 172.21.15.44:0/624912410 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea0f400640 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99dcf60 0x55fed9d88f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.153+0000 7fea0e000640 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d88690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fea0ea00640 1 -- 172.21.15.44:0/624912410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 msgr2=0x55fed9d88690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fea0ea00640 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d88690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fea0ea00640 1 -- 172.21.15.44:0/624912410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99dcf60 msgr2=0x55fed9d88f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fea0ea00640 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99dcf60 0x55fed9d88f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fea0ea00640 1 -- 172.21.15.44:0/624912410 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fed9d84b80 con 0x55fed99b40a0 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fea0ea00640 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d9f6b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8102030 tx=0x7fe9f803b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce04f96e1920864 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fe9ff400640 1 -- 172.21.15.44:0/624912410 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8049030 con 0x55fed99b40a0 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fe9ff400640 1 -- 172.21.15.44:0/624912410 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f80caf40 con 0x55fed99b40a0 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.154+0000 7fe9ff400640 1 -- 172.21.15.44:0/624912410 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8113070 con 0x55fed99b40a0 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 -- 172.21.15.44:0/624912410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 msgr2=0x55fed9d9f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d9f6b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8102030 tx=0x7fe9f803b460 comp rx=0 tx=0).stop 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 -- 172.21.15.44:0/624912410 shutdown_connections 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99dcf60 0x55fed9d88f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99bc280 0x55fed9d88690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 --2- 172.21.15.44:0/624912410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99b40a0 0x55fed9d9f6b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.155+0000 7fea1cc21740 1 -- 172.21.15.44:0/624912410 >> 172.21.15.44:0/624912410 conn(0x55fed99d1350 msgr2=0x55fed9d7ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:10.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.156+0000 7fea1cc21740 1 -- 172.21.15.44:0/624912410 shutdown_connections 2024-07-26T08:37:10.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.156+0000 7fea1cc21740 1 -- 172.21.15.44:0/624912410 wait complete. 2024-07-26T08:37:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.156+0000 7fea1cc21740 1 -- start start 2024-07-26T08:37:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.156+0000 7fea1cc21740 1 Processor -- start 2024-07-26T08:37:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9da8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 0x55fed9da6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9da6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fed9da02d0 con 0x55fed99b40a0 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fed9d9fc00 con 0x55fed99dcf60 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fed9da09a0 con 0x55fed99bc280 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea1cc21740 10 client.0 _ll_register_callbacks cb 0x7fea080208a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea0e000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 0x55fed9da6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea0ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9da8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea0e000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 0x55fed9da6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:51242/0 (socket says 172.21.15.44:51242) 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.157+0000 7fea0e000640 1 -- 172.21.15.44:0/2841658166 learned_addr learned my addr 172.21.15.44:0/2841658166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0f400640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9da6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0e000640 1 -- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 msgr2=0x55fed9da6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0e000640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9da6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0e000640 1 -- 172.21.15.44:0/2841658166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 msgr2=0x55fed9da8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0e000640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9da8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0e000640 1 -- 172.21.15.44:0/2841658166 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fed9da72b0 con 0x55fed99bc280 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0f400640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9da6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9da8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fea0e000640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 0x55fed9da6800 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea00012c70 tx=0x7fea000117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aab3dc8361f8d98 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.158+0000 7fe9fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2841658166 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00003b70 con 0x55fed99bc280 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2841658166 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fea00003d10 con 0x55fed99bc280 2024-07-26T08:37:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f80caf40 con 0x55fed99bc280 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2841658166 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea00028050 con 0x55fed99bc280 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fea1cc21740 10 client.25177 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fed9e08460 con 0x55fed99bc280 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.159+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2841658166 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9f80caf40 con 0x55fed99bc280 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2841658166 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fea0001c070 con 0x55fed99bc280 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea1cc21740 10 client.25177 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea1cc21740 10 client.25177 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea1cc21740 20 client.25177 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed9970c90 0x55fed9e0bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea1cc21740 4 client.25177 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:37:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "status"}) -- 0x55fed9cf6360 con 0x55fed9970c90 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.160+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed9970c90 0x55fed9e0bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea0ea00640 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed9970c90 0x55fed9e0bde0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f5a6f451f14c63d1 server_cookie=8e5f4ae5eae1b611 in_seq=0 out_seq=0 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fe9fcc00640 10 client.25177 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fe9fcc00640 1 -- 172.21.15.44:0/2841658166 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fed9cf6360 con 0x55fed9970c90 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fe9fcc00640 10 client.25177 handle_command_reply: tid=0 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 client.25177 shutdown 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 msgr2=0x55fed9da6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 0x55fed9da6800 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea00012c70 tx=0x7fea000117b0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 shutdown_connections 2024-07-26T08:37:10.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55fed9970c90 0x55fed9e0bde0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fed99bc280 0x55fed9da6800 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fed99dcf60 0x55fed9da6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 --2- 172.21.15.44:0/2841658166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fed99b40a0 0x55fed9da8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fe9fcc00640 0 client.25177 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.161+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 >> 172.21.15.44:0/2841658166 conn(0x55fed99d1350 msgr2=0x55fed9d88f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.162+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 shutdown_connections 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.162+0000 7fea1cc21740 1 -- 172.21.15.44:0/2841658166 wait complete. 2024-07-26T08:37:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.162+0000 7fea1cc21740 20 client.25177 trim_cache size 0 max 16384 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 3, 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24668, 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:37:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:37:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 160.08887580499999, 2024-07-26T08:37:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:37:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:37:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:37:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 282.22575038500003 2024-07-26T08:37:10.171 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9f40b3ca0 msgr2=0x7fe9f40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9f40b3ca0 0x7fe9f40b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fea08009e80 tx=0x7fea08005f90 comp rx=0 tx=0).stop 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 msgr2=0x7fea1019c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea1019c080 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8005ab0 tx=0x7fe9f8003c80 comp rx=0 tx=0).stop 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 shutdown_connections 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fe9f40b3ca0 0x7fe9f40b61b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea1010faf0 0x7fea101a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea10075b30 0x7fea1019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 --2- 172.21.15.44:0/2082403830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea100751e0 0x7fea1019c080 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:10.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 >> 172.21.15.44:0/2082403830 conn(0x7fea1007b280 msgr2=0x7fea10101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:10.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 shutdown_connections 2024-07-26T08:37:10.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:10.163+0000 7fea15e00640 1 -- 172.21.15.44:0/2082403830 wait complete. 2024-07-26T08:37:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:10 smithi044 ceph-mon[32404]: pgmap v437: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:10 smithi116 ceph-mon[37097]: pgmap v437: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:10 smithi171 ceph-mon[37789]: pgmap v437: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:11.107 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.i status 2024-07-26T08:37:11.617 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:12 smithi044 ceph-mon[32404]: pgmap v438: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:12 smithi116 ceph-mon[37097]: pgmap v438: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:12 smithi171 ceph-mon[37789]: pgmap v438: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 -- 172.21.15.44:0/3481931513 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c1055d0 msgr2=0x7f0a6c1059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3481931513 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c1059e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009900 tx=0x7f0a5c02a410 comp rx=0 tx=0).stop 2024-07-26T08:37:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 -- 172.21.15.44:0/3481931513 shutdown_connections 2024-07-26T08:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3481931513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c10b860 0x7f0a6c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3481931513 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c10af10 0x7f0a6c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3481931513 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c1059e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 -- 172.21.15.44:0/3481931513 >> 172.21.15.44:0/3481931513 conn(0x7f0a6c101040 msgr2=0x7f0a6c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 -- 172.21.15.44:0/3481931513 shutdown_connections 2024-07-26T08:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.052+0000 7f0a6be00640 1 -- 172.21.15.44:0/3481931513 wait complete. 2024-07-26T08:37:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 Processor -- start 2024-07-26T08:37:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 -- start start 2024-07-26T08:37:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c10af10 0x7f0a6c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c10b860 0x7f0a6c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6c115560 con 0x7f0a6c1055d0 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6c1153e0 con 0x7f0a6c10af10 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.054+0000 7f0a6be00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6c1156e0 con 0x7f0a6c10b860 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6b400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c10b860 0x7f0a6c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54762/0 (socket says 172.21.15.44:54762) 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a62000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c10af10 0x7f0a6c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6aa00640 1 -- 172.21.15.44:0/3691488406 learned_addr learned my addr 172.21.15.44:0/3691488406 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6b400640 1 -- 172.21.15.44:0/3691488406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c10af10 msgr2=0x7f0a6c19c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6b400640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c10af10 0x7f0a6c19c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6b400640 1 -- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c1055d0 msgr2=0x7f0a6c19bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6b400640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c19bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.055+0000 7f0a6b400640 1 -- 172.21.15.44:0/3691488406 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6c1a1e80 con 0x7f0a6c10b860 2024-07-26T08:37:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.056+0000 7f0a6b400640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c10b860 0x7f0a6c1a1750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009fd0 tx=0x7f0a6400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.056+0000 7f0a63e00640 1 -- 172.21.15.44:0/3691488406 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6400ce20 con 0x7f0a6c10b860 2024-07-26T08:37:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.056+0000 7f0a63e00640 1 -- 172.21.15.44:0/3691488406 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a64005020 con 0x7f0a6c10b860 2024-07-26T08:37:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.056+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6c1a2170 con 0x7f0a6c10b860 2024-07-26T08:37:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.056+0000 7f0a63e00640 1 -- 172.21.15.44:0/3691488406 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64010690 con 0x7f0a6c10b860 2024-07-26T08:37:14.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.056+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6c1a88d0 con 0x7f0a6c10b860 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.059+0000 7f0a787fe740 1 Processor -- start 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.060+0000 7f0a63e00640 1 -- 172.21.15.44:0/3691488406 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f0a640054d0 con 0x7f0a6c10b860 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.060+0000 7f0a787fe740 1 -- start start 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d16ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3950280 0x55eda3d171e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a787fe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eda3c8c950 con 0x55eda3950280 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a787fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eda3c7be50 con 0x55eda3970f60 2024-07-26T08:37:14.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a787fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55eda3c7fe90 con 0x55eda39480a0 2024-07-26T08:37:14.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d16ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a62000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d16ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:57120/0 (socket says 172.21.15.44:57120) 2024-07-26T08:37:14.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a6aa00640 1 -- 172.21.15.44:0/468216924 learned_addr learned my addr 172.21.15.44:0/468216924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:14.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.061+0000 7f0a62000640 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3950280 0x55eda3d171e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a6aa00640 1 -- 172.21.15.44:0/468216924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 msgr2=0x55eda3d16760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d16760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a6aa00640 1 -- 172.21.15.44:0/468216924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3950280 msgr2=0x55eda3d171e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3950280 0x55eda3d171e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a6aa00640 1 -- 172.21.15.44:0/468216924 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda3c230f0 con 0x55eda39480a0 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a62000640 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3950280 0x55eda3d171e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d16ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c006010 tx=0x7f0a5c004100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95be18458f96d3f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a63e00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0a500b3ca0 0x7f0a500b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.062+0000 7f0a5be00640 1 -- 172.21.15.44:0/468216924 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c040070 con 0x55eda39480a0 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.063+0000 7f0a5be00640 1 -- 172.21.15.44:0/468216924 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a5c0054d0 con 0x55eda39480a0 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.063+0000 7f0a5be00640 1 -- 172.21.15.44:0/468216924 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c035040 con 0x55eda39480a0 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.063+0000 7f0a63e00640 1 -- 172.21.15.44:0/3691488406 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f0a640e4790 con 0x7f0a6c10b860 2024-07-26T08:37:14.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.063+0000 7f0a62000640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0a500b3ca0 0x7f0a500b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.063+0000 7f0a787fe740 1 -- 172.21.15.44:0/468216924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 msgr2=0x55eda3d16ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.063+0000 7f0a787fe740 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d16ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c006010 tx=0x7f0a5c004100 comp rx=0 tx=0).stop 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a62000640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0a500b3ca0 0x7f0a500b61b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009730 tx=0x7f0a54013040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 -- 172.21.15.44:0/468216924 shutdown_connections 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d16ca0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3950280 0x55eda3d171e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 --2- 172.21.15.44:0/468216924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d16760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 -- 172.21.15.44:0/468216924 >> 172.21.15.44:0/468216924 conn(0x55eda3965350 msgr2=0x55eda3d14650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 -- 172.21.15.44:0/468216924 shutdown_connections 2024-07-26T08:37:14.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 -- 172.21.15.44:0/468216924 wait complete. 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.064+0000 7f0a787fe740 1 -- start start 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 Processor -- start 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d21f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 0x55eda3d224d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d2e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eda3adf5f0 con 0x55eda3970f60 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eda3c18520 con 0x55eda3950280 2024-07-26T08:37:14.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d24ee0 con 0x55eda39480a0 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.065+0000 7f0a787fe740 10 client.0 _ll_register_callbacks cb 0x55eda3c27ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 0x55eda3d224d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 0x55eda3d224d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34490/0 (socket says 172.21.15.44:34490) 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d21f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 -- 172.21.15.44:0/392034859 learned_addr learned my addr 172.21.15.44:0/392034859 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a6b400640 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d2e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 -- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 msgr2=0x55eda3d21f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d21f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 -- 172.21.15.44:0/392034859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 msgr2=0x55eda3d2e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d2e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.066+0000 7f0a62000640 1 -- 172.21.15.44:0/392034859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda39dc540 con 0x55eda3950280 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a62000640 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 0x55eda3d224d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a540038a0 tx=0x7f0a54004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb961264a4ce1fde server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:14.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a58c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a58c00640 1 -- 172.21.15.44:0/392034859 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400fa80 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a5c0054d0 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a58c00640 1 -- 172.21.15.44:0/392034859 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f0a5400fc20 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a58c00640 1 -- 172.21.15.44:0/392034859 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a5400fe20 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a787fe740 10 client.25187 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.067+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eda3d30e50 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a58c00640 1 -- 172.21.15.44:0/392034859 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a5c0054d0 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a58c00640 1 -- 172.21.15.44:0/392034859 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0a54029070 con 0x55eda3950280 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a787fe740 10 client.25187 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a787fe740 10 client.25187 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a787fe740 20 client.25187 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a787fe740 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda392f8e0 0x55eda3d34930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a787fe740 4 client.25187 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:37:14.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.068+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eda3c8e8a0 con 0x55eda392f8e0 2024-07-26T08:37:14.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.069+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda392f8e0 0x55eda3d34930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.069+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda392f8e0 0x55eda3d34930 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28a5c6ce94a7d02f server_cookie=8f693c53ce4ccaee in_seq=0 out_seq=0 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.069+0000 7f0a58c00640 10 client.25187 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a58c00640 1 -- 172.21.15.44:0/392034859 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eda3c8e8a0 con 0x55eda392f8e0 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a58c00640 10 client.25187 handle_command_reply: tid=0 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 client.25187 shutdown 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 msgr2=0x55eda3d224d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 0x55eda3d224d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a540038a0 tx=0x7f0a54004730 comp rx=0 tx=0).stop 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 shutdown_connections 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda392f8e0 0x55eda3d34930 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d2e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3950280 0x55eda3d224d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 --2- 172.21.15.44:0/392034859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda39480a0 0x55eda3d21f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a58c00640 0 client.25187 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 >> 172.21.15.44:0/392034859 conn(0x55eda3965350 msgr2=0x55eda3d198f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 shutdown_connections 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.071+0000 7f0a787fe740 1 -- 172.21.15.44:0/392034859 wait complete. 2024-07-26T08:37:14.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.072+0000 7f0a787fe740 20 client.25187 trim_cache size 0 max 16384 2024-07-26T08:37:14.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.091+0000 7f0a787fe740 1 Processor -- start 2024-07-26T08:37:14.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 -- start start 2024-07-26T08:37:14.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 0x55eda3d1c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d14570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d2e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d32c70 con 0x55eda39480a0 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d33340 con 0x55eda3970f60 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a787fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d33a10 con 0x55eda3950280 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 0x55eda3d1c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.092+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 0x55eda3d1c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54788/0 (socket says 172.21.15.44:54788) 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6aa00640 1 -- 172.21.15.44:0/1011522599 learned_addr learned my addr 172.21.15.44:0/1011522599 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a62000640 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d14570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6b400640 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d2e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6aa00640 1 -- 172.21.15.44:0/1011522599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 msgr2=0x55eda3d14570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d14570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6aa00640 1 -- 172.21.15.44:0/1011522599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 msgr2=0x55eda3d2e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d2e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a6aa00640 1 -- 172.21.15.44:0/1011522599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda3d14b90 con 0x55eda39480a0 2024-07-26T08:37:14.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.093+0000 7f0a62000640 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d14570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:14.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.094+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 0x55eda3d1c880 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c004ce0 tx=0x7f0a5c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09375d1bfedeb24 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:14.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.094+0000 7f0a5b400640 1 -- 172.21.15.44:0/1011522599 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c035040 con 0x55eda39480a0 2024-07-26T08:37:14.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.094+0000 7f0a5b400640 1 -- 172.21.15.44:0/1011522599 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a5c0049a0 con 0x55eda39480a0 2024-07-26T08:37:14.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.094+0000 7f0a5b400640 1 -- 172.21.15.44:0/1011522599 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c040040 con 0x55eda39480a0 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 -- 172.21.15.44:0/1011522599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 msgr2=0x55eda3d1c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 0x55eda3d1c880 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c004ce0 tx=0x7f0a5c004d10 comp rx=0 tx=0).stop 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 -- 172.21.15.44:0/1011522599 shutdown_connections 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d14570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda3970f60 0x55eda3d2e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 --2- 172.21.15.44:0/1011522599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda39480a0 0x55eda3d1c880 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 -- 172.21.15.44:0/1011522599 >> 172.21.15.44:0/1011522599 conn(0x55eda3965350 msgr2=0x55eda3d13e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.095+0000 7f0a787fe740 1 -- 172.21.15.44:0/1011522599 shutdown_connections 2024-07-26T08:37:14.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.096+0000 7f0a787fe740 1 -- 172.21.15.44:0/1011522599 wait complete. 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.096+0000 7f0a787fe740 1 -- start start 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.096+0000 7f0a787fe740 1 Processor -- start 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda39480a0 0x55eda3d3d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d35b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d36070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d347b0 con 0x55eda3970f60 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d340e0 con 0x55eda39480a0 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55eda3d34e80 con 0x55eda3950280 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a787fe740 10 client.0 _ll_register_callbacks cb 0x7f0a54004350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a6b400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d36070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a62000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d35b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a6aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda39480a0 0x55eda3d3d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a62000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d35b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:57152/0 (socket says 172.21.15.44:57152) 2024-07-26T08:37:14.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.097+0000 7f0a62000640 1 -- 172.21.15.44:0/3798682050 learned_addr learned my addr 172.21.15.44:0/3798682050 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a6b400640 1 -- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 msgr2=0x55eda3d35b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a6b400640 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d35b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a6b400640 1 -- 172.21.15.44:0/3798682050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda39480a0 msgr2=0x55eda3d3d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a6b400640 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda39480a0 0x55eda3d3d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a6b400640 1 -- 172.21.15.44:0/3798682050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda3d0f7d0 con 0x55eda3970f60 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a6b400640 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d36070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c760 tx=0x7f0a640ab990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7c5b8874caa2fc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.098+0000 7f0a58c00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a58c00640 1 -- 172.21.15.44:0/3798682050 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a640ca060 con 0x55eda3970f60 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a58c00640 1 -- 172.21.15.44:0/3798682050 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f0a640b3df0 con 0x55eda3970f60 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a5c0049a0 con 0x55eda3970f60 2024-07-26T08:37:14.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a58c00640 1 -- 172.21.15.44:0/3798682050 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a640b4280 con 0x55eda3970f60 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a787fe740 10 client.15066 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eda3d9c880 con 0x55eda3970f60 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.099+0000 7f0a58c00640 1 -- 172.21.15.44:0/3798682050 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a5c0049a0 con 0x55eda3970f60 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a58c00640 1 -- 172.21.15.44:0/3798682050 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0a640aa430 con 0x55eda3970f60 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a787fe740 10 client.15066 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a787fe740 10 client.15066 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a787fe740 20 client.15066 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a787fe740 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda3904c90 0x55eda3da0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:14.107 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a787fe740 4 client.15066 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "status"}) -- 0x55eda3c8a340 con 0x55eda3904c90 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.100+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda3904c90 0x55eda3da0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.101+0000 7f0a6aa00640 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda3904c90 0x55eda3da0160 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa429322688d0250 server_cookie=63a82787dad77222 in_seq=0 out_seq=0 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.101+0000 7f0a58c00640 10 client.15066 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.101+0000 7f0a58c00640 1 -- 172.21.15.44:0/3798682050 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55eda3c8a340 con 0x55eda3904c90 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.101+0000 7f0a58c00640 10 client.15066 handle_command_reply: tid=0 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.101+0000 7f0a787fe740 1 client.15066 shutdown 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 msgr2=0x55eda3d36070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d36070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c760 tx=0x7f0a640ab990 comp rx=0 tx=0).stop 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 shutdown_connections 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x55eda3904c90 0x55eda3da0160 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55eda3950280 0x55eda3d35b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.108 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55eda3970f60 0x55eda3d36070 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 --2- 172.21.15.44:0/3798682050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eda39480a0 0x55eda3d3d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 >> 172.21.15.44:0/3798682050 conn(0x55eda3965350 msgr2=0x55eda3d21d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a58c00640 0 client.15066 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 shutdown_connections 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 1 -- 172.21.15.44:0/3798682050 wait complete. 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.102+0000 7f0a787fe740 20 client.15066 trim_cache size 0 max 16384 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:37:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 2, 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24679, 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 165.04640909899999, 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 283.41269505999998 2024-07-26T08:37:14.110 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:37:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0a500b3ca0 msgr2=0x7f0a500b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0a500b3ca0 0x7f0a500b61b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009730 tx=0x7f0a54013040 comp rx=0 tx=0).stop 2024-07-26T08:37:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c10b860 msgr2=0x7f0a6c1a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c10b860 0x7f0a6c1a1750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009fd0 tx=0x7f0a6400ef10 comp rx=0 tx=0).stop 2024-07-26T08:37:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 shutdown_connections 2024-07-26T08:37:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f0a500b3ca0 0x7f0a500b61b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a6c10b860 0x7f0a6c1a1750 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a6c10af10 0x7f0a6c19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 --2- 172.21.15.44:0/3691488406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0a6c1055d0 0x7f0a6c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.103+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 >> 172.21.15.44:0/3691488406 conn(0x7f0a6c101040 msgr2=0x7f0a6c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.104+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 shutdown_connections 2024-07-26T08:37:14.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:14.104+0000 7f0a6be00640 1 -- 172.21.15.44:0/3691488406 wait complete. 2024-07-26T08:37:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:14 smithi044 ceph-mon[32404]: pgmap v439: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:14 smithi116 ceph-mon[37097]: pgmap v439: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:14 smithi171 ceph-mon[37789]: pgmap v439: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:15.014 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.g status 2024-07-26T08:37:15.539 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:16 smithi044 ceph-mon[32404]: pgmap v440: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:16 smithi116 ceph-mon[37097]: pgmap v440: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:16 smithi171 ceph-mon[37789]: pgmap v440: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:17.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.900+0000 7f12baa00640 1 -- 172.21.15.44:0/1604612824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4076db0 msgr2=0x7f12b40751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.900+0000 7f12baa00640 1 --2- 172.21.15.44:0/1604612824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4076db0 0x7f12b40751e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009900 tx=0x7f12a802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 -- 172.21.15.44:0/1604612824 shutdown_connections 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 --2- 172.21.15.44:0/1604612824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 0x7f12b4116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 --2- 172.21.15.44:0/1604612824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4075800 0x7f12b4075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 --2- 172.21.15.44:0/1604612824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4076db0 0x7f12b40751e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 -- 172.21.15.44:0/1604612824 >> 172.21.15.44:0/1604612824 conn(0x7f12b40fee70 msgr2=0x7f12b41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 -- 172.21.15.44:0/1604612824 shutdown_connections 2024-07-26T08:37:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.901+0000 7f12baa00640 1 -- 172.21.15.44:0/1604612824 wait complete. 2024-07-26T08:37:17.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.902+0000 7f12baa00640 1 Processor -- start 2024-07-26T08:37:17.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.902+0000 7f12baa00640 1 -- start start 2024-07-26T08:37:17.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 0x7f12b41a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12baa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4076db0 0x7f12b41a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12baa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 0x7f12b41a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12baa00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12b4119940 con 0x7f12b4076db0 2024-07-26T08:37:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12baa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41197c0 con 0x7f12b4075800 2024-07-26T08:37:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12baa00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f12b4119ac0 con 0x7f12b4113cb0 2024-07-26T08:37:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.903+0000 7f12b9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 0x7f12b41a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 0x7f12b41a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34528/0 (socket says 172.21.15.44:34528) 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 -- 172.21.15.44:0/2788137376 learned_addr learned my addr 172.21.15.44:0/2788137376 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12ba000640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 0x7f12b41a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b8c00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4076db0 0x7f12b41a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 -- 172.21.15.44:0/2788137376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 msgr2=0x7f12b41a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 0x7f12b41a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 -- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4076db0 msgr2=0x7f12b41a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4076db0 0x7f12b41a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b9600640 1 -- 172.21.15.44:0/2788137376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b41a6280 con 0x7f12b4075800 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12ba000640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 0x7f12b41a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.904+0000 7f12b8c00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4076db0 0x7f12b41a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:17.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.905+0000 7f12b9600640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 0x7f12b41a0310 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005ab0 tx=0x7f12a8003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:17.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.905+0000 7f12b2000640 1 -- 172.21.15.44:0/2788137376 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a803f070 con 0x7f12b4075800 2024-07-26T08:37:17.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.905+0000 7f12b2000640 1 -- 172.21.15.44:0/2788137376 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a8031d60 con 0x7f12b4075800 2024-07-26T08:37:17.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.905+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b41a6510 con 0x7f12b4075800 2024-07-26T08:37:17.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.905+0000 7f12b2000640 1 -- 172.21.15.44:0/2788137376 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8035030 con 0x7f12b4075800 2024-07-26T08:37:17.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.905+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b41accd0 con 0x7f12b4075800 2024-07-26T08:37:17.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.909+0000 7f12c2dc2740 1 Processor -- start 2024-07-26T08:37:17.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 -- start start 2024-07-26T08:37:17.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12b2000640 1 -- 172.21.15.44:0/2788137376 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f12a8042070 con 0x7f12b4075800 2024-07-26T08:37:17.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebb3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 0x5591eebb3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5591eeb29950 con 0x5591ee7ed280 2024-07-26T08:37:17.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5591eeb18e50 con 0x5591ee7e50a0 2024-07-26T08:37:17.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5591eeb1ce90 con 0x5591ee80df60 2024-07-26T08:37:17.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12b9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 0x5591eebb3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12ba000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebb3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12ba000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebb3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:57168/0 (socket says 172.21.15.44:57168) 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12b9600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 0x5591eebb3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34542/0 (socket says 172.21.15.44:34542) 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.910+0000 7f12ba000640 1 -- 172.21.15.44:0/3463434013 learned_addr learned my addr 172.21.15.44:0/3463434013 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.911+0000 7f12b8c00640 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.911+0000 7f12b9600640 1 -- 172.21.15.44:0/3463434013 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 msgr2=0x5591eebb3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.911+0000 7f12b9600640 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebb3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.911+0000 7f12b9600640 1 -- 172.21.15.44:0/3463434013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 msgr2=0x5591eebb41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.911+0000 7f12b9600640 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.911+0000 7f12b9600640 1 -- 172.21.15.44:0/3463434013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591eeac00f0 con 0x5591ee7e50a0 2024-07-26T08:37:17.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12b9600640 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 0x5591eebb3ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f12a8101ed0 tx=0x7f12a810acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f64eb86b7b76b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12a6600640 1 -- 172.21.15.44:0/3463434013 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8113070 con 0x5591ee7e50a0 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12a6600640 1 -- 172.21.15.44:0/3463434013 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a810cc60 con 0x5591ee7e50a0 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12a6600640 1 -- 172.21.15.44:0/3463434013 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8106330 con 0x5591ee7e50a0 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12b2000640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f129c0b3d20 0x7f129c0b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12b2000640 1 -- 172.21.15.44:0/2788137376 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f12a8104940 con 0x7f12b4075800 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.912+0000 7f12ba000640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f129c0b3d20 0x7f129c0b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 -- 172.21.15.44:0/3463434013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 msgr2=0x5591eebb3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 0x5591eebb3ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f12a8101ed0 tx=0x7f12a810acb0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 -- 172.21.15.44:0/3463434013 shutdown_connections 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7e50a0 0x5591eebb3ca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/3463434013 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebb3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 -- 172.21.15.44:0/3463434013 >> 172.21.15.44:0/3463434013 conn(0x5591ee802350 msgr2=0x5591eebb1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:17.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12ba000640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f129c0b3d20 0x7f129c0b6230 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009fd0 tx=0x7f12ac009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:17.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 -- 172.21.15.44:0/3463434013 shutdown_connections 2024-07-26T08:37:17.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.913+0000 7f12c2dc2740 1 -- 172.21.15.44:0/3463434013 wait complete. 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.914+0000 7f12c2dc2740 1 -- start start 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.914+0000 7f12c2dc2740 1 Processor -- start 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebbefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7ed280 0x5591eebbf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebcaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5591ee97c5f0 con 0x5591ee7e50a0 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5591eeab5520 con 0x5591ee80df60 2024-07-26T08:37:17.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5591eebc1d00 con 0x5591ee7ed280 2024-07-26T08:37:17.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12c2dc2740 10 client.0 _ll_register_callbacks cb 0x5591eeac4ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:17.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.915+0000 7f12b9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebbefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebbefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54838/0 (socket says 172.21.15.44:54838) 2024-07-26T08:37:17.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 -- 172.21.15.44:0/1633096818 learned_addr learned my addr 172.21.15.44:0/1633096818 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:17.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12ba000640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebcaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b8c00640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7ed280 0x5591eebbf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 -- 172.21.15.44:0/1633096818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7ed280 msgr2=0x5591eebbf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7ed280 0x5591eebbf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 -- 172.21.15.44:0/1633096818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 msgr2=0x5591eebcaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebcaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 -- 172.21.15.44:0/1633096818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591ee879540 con 0x5591ee7e50a0 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f12b9600640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebbefc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f12a802a9b0 tx=0x7f12a803b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfb744e573637c9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:17.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.916+0000 7f129aa00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f129aa00640 1 -- 172.21.15.44:0/1633096818 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8049020 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f129aa00640 1 -- 172.21.15.44:0/1633096818 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f12a810ca60 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a810cc60 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f129aa00640 1 -- 172.21.15.44:0/1633096818 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f12a8113070 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f12c2dc2740 10 client.15078 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591eebcdb30 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.917+0000 7f129aa00640 1 -- 172.21.15.44:0/1633096818 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a810cc60 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f129aa00640 1 -- 172.21.15.44:0/1633096818 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f12a80c7610 con 0x5591ee7e50a0 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12c2dc2740 10 client.15078 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12c2dc2740 10 client.15078 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T08:37:17.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12c2dc2740 20 client.15078 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:17.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7cc8e0 0x5591eebd1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12c2dc2740 4 client.15078 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:37:17.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591eeb2b8a0 con 0x5591ee7cc8e0 2024-07-26T08:37:17.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.918+0000 7f12b8c00640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7cc8e0 0x5591eebd1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.919+0000 7f12b8c00640 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7cc8e0 0x5591eebd1120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a2ce50cea312f71 server_cookie=353704ba095345a6 in_seq=0 out_seq=0 2024-07-26T08:37:17.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.919+0000 7f129aa00640 10 client.15078 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f129aa00640 1 -- 172.21.15.44:0/1633096818 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591eeb2b8a0 con 0x5591ee7cc8e0 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f129aa00640 10 client.15078 handle_command_reply: tid=0 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 client.15078 shutdown 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 msgr2=0x5591eebbefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebbefc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f12a802a9b0 tx=0x7f12a803b460 comp rx=0 tx=0).stop 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 shutdown_connections 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7cc8e0 0x5591eebd1120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7ed280 0x5591eebbf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebcaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1633096818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebbefc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f129aa00640 0 client.15078 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.920+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 >> 172.21.15.44:0/1633096818 conn(0x5591ee802350 msgr2=0x5591eebb68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.921+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 shutdown_connections 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.921+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1633096818 wait complete. 2024-07-26T08:37:17.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.921+0000 7f12c2dc2740 20 client.15078 trim_cache size 0 max 16384 2024-07-26T08:37:17.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 Processor -- start 2024-07-26T08:37:17.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 -- start start 2024-07-26T08:37:17.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 0x5591eebd0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebbd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5591eebcf460 con 0x5591ee7ed280 2024-07-26T08:37:17.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5591eebcfb30 con 0x5591ee80df60 2024-07-26T08:37:17.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.941+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5591eebd0200 con 0x5591ee7e50a0 2024-07-26T08:37:17.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 0x5591eebd0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 0x5591eebd0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:57192/0 (socket says 172.21.15.44:57192) 2024-07-26T08:37:17.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 -- 172.21.15.44:0/1190054690 learned_addr learned my addr 172.21.15.44:0/1190054690 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b8c00640 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12ba000640 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebbd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 -- 172.21.15.44:0/1190054690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 msgr2=0x5591eebbd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebbd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 -- 172.21.15.44:0/1190054690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 msgr2=0x5591eebb95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b9600640 1 -- 172.21.15.44:0/1190054690 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591eebc0280 con 0x5591ee7e50a0 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.942+0000 7f12b8c00640 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:17.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.943+0000 7f12b9600640 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 0x5591eebd0a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f12a80c85f0 tx=0x7f12a8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2042b4df75fe047 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:17.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.943+0000 7f12a5c00640 1 -- 172.21.15.44:0/1190054690 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8049070 con 0x5591ee7e50a0 2024-07-26T08:37:17.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.943+0000 7f12a5c00640 1 -- 172.21.15.44:0/1190054690 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a80cba80 con 0x5591ee7e50a0 2024-07-26T08:37:17.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.943+0000 7f12a5c00640 1 -- 172.21.15.44:0/1190054690 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a80c7240 con 0x5591ee7e50a0 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1190054690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 msgr2=0x5591eebd0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 0x5591eebd0a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f12a80c85f0 tx=0x7f12a8009260 comp rx=0 tx=0).stop 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1190054690 shutdown_connections 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee80df60 0x5591eebbd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7ed280 0x5591eebb95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/1190054690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee7e50a0 0x5591eebd0a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.944+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1190054690 >> 172.21.15.44:0/1190054690 conn(0x5591ee802350 msgr2=0x5591eebbed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.945+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1190054690 shutdown_connections 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.945+0000 7f12c2dc2740 1 -- 172.21.15.44:0/1190054690 wait complete. 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.945+0000 7f12c2dc2740 1 -- start start 2024-07-26T08:37:17.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 Processor -- start 2024-07-26T08:37:17.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebd3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7ed280 0x5591eebd4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebd7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5591eebd0fa0 con 0x5591ee7e50a0 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5591eebd08d0 con 0x5591ee7ed280 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5591eebd1670 con 0x5591ee80df60 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12c2dc2740 10 client.0 _ll_register_callbacks cb 0x7f12a8101f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.946+0000 7f12b9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebd3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12b9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebd3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54864/0 (socket says 172.21.15.44:54864) 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12ba000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebd7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12b9600640 1 -- 172.21.15.44:0/4152220742 learned_addr learned my addr 172.21.15.44:0/4152220742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12b8c00640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7ed280 0x5591eebd4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12ba000640 1 -- 172.21.15.44:0/4152220742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7ed280 msgr2=0x5591eebd4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12ba000640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7ed280 0x5591eebd4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12ba000640 1 -- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 msgr2=0x5591eebd3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:17.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12ba000640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebd3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12ba000640 1 -- 172.21.15.44:0/4152220742 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591eebd82a0 con 0x5591ee80df60 2024-07-26T08:37:17.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.947+0000 7f12b9600640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebd3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f12ba000640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebd7c30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f12ac013010 tx=0x7f12ac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d47b6d5c14a71b09 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f129aa00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f129aa00640 1 -- 172.21.15.44:0/4152220742 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac003b80 con 0x5591ee80df60 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a80cba80 con 0x5591ee80df60 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f129aa00640 1 -- 172.21.15.44:0/4152220742 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f12ac003d20 con 0x5591ee80df60 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f129aa00640 1 -- 172.21.15.44:0/4152220742 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f12ac011910 con 0x5591ee80df60 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f12c2dc2740 10 client.25213 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:17.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.948+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591eebd9710 con 0x5591ee80df60 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f129aa00640 1 -- 172.21.15.44:0/4152220742 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a80cba80 con 0x5591ee80df60 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f129aa00640 1 -- 172.21.15.44:0/4152220742 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f12ac02a430 con 0x5591ee80df60 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f12c2dc2740 10 client.25213 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f12c2dc2740 10 client.25213 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f12c2dc2740 20 client.25213 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7a1c90 0x5591eec3d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f12c2dc2740 4 client.25213 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.949+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "status"}) -- 0x5591eeb27340 con 0x5591ee7a1c90 2024-07-26T08:37:17.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.950+0000 7f12b9600640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7a1c90 0x5591eec3d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:17.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.950+0000 7f12b9600640 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7a1c90 0x5591eec3d100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3187d542d365a479 server_cookie=8a76f9cf683bbe0e in_seq=0 out_seq=0 2024-07-26T08:37:17.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.950+0000 7f129aa00640 10 client.25213 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f129aa00640 1 -- 172.21.15.44:0/4152220742 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5591eeb27340 con 0x5591ee7a1c90 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f129aa00640 10 client.25213 handle_command_reply: tid=0 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 client.25213 shutdown 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 msgr2=0x5591eebd7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebd7c30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f12ac013010 tx=0x7f12ac004a40 comp rx=0 tx=0).stop 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 shutdown_connections 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5591ee7a1c90 0x5591eec3d100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5591ee80df60 0x5591eebd7c30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ee7ed280 0x5591eebd4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 --2- 172.21.15.44:0/4152220742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591ee7e50a0 0x5591eebd3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f129aa00640 0 client.25213 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 >> 172.21.15.44:0/4152220742 conn(0x5591ee802350 msgr2=0x5591eebb0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:17.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 shutdown_connections 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.951+0000 7f12c2dc2740 1 -- 172.21.15.44:0/4152220742 wait complete. 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12c2dc2740 20 client.25213 trim_cache size 0 max 16384 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 1, 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24686, 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:37:17.956 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 169.90691507099999, 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 282.77128256600002 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:37:17.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f129c0b3d20 msgr2=0x7f129c0b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f129c0b3d20 0x7f129c0b6230 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009fd0 tx=0x7f12ac009500 comp rx=0 tx=0).stop 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 msgr2=0x7f12b41a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 0x7f12b41a0310 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005ab0 tx=0x7f12a8003c80 comp rx=0 tx=0).stop 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 shutdown_connections 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f129c0b3d20 0x7f129c0b6230 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.952+0000 7f12baa00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b4113cb0 0x7f12b41a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.953+0000 7f12baa00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12b4076db0 0x7f12b41a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.953+0000 7f12baa00640 1 --2- 172.21.15.44:0/2788137376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b4075800 0x7f12b41a0310 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.953+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 >> 172.21.15.44:0/2788137376 conn(0x7f12b40fee70 msgr2=0x7f12b410fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.953+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 shutdown_connections 2024-07-26T08:37:17.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:17.953+0000 7f12baa00640 1 -- 172.21.15.44:0/2788137376 wait complete. 2024-07-26T08:37:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:18 smithi044 ceph-mon[32404]: pgmap v441: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:18 smithi116 ceph-mon[37097]: pgmap v441: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:18 smithi171 ceph-mon[37789]: pgmap v441: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:18.890 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T08:37:19.380 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:20 smithi044 ceph-mon[32404]: pgmap v442: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:20 smithi116 ceph-mon[37097]: pgmap v442: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:20 smithi171 ceph-mon[37789]: pgmap v442: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 -- 172.21.15.44:0/2737649590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a41055d0 msgr2=0x7f51a41059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2737649590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a41055d0 0x7f51a41059e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5194009860 tx=0x7f519402a340 comp rx=0 tx=0).stop 2024-07-26T08:37:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 -- 172.21.15.44:0/2737649590 shutdown_connections 2024-07-26T08:37:21.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2737649590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410b860 0x7f51a4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2737649590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a410af10 0x7f51a410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2737649590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a41055d0 0x7f51a41059e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 -- 172.21.15.44:0/2737649590 >> 172.21.15.44:0/2737649590 conn(0x7f51a4101040 msgr2=0x7f51a4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 -- 172.21.15.44:0/2737649590 shutdown_connections 2024-07-26T08:37:21.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.874+0000 7f51a8a00640 1 -- 172.21.15.44:0/2737649590 wait complete. 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 Processor -- start 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 -- start start 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 0x7f51a419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410af10 0x7f51a419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a410b860 0x7f51a41a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f51a4115510 con 0x7f51a41055d0 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f51a4115390 con 0x7f51a410af10 2024-07-26T08:37:21.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.876+0000 7f51a8a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f51a4115690 con 0x7f51a410b860 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 0x7f51a419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f519aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410af10 0x7f51a419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 0x7f51a419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46352/0 (socket says 172.21.15.44:46352) 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 -- 172.21.15.44:0/2142845024 learned_addr learned my addr 172.21.15.44:0/2142845024 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3e00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a410b860 0x7f51a41a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 -- 172.21.15.44:0/2142845024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a410b860 msgr2=0x7f51a41a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a410b860 0x7f51a41a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 -- 172.21.15.44:0/2142845024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410af10 msgr2=0x7f51a419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410af10 0x7f51a419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f51a3400640 1 -- 172.21.15.44:0/2142845024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a41a1e90 con 0x7f51a41055d0 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.877+0000 7f519aa00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410af10 0x7f51a419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:21.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.878+0000 7f51a3400640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 0x7f51a419bf20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5194035040 tx=0x7f5194032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.878+0000 7f51a0c00640 1 -- 172.21.15.44:0/2142845024 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5194037040 con 0x7f51a41055d0 2024-07-26T08:37:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.878+0000 7f51a0c00640 1 -- 172.21.15.44:0/2142845024 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5194041070 con 0x7f51a41055d0 2024-07-26T08:37:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.878+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a41a2120 con 0x7f51a41055d0 2024-07-26T08:37:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.878+0000 7f51a0c00640 1 -- 172.21.15.44:0/2142845024 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5194003850 con 0x7f51a41055d0 2024-07-26T08:37:21.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.878+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a41a25a0 con 0x7f51a41055d0 2024-07-26T08:37:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 Processor -- start 2024-07-26T08:37:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 -- start start 2024-07-26T08:37:21.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471dc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471dcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471dd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d547152950 con 0x55d546e16280 2024-07-26T08:37:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d547141e50 con 0x55d546e36f60 2024-07-26T08:37:21.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.883+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d547145e90 con 0x55d546e0e0a0 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f519aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471dd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a0c00640 1 -- 172.21.15.44:0/2142845024 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f51940039f0 con 0x7f51a41055d0 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f519aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471dd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46358/0 (socket says 172.21.15.44:46358) 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f519aa00640 1 -- 172.21.15.44:0/3602992895 learned_addr learned my addr 172.21.15.44:0/3602992895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3e00640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471dc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3400640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471dcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3e00640 1 -- 172.21.15.44:0/3602992895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 msgr2=0x55d5471dcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3e00640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471dcca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3e00640 1 -- 172.21.15.44:0/3602992895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 msgr2=0x55d5471dd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3e00640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471dd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3e00640 1 -- 172.21.15.44:0/3602992895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5470e90f0 con 0x55d546e36f60 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f519aa00640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471dd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.884+0000 7f51a3400640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471dcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:37:21.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.885+0000 7f51a3e00640 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471dc760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f519c0099b0 tx=0x7f519c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbdfba978b2ae851 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.885+0000 7f5193e00640 1 -- 172.21.15.44:0/3602992895 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c003970 con 0x55d546e36f60 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.885+0000 7f5193e00640 1 -- 172.21.15.44:0/3602992895 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519c00d6b0 con 0x55d546e36f60 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.885+0000 7f5193e00640 1 -- 172.21.15.44:0/3602992895 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c004240 con 0x55d546e36f60 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.886+0000 7f51a0c00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f51880b3c20 0x7f51880b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.886+0000 7f51a0c00640 1 -- 172.21.15.44:0/2142845024 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f5194103c90 con 0x7f51a41055d0 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.886+0000 7f51a3400640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f51880b3c20 0x7f51880b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.886+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3602992895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 msgr2=0x55d5471dc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.886+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471dc760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f519c0099b0 tx=0x7f519c0095c0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3602992895 shutdown_connections 2024-07-26T08:37:21.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471dcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471dd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3602992895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471dc760 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3602992895 >> 172.21.15.44:0/3602992895 conn(0x55d546e2b350 msgr2=0x55d5471da650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3602992895 shutdown_connections 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3602992895 wait complete. 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.887+0000 7f51a3400640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f51880b3c20 0x7f51880b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5194101280 tx=0x7f51940303b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:21.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 -- start start 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 Processor -- start 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471e7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471e85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d546fa55f0 con 0x55d546e16280 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5470de520 con 0x55d546e36f60 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.888+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d5471eaf40 con 0x55d546e0e0a0 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f51b0c7b740 10 client.0 _ll_register_callbacks cb 0x55d5470edef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471e85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471e7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471e7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:43632/0 (socket says 172.21.15.44:43632) 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f51a3e00640 1 -- 172.21.15.44:0/4206513930 learned_addr learned my addr 172.21.15.44:0/4206513930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471e85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46366/0 (socket says 172.21.15.44:46366) 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f51a3e00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 -- 172.21.15.44:0/4206513930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 msgr2=0x55d5471e7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471e7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 -- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 msgr2=0x55d5471f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f519aa00640 1 -- 172.21.15.44:0/4206513930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d546ea2540 con 0x55d546e16280 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.889+0000 7f51a3e00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.890+0000 7f519aa00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471e85a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f518c012d20 tx=0x7f518c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feefa12550be9cfa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:21.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.890+0000 7f5190c00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:37:21.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.890+0000 7f5190c00640 1 -- 172.21.15.44:0/4206513930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c0120d0 con 0x55d546e16280 2024-07-26T08:37:21.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.890+0000 7f5190c00640 1 -- 172.21.15.44:0/4206513930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f518c004030 con 0x55d546e16280 2024-07-26T08:37:21.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.890+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f519c00d6b0 con 0x55d546e16280 2024-07-26T08:37:21.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f5190c00640 1 -- 172.21.15.44:0/4206513930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f518c012270 con 0x55d546e16280 2024-07-26T08:37:21.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f51b0c7b740 10 client.15096 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:21.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5471f6e30 con 0x55d546e16280 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f5190c00640 1 -- 172.21.15.44:0/4206513930 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519c00d6b0 con 0x55d546e16280 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f5190c00640 1 -- 172.21.15.44:0/4206513930 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f518c01c460 con 0x55d546e16280 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f51b0c7b740 10 client.15096 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f51b0c7b740 10 client.15096 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f51b0c7b740 20 client.15096 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.891+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546df58e0 0x55d5471fa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.892+0000 7f51b0c7b740 4 client.15096 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.892+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5471548a0 con 0x55d546df58e0 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.892+0000 7f51a3e00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546df58e0 0x55d5471fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.892+0000 7f51a3e00640 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546df58e0 0x55d5471fa900 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7982c6608b527e47 server_cookie=2384244162c06584 in_seq=0 out_seq=0 2024-07-26T08:37:21.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.892+0000 7f5190c00640 10 client.15096 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.893+0000 7f5190c00640 1 -- 172.21.15.44:0/4206513930 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d5471548a0 con 0x55d546df58e0 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.893+0000 7f5190c00640 10 client.15096 handle_command_reply: tid=0 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 client.15096 shutdown 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 msgr2=0x55d5471e85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471e85a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f518c012d20 tx=0x7f518c00f790 comp rx=0 tx=0).stop 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 shutdown_connections 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546df58e0 0x55d5471fa900 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e16280 0x55d5471e85a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/4206513930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e0e0a0 0x55d5471e7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f5190c00640 0 client.15096 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:37:21.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 >> 172.21.15.44:0/4206513930 conn(0x55d546e2b350 msgr2=0x55d5471df8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:21.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 shutdown_connections 2024-07-26T08:37:21.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 1 -- 172.21.15.44:0/4206513930 wait complete. 2024-07-26T08:37:21.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.894+0000 7f51b0c7b740 20 client.15096 trim_cache size 0 max 16384 2024-07-26T08:37:21.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.914+0000 7f51b0c7b740 1 Processor -- start 2024-07-26T08:37:21.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.914+0000 7f51b0c7b740 1 -- start start 2024-07-26T08:37:21.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e0e0a0 0x55d5471e2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d5471da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d5471f8c40 con 0x55d546e0e0a0 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5471f9310 con 0x55d546e36f60 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d5471f99e0 con 0x55d546e16280 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e0e0a0 0x55d5471e2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e0e0a0 0x55d5471e2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46378/0 (socket says 172.21.15.44:46378) 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51a3e00640 1 -- 172.21.15.44:0/2900612334 learned_addr learned my addr 172.21.15.44:0/2900612334 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f519aa00640 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d5471da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.915+0000 7f51a3e00640 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f51a3e00640 1 -- 172.21.15.44:0/2900612334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 msgr2=0x55d5471da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f51a3e00640 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d5471da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f51a3e00640 1 -- 172.21.15.44:0/2900612334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e0e0a0 msgr2=0x55d5471e2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f51a3e00640 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e0e0a0 0x55d5471e2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f51a3e00640 1 -- 172.21.15.44:0/2900612334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5471d57d0 con 0x55d546e36f60 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f519aa00640 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d5471da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.916+0000 7f51a3e00640 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55d5471f6e00 tx=0x7f519c010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b98c160a51c88a0d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.917+0000 7f5193400640 1 -- 172.21.15.44:0/2900612334 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c0039f0 con 0x55d546e36f60 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.917+0000 7f5193400640 1 -- 172.21.15.44:0/2900612334 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519c003b90 con 0x55d546e36f60 2024-07-26T08:37:21.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.917+0000 7f5193400640 1 -- 172.21.15.44:0/2900612334 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c024450 con 0x55d546e36f60 2024-07-26T08:37:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.917+0000 7f51b0c7b740 1 -- 172.21.15.44:0/2900612334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 msgr2=0x55d5471f4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.917+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55d5471f6e00 tx=0x7f519c010040 comp rx=0 tx=0).stop 2024-07-26T08:37:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 -- 172.21.15.44:0/2900612334 shutdown_connections 2024-07-26T08:37:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d5471da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e36f60 0x55d5471f4200 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/2900612334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e0e0a0 0x55d5471e2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 -- 172.21.15.44:0/2900612334 >> 172.21.15.44:0/2900612334 conn(0x55d546e2b350 msgr2=0x55d5471d9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 -- 172.21.15.44:0/2900612334 shutdown_connections 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 -- 172.21.15.44:0/2900612334 wait complete. 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 -- start start 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.918+0000 7f51b0c7b740 1 Processor -- start 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e0e0a0 0x55d547200a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d547200ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e36f60 0x55d5471fc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d5471fa780 con 0x55d546e36f60 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5471fa0b0 con 0x55d546e0e0a0 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d5471fae50 con 0x55d546e16280 2024-07-26T08:37:21.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51b0c7b740 10 client.0 _ll_register_callbacks cb 0x7f518c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:37:21.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e0e0a0 0x55d547200a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e36f60 0x55d5471fc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f519aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d547200ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.919+0000 7f519aa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d547200ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:43652/0 (socket says 172.21.15.44:43652) 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 -- 172.21.15.44:0/3519519779 learned_addr learned my addr 172.21.15.44:0/3519519779 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 -- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e0e0a0 msgr2=0x55d547200a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e0e0a0 0x55d547200a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 -- 172.21.15.44:0/3519519779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e36f60 msgr2=0x55d5471fc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e36f60 0x55d5471fc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 -- 172.21.15.44:0/3519519779 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5471fc790 con 0x55d546e16280 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f519aa00640 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d547200ff0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f518c01c880 tx=0x7f518c01c690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d24c6c91e5a98068 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.920+0000 7f5190c00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f5190c00640 1 -- 172.21.15.44:0/3519519779 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c02e060 con 0x55d546e16280 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f5190c00640 1 -- 172.21.15.44:0/3519519779 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f518c005310 con 0x55d546e16280 2024-07-26T08:37:21.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f519c003b90 con 0x55d546e16280 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f5190c00640 1 -- 172.21.15.44:0/3519519779 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f518c005510 con 0x55d546e16280 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f51b0c7b740 10 client.25228 fetch_fsmap learned FSMap version 29 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d547202270 con 0x55d546e16280 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.921+0000 7f5190c00640 1 -- 172.21.15.44:0/3519519779 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519c003b90 con 0x55d546e16280 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f5190c00640 1 -- 172.21.15.44:0/3519519779 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f518c010430 con 0x55d546e16280 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51b0c7b740 10 client.25228 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51b0c7b740 10 client.25228 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51b0c7b740 20 client.25228 populate_metadata read hostname 'smithi044' 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546dcac90 0x55d547266e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51b0c7b740 4 client.25228 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "status"}) -- 0x55d547150340 con 0x55d546dcac90 2024-07-26T08:37:21.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.922+0000 7f51a3e00640 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546dcac90 0x55d547266e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f51a3e00640 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546dcac90 0x55d547266e60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=433128da941eea2e server_cookie=69632602331db6ec in_seq=0 out_seq=0 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f5190c00640 10 client.25228 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f5190c00640 1 -- 172.21.15.44:0/3519519779 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d547150340 con 0x55d546dcac90 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f5190c00640 10 client.25228 handle_command_reply: tid=0 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f51b0c7b740 1 client.25228 shutdown 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 msgr2=0x55d547200ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.923+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d547200ff0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f518c01c880 tx=0x7f518c01c690 comp rx=0 tx=0).stop 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 shutdown_connections 2024-07-26T08:37:21.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55d546dcac90 0x55d547266e60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d546e16280 0x55d547200ff0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d546e36f60 0x55d5471fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 --2- 172.21.15.44:0/3519519779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d546e0e0a0 0x55d547200a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f5190c00640 0 client.25228 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 >> 172.21.15.44:0/3519519779 conn(0x55d546e2b350 msgr2=0x55d5471d8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 shutdown_connections 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 1 -- 172.21.15.44:0/3519519779 wait complete. 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.924+0000 7f51b0c7b740 20 client.25228 trim_cache size 0 max 16384 2024-07-26T08:37:21.928 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24701, 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 255.75863597899999, 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 278.68786782400002 2024-07-26T08:37:21.929 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:37:21.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f51880b3c20 msgr2=0x7f51880b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f51880b3c20 0x7f51880b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5194101280 tx=0x7f51940303b0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 msgr2=0x7f51a419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:21.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 0x7f51a419bf20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5194035040 tx=0x7f5194032d50 comp rx=0 tx=0).stop 2024-07-26T08:37:21.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 shutdown_connections 2024-07-26T08:37:21.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f51880b3c20 0x7f51880b6130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a410b860 0x7f51a41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a410af10 0x7f51a419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 --2- 172.21.15.44:0/2142845024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51a41055d0 0x7f51a419bf20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:21.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 >> 172.21.15.44:0/2142845024 conn(0x7f51a4101040 msgr2=0x7f51a4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:21.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 shutdown_connections 2024-07-26T08:37:21.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:21.925+0000 7f51a8a00640 1 -- 172.21.15.44:0/2142845024 wait complete. 2024-07-26T08:37:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:22 smithi044 ceph-mon[32404]: pgmap v443: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:22 smithi116 ceph-mon[37097]: pgmap v443: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:22 smithi171 ceph-mon[37789]: pgmap v443: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:22.812 INFO:tasks.cephadm:Setup complete, yielding 2024-07-26T08:37:22.813 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T08:37:22.823 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:37:22.824 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch status' 2024-07-26T08:37:23.296 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:24 smithi044 ceph-mon[32404]: pgmap v444: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:24 smithi116 ceph-mon[37097]: pgmap v444: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:24 smithi171 ceph-mon[37789]: pgmap v444: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 -- 172.21.15.44:0/3126279863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410ba70 msgr2=0x7fb6941122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 --2- 172.21.15.44:0/3126279863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410ba70 0x7fb6941122a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009be0 tx=0x7fb68c02a6c0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 -- 172.21.15.44:0/3126279863 shutdown_connections 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 --2- 172.21.15.44:0/3126279863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410ba70 0x7fb6941122a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 --2- 172.21.15.44:0/3126279863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb69410b120 0x7fb69410b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 --2- 172.21.15.44:0/3126279863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6941057e0 0x7fb694105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 -- 172.21.15.44:0/3126279863 >> 172.21.15.44:0/3126279863 conn(0x7fb6941011f0 msgr2=0x7fb694103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:25.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 -- 172.21.15.44:0/3126279863 shutdown_connections 2024-07-26T08:37:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.501+0000 7fb699800640 1 -- 172.21.15.44:0/3126279863 wait complete. 2024-07-26T08:37:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.503+0000 7fb699800640 1 Processor -- start 2024-07-26T08:37:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.503+0000 7fb699800640 1 -- start start 2024-07-26T08:37:25.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.503+0000 7fb699800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 0x7fb69419c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:25.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.503+0000 7fb699800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410b120 0x7fb69419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:25.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.503+0000 7fb699800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 0x7fb6941a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:25.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb699800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb694115720 con 0x7fb69410ba70 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb699800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb6941155a0 con 0x7fb6941057e0 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb699800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb6941158a0 con 0x7fb69410b120 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 0x7fb6941a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb692a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 0x7fb69419c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb68be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410b120 0x7fb69419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 0x7fb6941a19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46412/0 (socket says 172.21.15.44:46412) 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb692a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 0x7fb69419c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47738/0 (socket says 172.21.15.44:47738) 2024-07-26T08:37:25.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 -- 172.21.15.44:0/4004460179 learned_addr learned my addr 172.21.15.44:0/4004460179 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 -- 172.21.15.44:0/4004460179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410b120 msgr2=0x7fb69419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410b120 0x7fb69419c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 -- 172.21.15.44:0/4004460179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 msgr2=0x7fb69419c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 0x7fb69419c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb693400640 1 -- 172.21.15.44:0/4004460179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6941a2120 con 0x7fb69410ba70 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.504+0000 7fb692a00640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 0x7fb69419c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:25.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.505+0000 7fb693400640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 0x7fb6941a19f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb68c004af0 tx=0x7fb68c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:25.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.505+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c041020 con 0x7fb69410ba70 2024-07-26T08:37:25.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.505+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6941a23b0 con 0x7fb69410ba70 2024-07-26T08:37:25.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.505+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb68c003e10 con 0x7fb69410ba70 2024-07-26T08:37:25.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.505+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c036040 con 0x7fb69410ba70 2024-07-26T08:37:25.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.505+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6941a8b70 con 0x7fb69410ba70 2024-07-26T08:37:25.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.507+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb694105bf0 con 0x7fb69410ba70 2024-07-26T08:37:25.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.511+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fb68c044020 con 0x7fb69410ba70 2024-07-26T08:37:25.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.512+0000 7fb68b400640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb6780b3ca0 0x7fb6780b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:25.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.512+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb68c104ff0 con 0x7fb69410ba70 2024-07-26T08:37:25.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.512+0000 7fb692a00640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb6780b3ca0 0x7fb6780b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:25.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.513+0000 7fb692a00640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb6780b3ca0 0x7fb6780b61b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6840096c0 tx=0x7fb684009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:25.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.514+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb68c04b020 con 0x7fb69410ba70 2024-07-26T08:37:25.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.737+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fb694063230 con 0x7fb6780b3ca0 2024-07-26T08:37:25.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.740+0000 7fb68b400640 1 -- 172.21.15.44:0/4004460179 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fb694063230 con 0x7fb6780b3ca0 2024-07-26T08:37:25.741 INFO:teuthology.orchestra.run.smithi044.stdout:Backend: cephadm 2024-07-26T08:37:25.741 INFO:teuthology.orchestra.run.smithi044.stdout:Available: Yes 2024-07-26T08:37:25.741 INFO:teuthology.orchestra.run.smithi044.stdout:Paused: No 2024-07-26T08:37:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb6780b3ca0 msgr2=0x7fb6780b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb6780b3ca0 0x7fb6780b61b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6840096c0 tx=0x7fb684009260 comp rx=0 tx=0).stop 2024-07-26T08:37:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 msgr2=0x7fb6941a19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:25.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 0x7fb6941a19f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb68c004af0 tx=0x7fb68c003c20 comp rx=0 tx=0).stop 2024-07-26T08:37:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 shutdown_connections 2024-07-26T08:37:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb6780b3ca0 0x7fb6780b61b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb69410b120 0x7fb69419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb69410ba70 0x7fb6941a19f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 --2- 172.21.15.44:0/4004460179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6941057e0 0x7fb69419c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:25.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 >> 172.21.15.44:0/4004460179 conn(0x7fb6941011f0 msgr2=0x7fb694104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:25.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.743+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 shutdown_connections 2024-07-26T08:37:25.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:25.744+0000 7fb699800640 1 -- 172.21.15.44:0/4004460179 wait complete. 2024-07-26T08:37:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:26 smithi044 ceph-mon[32404]: pgmap v445: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:26 smithi116 ceph-mon[37097]: pgmap v445: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:26 smithi171 ceph-mon[37789]: pgmap v445: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:26.685 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-26T08:37:27.156 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:27.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:27 smithi044 ceph-mon[32404]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:27 smithi171 ceph-mon[37789]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:27 smithi116 ceph-mon[37097]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:28.534 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:28 smithi044 ceph-mon[32404]: pgmap v446: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:37:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:28 smithi171 ceph-mon[37789]: pgmap v446: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:37:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:28 smithi116 ceph-mon[37097]: pgmap v446: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:37:29.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 -- 172.21.15.44:0/1013653352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc107be0 msgr2=0x7f02fc107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:29.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 --2- 172.21.15.44:0/1013653352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc107be0 0x7f02fc107ff0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009900 tx=0x7f02e402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 -- 172.21.15.44:0/1013653352 shutdown_connections 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 --2- 172.21.15.44:0/1013653352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc108530 0x7f02fc10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 --2- 172.21.15.44:0/1013653352 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc107be0 0x7f02fc107ff0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 --2- 172.21.15.44:0/1013653352 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc1022a0 0x7f02fc1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.447+0000 7f02fbe00640 1 -- 172.21.15.44:0/1013653352 >> 172.21.15.44:0/1013653352 conn(0x7f02fc0fdcf0 msgr2=0x7f02fc100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.448+0000 7f02fbe00640 1 -- 172.21.15.44:0/1013653352 shutdown_connections 2024-07-26T08:37:29.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.448+0000 7f02fbe00640 1 -- 172.21.15.44:0/1013653352 wait complete. 2024-07-26T08:37:29.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.449+0000 7f02fbe00640 1 Processor -- start 2024-07-26T08:37:29.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 -- start start 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 0x7f02fc199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc107be0 0x7f02fc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc108530 0x7f02fc19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1121f0 con 0x7f02fc1022a0 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc112070 con 0x7f02fc107be0 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02fbe00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc112370 con 0x7f02fc108530 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02faa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 0x7f02fc199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:29.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02faa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 0x7f02fc199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46428/0 (socket says 172.21.15.44:46428) 2024-07-26T08:37:29.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.450+0000 7f02faa00640 1 -- 172.21.15.44:0/2883637130 learned_addr learned my addr 172.21.15.44:0/2883637130 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:29.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02f3e00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc107be0 0x7f02fc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:29.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02fb400640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc108530 0x7f02fc19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:29.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02faa00640 1 -- 172.21.15.44:0/2883637130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc108530 msgr2=0x7f02fc19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:29.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02faa00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc108530 0x7f02fc19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02faa00640 1 -- 172.21.15.44:0/2883637130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc107be0 msgr2=0x7f02fc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:29.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02faa00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc107be0 0x7f02fc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02faa00640 1 -- 172.21.15.44:0/2883637130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc19fc60 con 0x7f02fc1022a0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02fb400640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc108530 0x7f02fc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.451+0000 7f02faa00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 0x7f02fc199cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.452+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00cd80 con 0x7f02fc1022a0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.452+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02ec0108e0 con 0x7f02fc1022a0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.452+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec018980 con 0x7f02fc1022a0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.452+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc19ff50 con 0x7f02fc1022a0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.452+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc1a66b0 con 0x7f02fc1022a0 2024-07-26T08:37:29.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.453+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc1026b0 con 0x7f02fc1022a0 2024-07-26T08:37:29.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.456+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f02ec010430 con 0x7f02fc1022a0 2024-07-26T08:37:29.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.459+0000 7f0300a00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f02e00b3ca0 0x7f02e00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:29.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.459+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f02ec0e49d0 con 0x7f02fc1022a0 2024-07-26T08:37:29.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.459+0000 7f02f3e00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f02e00b3ca0 0x7f02e00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:29.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.460+0000 7f02f3e00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f02e00b3ca0 0x7f02e00b61b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f02e402a9e0 tx=0x7f02e4005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:29.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.461+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f02ec0abb10 con 0x7f02fc1022a0 2024-07-26T08:37:29.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.672+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f02fc06afa0 con 0x7f02e00b3ca0 2024-07-26T08:37:29.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.686+0000 7f0300a00640 1 -- 172.21.15.44:0/2883637130 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f02fc06afa0 con 0x7f02e00b3ca0 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.a smithi044 running (4m) 2m ago 4m 80.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa e06cea071add 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.b smithi116 running (4m) 2m ago 4m 93.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa bad20a32fc44 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.c smithi171 running (5m) 2m ago 5m 85.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1ffdcba20478 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.d smithi044 running (5m) 2m ago 5m 82.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 75404d7d9775 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.e smithi116 running (5m) 2m ago 5m 83.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 58f14acae1ac 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.f smithi171 running (5m) 2m ago 5m 84.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 22bb25534d60 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.g smithi044 running (4m) 2m ago 4m 84.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d9bdde87056b 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.h smithi116 running (5m) 2m ago 5m 85.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 39f632c20084 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.i smithi171 running (4m) 2m ago 4m 88.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa a8ae39647f3a 2024-07-26T08:37:29.689 INFO:teuthology.orchestra.run.smithi044.stdout:mds.j smithi044 running (5m) 2m ago 5m 80.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa bd4dfc52cffd 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mds.k smithi116 running (5m) 2m ago 5m 86.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 484d029805f4 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mds.l smithi171 running (5m) 2m ago 5m 84.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 2b05a4793a1b 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi044 *:9283,8765 running (14m) 2m ago 14m 584M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d9075cef9982 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi116 *:8443,8765 running (12m) 2m ago 12m 524M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 48736d96195b 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.z smithi171 *:8443,8765 running (12m) 2m ago 12m 525M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa b5c1d34d8160 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (14m) 2m ago 14m 117M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa dde30f330908 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi116 running (12m) 2m ago 12m 113M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 0b9bd2d45187 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi171 running (12m) 2m ago 12m 112M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 8130b0964b03 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (10m) 2m ago 10m 113M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 63c2f4d08126 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi116 running (10m) 2m ago 10m 122M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa c04734ada693 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi171 running (9m) 2m ago 9m 125M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa eed73cf1c3f3 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (9m) 2m ago 9m 110M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e2b9a3dbdec9 2024-07-26T08:37:29.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi116 running (8m) 2m ago 8m 125M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa f0a443f96cf4 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi171 running (8m) 2m ago 8m 121M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 9259af394438 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi044 running (8m) 2m ago 8m 121M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 69a7880ed7d0 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi116 running (7m) 2m ago 7m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 3d75033cea40 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 smithi171 running (7m) 2m ago 7m 113M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa bbf088c31e0b 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 smithi044 running (6m) 2m ago 6m 108M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 700a0ae3485b 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 smithi116 running (6m) 2m ago 6m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa ed71d2de1f46 2024-07-26T08:37:29.691 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 smithi171 running (5m) 2m ago 5m 120M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 232287c7d84a 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.688+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f02e00b3ca0 msgr2=0x7f02e00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.688+0000 7f02fbe00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f02e00b3ca0 0x7f02e00b61b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f02e402a9e0 tx=0x7f02e4005a20 comp rx=0 tx=0).stop 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 msgr2=0x7f02fc199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 0x7f02fc199cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 shutdown_connections 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f02e00b3ca0 0x7f02e00b61b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02fc108530 0x7f02fc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc107be0 0x7f02fc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 --2- 172.21.15.44:0/2883637130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc1022a0 0x7f02fc199cf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 >> 172.21.15.44:0/2883637130 conn(0x7f02fc0fdcf0 msgr2=0x7f02fc06a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 shutdown_connections 2024-07-26T08:37:29.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:29.689+0000 7f02fbe00640 1 -- 172.21.15.44:0/2883637130 wait complete. 2024-07-26T08:37:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:30 smithi044 ceph-mon[32404]: pgmap v447: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:37:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:30 smithi171 ceph-mon[37789]: pgmap v447: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:37:30.695 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-26T08:37:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:30 smithi116 ceph-mon[37097]: pgmap v447: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T08:37:31.204 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:31.570 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:31 smithi044 ceph-mon[32404]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:31 smithi171 ceph-mon[37789]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:31 smithi116 ceph-mon[37097]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:32 smithi044 ceph-mon[32404]: pgmap v448: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:32 smithi171 ceph-mon[37789]: pgmap v448: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:32 smithi116 ceph-mon[37097]: pgmap v448: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 -- 172.21.15.44:0/2049551634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 msgr2=0x7f8d74109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 --2- 172.21.15.44:0/2049551634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d74109e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009900 tx=0x7f8d6402a400 comp rx=0 tx=0).stop 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 -- 172.21.15.44:0/2049551634 shutdown_connections 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 --2- 172.21.15.44:0/2049551634 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410fc80 0x7f8d741164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 --2- 172.21.15.44:0/2049551634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7410f330 0x7f8d7410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 --2- 172.21.15.44:0/2049551634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d74109e00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 -- 172.21.15.44:0/2049551634 >> 172.21.15.44:0/2049551634 conn(0x7f8d74078ff0 msgr2=0x7f8d74107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:33.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 -- 172.21.15.44:0/2049551634 shutdown_connections 2024-07-26T08:37:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.516+0000 7f8d73200640 1 -- 172.21.15.44:0/2049551634 wait complete. 2024-07-26T08:37:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.517+0000 7f8d73200640 1 Processor -- start 2024-07-26T08:37:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 -- start start 2024-07-26T08:37:33.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d741a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410f330 0x7f8d741a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7410fc80 0x7f8d741a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74119990 con 0x7f8d741099f0 2024-07-26T08:37:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74119810 con 0x7f8d7410fc80 2024-07-26T08:37:33.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.518+0000 7f8d73200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74119b10 con 0x7f8d7410f330 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d71e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d741a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d71e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d741a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46870/0 (socket says 172.21.15.44:46870) 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d71e00640 1 -- 172.21.15.44:0/2545739454 learned_addr learned my addr 172.21.15.44:0/2545739454 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d6be00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410f330 0x7f8d741a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d72800640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7410fc80 0x7f8d741a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d6be00640 1 -- 172.21.15.44:0/2545739454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7410fc80 msgr2=0x7f8d741a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d6be00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7410fc80 0x7f8d741a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d6be00640 1 -- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 msgr2=0x7f8d741a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:33.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d6be00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d741a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.519+0000 7f8d6be00640 1 -- 172.21.15.44:0/2545739454 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d741a6350 con 0x7f8d7410f330 2024-07-26T08:37:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.520+0000 7f8d6be00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410f330 0x7f8d741a0980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8d7406c270 tx=0x7f8d5c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.520+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00cc40 con 0x7f8d7410f330 2024-07-26T08:37:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.520+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d741a6640 con 0x7f8d7410f330 2024-07-26T08:37:33.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.520+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d5c00cde0 con 0x7f8d7410f330 2024-07-26T08:37:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.520+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c018760 con 0x7f8d7410f330 2024-07-26T08:37:33.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.520+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d741acda0 con 0x7f8d7410f330 2024-07-26T08:37:33.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.522+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d74109e00 con 0x7f8d7410f330 2024-07-26T08:37:33.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.524+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f8d5c010a90 con 0x7f8d7410f330 2024-07-26T08:37:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.526+0000 7f8d6aa00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d580b3c20 0x7f8d580b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.526+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8d5c0e4400 con 0x7f8d7410f330 2024-07-26T08:37:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.527+0000 7f8d71e00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d580b3c20 0x7f8d580b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:33.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.527+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8d5c0aaef0 con 0x7f8d7410f330 2024-07-26T08:37:33.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.528+0000 7f8d71e00640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d580b3c20 0x7f8d580b6130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8d640096c0 tx=0x7f8d64005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:33.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.738+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8d74111ab0 con 0x7f8d580b3c20 2024-07-26T08:37:33.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.745+0000 7f8d6aa00640 1 -- 172.21.15.44:0/2545739454 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8d74111ab0 con 0x7f8d580b3c20 2024-07-26T08:37:33.746 INFO:teuthology.orchestra.run.smithi044.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-26T08:37:33.746 INFO:teuthology.orchestra.run.smithi044.stdout:mds.all 12/12 2m ago 5m smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-07-26T08:37:33.747 INFO:teuthology.orchestra.run.smithi044.stdout:mgr 3/3 2m ago 12m smithi044=x;smithi116=y;smithi171=z;count:3 2024-07-26T08:37:33.747 INFO:teuthology.orchestra.run.smithi044.stdout:mon 3/3 2m ago 12m smithi044:172.21.15.44=a;smithi116:172.21.15.116=b;smithi171:172.21.15.171=c;count:3 2024-07-26T08:37:33.747 INFO:teuthology.orchestra.run.smithi044.stdout:osd 12 2m ago - 2024-07-26T08:37:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.747+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d580b3c20 msgr2=0x7f8d580b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.747+0000 7f8d73200640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d580b3c20 0x7f8d580b6130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8d640096c0 tx=0x7f8d64005a20 comp rx=0 tx=0).stop 2024-07-26T08:37:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.747+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410f330 msgr2=0x7f8d741a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.747+0000 7f8d73200640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410f330 0x7f8d741a0980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8d7406c270 tx=0x7f8d5c00ee50 comp rx=0 tx=0).stop 2024-07-26T08:37:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 shutdown_connections 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8d580b3c20 0x7f8d580b6130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d7410f330 0x7f8d741a0980 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7410fc80 0x7f8d741a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 --2- 172.21.15.44:0/2545739454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d741099f0 0x7f8d741a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 >> 172.21.15.44:0/2545739454 conn(0x7f8d74078ff0 msgr2=0x7f8d74107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 shutdown_connections 2024-07-26T08:37:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:33.748+0000 7f8d73200640 1 -- 172.21.15.44:0/2545739454 wait complete. 2024-07-26T08:37:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:34 smithi044 ceph-mon[32404]: pgmap v449: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:34.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:34 smithi116 ceph-mon[37097]: pgmap v449: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:34 smithi171 ceph-mon[37789]: pgmap v449: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:34.609 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-26T08:37:35.102 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:35 smithi044 ceph-mon[32404]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:35 smithi171 ceph-mon[37789]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:35 smithi116 ceph-mon[37097]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:36 smithi044 ceph-mon[32404]: pgmap v450: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:36 smithi171 ceph-mon[37789]: pgmap v450: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:36 smithi116 ceph-mon[37097]: pgmap v450: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:37:37.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.560+0000 7fb16f200640 1 -- 172.21.15.44:0/2485769402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 msgr2=0x7fb168111de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.560+0000 7fb16f200640 1 --2- 172.21.15.44:0/2485769402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 0x7fb168111de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb164009be0 tx=0x7fb16402a750 comp rx=0 tx=0).stop 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 -- 172.21.15.44:0/2485769402 shutdown_connections 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 --2- 172.21.15.44:0/2485769402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 0x7fb16810f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 --2- 172.21.15.44:0/2485769402 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 0x7fb168111de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 --2- 172.21.15.44:0/2485769402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb168102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 -- 172.21.15.44:0/2485769402 >> 172.21.15.44:0/2485769402 conn(0x7fb1680fdcf0 msgr2=0x7fb168100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 -- 172.21.15.44:0/2485769402 shutdown_connections 2024-07-26T08:37:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.561+0000 7fb16f200640 1 -- 172.21.15.44:0/2485769402 wait complete. 2024-07-26T08:37:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.562+0000 7fb16f200640 1 Processor -- start 2024-07-26T08:37:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.562+0000 7fb16f200640 1 -- start start 2024-07-26T08:37:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb16819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16f200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 0x7fb16819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 0x7fb1681a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb168115480 con 0x7fb16810f9c0 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb168115300 con 0x7fb168101e10 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16f200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb168115600 con 0x7fb168102760 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb16819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb16819be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:45246/0 (socket says 172.21.15.44:45246) 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16de00640 1 -- 172.21.15.44:0/3364430947 learned_addr learned my addr 172.21.15.44:0/3364430947 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16e800640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 0x7fb1681a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.563+0000 7fb16d400640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 0x7fb16819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16de00640 1 -- 172.21.15.44:0/3364430947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 msgr2=0x7fb16819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16de00640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 0x7fb16819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16de00640 1 -- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 msgr2=0x7fb1681a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16de00640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 0x7fb1681a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16de00640 1 -- 172.21.15.44:0/3364430947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1681a1de0 con 0x7fb168101e10 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16d400640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 0x7fb16819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16de00640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb16819be70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00e9c0 tx=0x7fb15c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00cde0 con 0x7fb168101e10 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1681a20d0 con 0x7fb168101e10 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb15c0035f0 con 0x7fb168101e10 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.564+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c010690 con 0x7fb168101e10 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.565+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1681a8830 con 0x7fb168101e10 2024-07-26T08:37:37.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.566+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb168102220 con 0x7fb168101e10 2024-07-26T08:37:37.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.568+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fb15c004000 con 0x7fb168101e10 2024-07-26T08:37:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.570+0000 7fb162a00640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1500b3ca0 0x7fb1500b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:37.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.570+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb15c0e4550 con 0x7fb168101e10 2024-07-26T08:37:37.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.570+0000 7fb16d400640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1500b3ca0 0x7fb1500b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.572+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb15c0ab690 con 0x7fb168101e10 2024-07-26T08:37:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.572+0000 7fb16d400640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1500b3ca0 0x7fb1500b61b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1580095f0 tx=0x7fb158005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:37.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.781+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fb168063230 con 0x7fb1500b3ca0 2024-07-26T08:37:37.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.784+0000 7fb162a00640 1 -- 172.21.15.44:0/3364430947 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fb168063230 con 0x7fb1500b3ca0 2024-07-26T08:37:37.785 INFO:teuthology.orchestra.run.smithi044.stdout:HOST ADDR LABELS STATUS 2024-07-26T08:37:37.786 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 172.21.15.44 2024-07-26T08:37:37.786 INFO:teuthology.orchestra.run.smithi044.stdout:smithi116 172.21.15.116 2024-07-26T08:37:37.786 INFO:teuthology.orchestra.run.smithi044.stdout:smithi171 172.21.15.171 2024-07-26T08:37:37.786 INFO:teuthology.orchestra.run.smithi044.stdout:3 hosts in cluster 2024-07-26T08:37:37.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1500b3ca0 msgr2=0x7fb1500b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:37.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1500b3ca0 0x7fb1500b61b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1580095f0 tx=0x7fb158005f90 comp rx=0 tx=0).stop 2024-07-26T08:37:37.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 msgr2=0x7fb16819be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:37.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb16819be70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00e9c0 tx=0x7fb15c00ee80 comp rx=0 tx=0).stop 2024-07-26T08:37:37.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 shutdown_connections 2024-07-26T08:37:37.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1500b3ca0 0x7fb1500b61b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb168102760 0x7fb16819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb16810f9c0 0x7fb1681a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 --2- 172.21.15.44:0/3364430947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168101e10 0x7fb16819be70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:37.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.786+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 >> 172.21.15.44:0/3364430947 conn(0x7fb1680fdcf0 msgr2=0x7fb168111b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:37.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.787+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 shutdown_connections 2024-07-26T08:37:37.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:37.787+0000 7fb16f200640 1 -- 172.21.15.44:0/3364430947 wait complete. 2024-07-26T08:37:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:38 smithi044 ceph-mon[32404]: pgmap v451: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:37:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:38 smithi171 ceph-mon[37789]: pgmap v451: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:37:38.749 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-26T08:37:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:38 smithi116 ceph-mon[37097]: pgmap v451: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:37:39.250 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:39.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:39 smithi044 ceph-mon[32404]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:39 smithi171 ceph-mon[37789]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:39 smithi116 ceph-mon[37097]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:40 smithi171 ceph-mon[37789]: pgmap v452: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:40.672 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:40 smithi044 ceph-mon[32404]: pgmap v452: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:40 smithi116 ceph-mon[37097]: pgmap v452: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 -- 172.21.15.44:0/3241918275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35681043e0 msgr2=0x7f35681047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 --2- 172.21.15.44:0/3241918275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35681043e0 0x7f35681047f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3558009900 tx=0x7f355802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 -- 172.21.15.44:0/3241918275 shutdown_connections 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 --2- 172.21.15.44:0/3241918275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3568104d30 0x7f356810d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 --2- 172.21.15.44:0/3241918275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f356810d840 0x7f356810fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 --2- 172.21.15.44:0/3241918275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f35681043e0 0x7f35681047f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 -- 172.21.15.44:0/3241918275 >> 172.21.15.44:0/3241918275 conn(0x7f3568079160 msgr2=0x7f3568079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:41.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 -- 172.21.15.44:0/3241918275 shutdown_connections 2024-07-26T08:37:41.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.575+0000 7f356e600640 1 -- 172.21.15.44:0/3241918275 wait complete. 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.576+0000 7f356e600640 1 Processor -- start 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.576+0000 7f356e600640 1 -- start start 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f356e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 0x7f356819c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f356e600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f356810d840 0x7f356819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f356e600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35681a19b0 0x7f35681a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f356e600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f35681134b0 con 0x7f356810d840 2024-07-26T08:37:41.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f356e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113330 con 0x7f3568104d30 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f356e600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3568113630 con 0x7f35681a19b0 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f3566a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f356810d840 0x7f356819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f3567400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 0x7f356819c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f3567400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 0x7f356819c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34386/0 (socket says 172.21.15.44:34386) 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f3567400640 1 -- 172.21.15.44:0/906461607 learned_addr learned my addr 172.21.15.44:0/906461607 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.577+0000 7f3567e00640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35681a19b0 0x7f35681a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3567400640 1 -- 172.21.15.44:0/906461607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35681a19b0 msgr2=0x7f35681a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3567400640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35681a19b0 0x7f35681a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3567400640 1 -- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f356810d840 msgr2=0x7f356819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3567400640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f356810d840 0x7f356819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3567400640 1 -- 172.21.15.44:0/906461607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35681a2590 con 0x7f3568104d30 2024-07-26T08:37:41.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3566a00640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f356810d840 0x7f356819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:41.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f3567400640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 0x7f356819c230 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3558006010 tx=0x7f35580040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:41.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.578+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355803f030 con 0x7f3568104d30 2024-07-26T08:37:41.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.579+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35681a8b30 con 0x7f3568104d30 2024-07-26T08:37:41.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.579+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35580044c0 con 0x7f3568104d30 2024-07-26T08:37:41.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.579+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558035070 con 0x7f3568104d30 2024-07-26T08:37:41.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.579+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35681a8ed0 con 0x7f3568104d30 2024-07-26T08:37:41.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.580+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35681047f0 con 0x7f3568104d30 2024-07-26T08:37:41.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.583+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f3558042030 con 0x7f3568104d30 2024-07-26T08:37:41.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.584+0000 7f355fe00640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f354c0b3c20 0x7f354c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:41.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.584+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f35581042d0 con 0x7f3568104d30 2024-07-26T08:37:41.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.584+0000 7f3566a00640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f354c0b3c20 0x7f354c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:41.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.585+0000 7f3566a00640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f354c0b3c20 0x7f354c0b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f356810f670 tx=0x7f355400b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:41.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.586+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3558049070 con 0x7f3568104d30 2024-07-26T08:37:41.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.802+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3568100750 con 0x7f354c0b3c20 2024-07-26T08:37:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.807+0000 7f355fe00640 1 -- 172.21.15.44:0/906461607 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3568100750 con 0x7f354c0b3c20 2024-07-26T08:37:41.808 INFO:teuthology.orchestra.run.smithi044.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-26T08:37:41.808 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001K400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T08:37:41.808 INFO:teuthology.orchestra.run.smithi044.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T08:37:41.809 INFO:teuthology.orchestra.run.smithi044.stdout:smithi171 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001S400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T08:37:41.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f354c0b3c20 msgr2=0x7f354c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:41.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f354c0b3c20 0x7f354c0b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f356810f670 tx=0x7f355400b390 comp rx=0 tx=0).stop 2024-07-26T08:37:41.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 msgr2=0x7f356819c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 0x7f356819c230 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3558006010 tx=0x7f35580040c0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 shutdown_connections 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f354c0b3c20 0x7f354c0b6130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35681a19b0 0x7f35681a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f356810d840 0x7f356819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 --2- 172.21.15.44:0/906461607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3568104d30 0x7f356819c230 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:41.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 >> 172.21.15.44:0/906461607 conn(0x7f3568079160 msgr2=0x7f3568100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.810+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 shutdown_connections 2024-07-26T08:37:41.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:41.811+0000 7f356e600640 1 -- 172.21.15.44:0/906461607 wait complete. 2024-07-26T08:37:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:42 smithi044 ceph-mon[32404]: pgmap v453: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:42 smithi116 ceph-mon[37097]: pgmap v453: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:42.785 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T08:37:42.795 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:37:42.795 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-26T08:37:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:42 smithi171 ceph-mon[37789]: pgmap v453: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:43.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:43 smithi044 ceph-mon[32404]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:43 smithi116 ceph-mon[37097]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:43 smithi171 ceph-mon[37789]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:37:44.486 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:44 smithi044 ceph-mon[32404]: pgmap v454: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:44 smithi116 ceph-mon[37097]: pgmap v454: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:44 smithi171 ceph-mon[37789]: pgmap v454: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 -- 172.21.15.44:0/3093146946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc41055c0 msgr2=0x7fcdc41059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/3093146946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc41055c0 0x7fcdc41059d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009900 tx=0x7fcdb402a410 comp rx=0 tx=0).stop 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 -- 172.21.15.44:0/3093146946 shutdown_connections 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/3093146946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc410af00 0x7fcdc410b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/3093146946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc410b850 0x7fcdc4112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/3093146946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc41055c0 0x7fcdc41059d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 -- 172.21.15.44:0/3093146946 >> 172.21.15.44:0/3093146946 conn(0x7fcdc4101010 msgr2=0x7fcdc4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 -- 172.21.15.44:0/3093146946 shutdown_connections 2024-07-26T08:37:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.616+0000 7fcdc8a00640 1 -- 172.21.15.44:0/3093146946 wait complete. 2024-07-26T08:37:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 Processor -- start 2024-07-26T08:37:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 -- start start 2024-07-26T08:37:45.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc41055c0 0x7fcdc419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 0x7fcdc419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc410b850 0x7fcdc41a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4115520 con 0x7fcdc410af00 2024-07-26T08:37:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41153a0 con 0x7fcdc410b850 2024-07-26T08:37:45.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.618+0000 7fcdc8a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41156a0 con 0x7fcdc41055c0 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdbaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 0x7fcdc419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc41055c0 0x7fcdc419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdbaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 0x7fcdc419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44082/0 (socket says 172.21.15.44:44082) 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdbaa00640 1 -- 172.21.15.44:0/371213648 learned_addr learned my addr 172.21.15.44:0/371213648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3e00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc410b850 0x7fcdc41a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3400640 1 -- 172.21.15.44:0/371213648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc410b850 msgr2=0x7fcdc41a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3400640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc410b850 0x7fcdc41a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3400640 1 -- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 msgr2=0x7fcdc419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3400640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 0x7fcdc419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdc3400640 1 -- 172.21.15.44:0/371213648 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc41a1ee0 con 0x7fcdc41055c0 2024-07-26T08:37:45.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.619+0000 7fcdbaa00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 0x7fcdc419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.620+0000 7fcdc3400640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc41055c0 0x7fcdc419bf70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcdb402a930 tx=0x7fcdb4031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.620+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb403f030 con 0x7fcdc41055c0 2024-07-26T08:37:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.620+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc41a2170 con 0x7fcdc41055c0 2024-07-26T08:37:45.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.620+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb40043d0 con 0x7fcdc41055c0 2024-07-26T08:37:45.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.620+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4035070 con 0x7fcdc41055c0 2024-07-26T08:37:45.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.620+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc41a8930 con 0x7fcdc41055c0 2024-07-26T08:37:45.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.621+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc41059d0 con 0x7fcdc41055c0 2024-07-26T08:37:45.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.624+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fcdb4042030 con 0x7fcdc41055c0 2024-07-26T08:37:45.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.625+0000 7fcdc0c00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcda80b3c20 0x7fcda80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.625+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fcdb4104520 con 0x7fcdc41055c0 2024-07-26T08:37:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.626+0000 7fcdbaa00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcda80b3c20 0x7fcda80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.626+0000 7fcdbaa00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcda80b3c20 0x7fcda80b6130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcdc410aae0 tx=0x7fcdac005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:45.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.627+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcdb40cb6c0 con 0x7fcdc41055c0 2024-07-26T08:37:45.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.870+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fcdc419d0a0 con 0x7fcdc41055c0 2024-07-26T08:37:45.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.871+0000 7fcdc0c00640 1 -- 172.21.15.44:0/371213648 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) ==== 76+0+3140 (secure 0 0 0) 0x7fcdb40cafc0 con 0x7fcdc41055c0 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:e29 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:btime 2024-07-26T08:34:37:118441+0000 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-07-26T08:37:45.874 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 29 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-07-26T08:33:06.110522+0000 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-07-26T08:34:36.302968+0000 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 300 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-07-26T08:37:45.875 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 5 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:in 0,1,2,3,4 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=24701,1=24686,2=24679,3=24668,4=24661} 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3,4] 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-07-26T08:37:45.876 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 0 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 24701 members: 24661,24668,24679,24686,24701 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b{0:24701} state up:active seq 7 addr [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.f{0:24655} state up:standby-replay seq 1 addr [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.g{1:24686} state up:active seq 30 addr [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.e{1:24650} state up:standby-replay seq 1 addr [v2:172.21.15.116:6834/1052813979,v1:172.21.15.116:6835/1052813979] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.i{2:24679} state up:active seq 31 addr [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d{2:24638} state up:standby-replay seq 1 addr [v2:172.21.15.44:6836/3938332919,v1:172.21.15.44:6837/3938332919] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.h{3:24668} state up:active seq 32 addr [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.k{3:24620} state up:standby-replay seq 1 addr [v2:172.21.15.116:6832/2097534847,v1:172.21.15.116:6833/2097534847] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c{4:24661} state up:active seq 33 addr [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.877 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.l{4:24613} state up:standby-replay seq 1 addr [v2:172.21.15.171:6832/4260884949,v1:172.21.15.171:6833/4260884949] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.878 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:45.878 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:45.878 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-07-26T08:37:45.878 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:45.878 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.j{-1:14526} state up:standby seq 1 addr [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.878 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a{-1:14574} state up:standby seq 1 addr [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcda80b3c20 msgr2=0x7fcda80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcda80b3c20 0x7fcda80b6130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcdc410aae0 tx=0x7fcdac005f90 comp rx=0 tx=0).stop 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc41055c0 msgr2=0x7fcdc419bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc41055c0 0x7fcdc419bf70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcdb402a930 tx=0x7fcdb4031a20 comp rx=0 tx=0).stop 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 shutdown_connections 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcda80b3c20 0x7fcda80b6130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc410b850 0x7fcdc41a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdc410af00 0x7fcdc419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 --2- 172.21.15.44:0/371213648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcdc41055c0 0x7fcdc419bf70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.874+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 >> 172.21.15.44:0/371213648 conn(0x7fcdc4101010 msgr2=0x7fcdc4103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.875+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 shutdown_connections 2024-07-26T08:37:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:45.875+0000 7fcdc8a00640 1 -- 172.21.15.44:0/371213648 wait complete. 2024-07-26T08:37:46.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:46 smithi044 ceph-mon[32404]: pgmap v455: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:46.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:46 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/371213648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T08:37:46.766 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-26T08:37:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:46 smithi116 ceph-mon[37097]: pgmap v455: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:46 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/371213648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T08:37:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:46 smithi171 ceph-mon[37789]: pgmap v455: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:46 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/371213648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T08:37:47.254 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:48.636 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:48 smithi044 ceph-mon[32404]: pgmap v456: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:48 smithi116 ceph-mon[37097]: pgmap v456: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:48 smithi171 ceph-mon[37789]: pgmap v456: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 -- 172.21.15.44:0/757553257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c810d650 msgr2=0x7f66c810fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 --2- 172.21.15.44:0/757553257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c810d650 0x7f66c810fb40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009ca0 tx=0x7f66c002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 -- 172.21.15.44:0/757553257 shutdown_connections 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 --2- 172.21.15.44:0/757553257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 0x7f66c810d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 --2- 172.21.15.44:0/757553257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c810d650 0x7f66c810fb40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 --2- 172.21.15.44:0/757553257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c81041f0 0x7f66c8104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 -- 172.21.15.44:0/757553257 >> 172.21.15.44:0/757553257 conn(0x7f66c80fdcd0 msgr2=0x7f66c8100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 -- 172.21.15.44:0/757553257 shutdown_connections 2024-07-26T08:37:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.621+0000 7f66ce000640 1 -- 172.21.15.44:0/757553257 wait complete. 2024-07-26T08:37:49.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.622+0000 7f66ce000640 1 Processor -- start 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.622+0000 7f66ce000640 1 -- start start 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ce000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c81041f0 0x7f66c819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ce000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 0x7f66c819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ce000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c810d650 0x7f66c81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ce000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8113330 con 0x7f66c81041f0 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ce000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81131b0 con 0x7f66c810d650 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ce000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81134b0 con 0x7f66c8104b40 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ccc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c81041f0 0x7f66c819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66cd600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c810d650 0x7f66c81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ccc00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c81041f0 0x7f66c819bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44106/0 (socket says 172.21.15.44:44106) 2024-07-26T08:37:49.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66ccc00640 1 -- 172.21.15.44:0/599853586 learned_addr learned my addr 172.21.15.44:0/599853586 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:49.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.623+0000 7f66c7e00640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 0x7f66c819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:49.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66cd600640 1 -- 172.21.15.44:0/599853586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 msgr2=0x7f66c819c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:49.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66cd600640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 0x7f66c819c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66cd600640 1 -- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c81041f0 msgr2=0x7f66c819bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66cd600640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c81041f0 0x7f66c819bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66cd600640 1 -- 172.21.15.44:0/599853586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81a1e90 con 0x7f66c810d650 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66c7e00640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 0x7f66c819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.624+0000 7f66cd600640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c810d650 0x7f66c81a1760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005e30 tx=0x7f66c00317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.625+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0003b70 con 0x7f66c810d650 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.625+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c81a2120 con 0x7f66c810d650 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.625+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c81a88e0 con 0x7f66c810d650 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.625+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c0003d10 con 0x7f66c810d650 2024-07-26T08:37:49.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.625+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c002fa00 con 0x7f66c810d650 2024-07-26T08:37:49.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.626+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c8104600 con 0x7f66c810d650 2024-07-26T08:37:49.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.629+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f66c0004580 con 0x7f66c810d650 2024-07-26T08:37:49.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.630+0000 7f66c5600640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66b00b3ca0 0x7f66b00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:49.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.631+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f66c0105530 con 0x7f66c810d650 2024-07-26T08:37:49.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.631+0000 7f66ccc00640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66b00b3ca0 0x7f66b00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:49.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.632+0000 7f66ccc00640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66b00b3ca0 0x7f66b00b61b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f66bc005fd0 tx=0x7f66bc00b4a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:49.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.632+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f66c00cc670 con 0x7f66c810d650 2024-07-26T08:37:49.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.838+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f66c81a8bc0 con 0x7f66c810d650 2024-07-26T08:37:49.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.840+0000 7f66c5600640 1 -- 172.21.15.44:0/599853586 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+6091 (secure 0 0 0) 0x7f66c00cbf70 con 0x7f66c810d650 2024-07-26T08:37:49.842 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 91 2024-07-26T08:37:49.842 INFO:teuthology.orchestra.run.smithi044.stdout:fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 2024-07-26T08:37:49.842 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-07-26T08:22:42.161051+0000 2024-07-26T08:37:49.842 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-07-26T08:34:13.946257+0000 2024-07-26T08:37:49.842 INFO:teuthology.orchestra.run.smithi044.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:crush_version 27 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:full_ratio 0.95 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:backfillfull_ratio 0.9 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:nearfull_ratio 0.85 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:require_min_compat_client luminous 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:min_compat_client luminous 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:require_osd_release squid 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:stretch_mode_enabled false 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:max_osd 12 2024-07-26T08:37:49.843 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 up in weight 1 up_from 9 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6802/3315928929,v1:172.21.15.44:6803/3315928929] [v2:172.21.15.44:6804/3315928929,v1:172.21.15.44:6805/3315928929] exists,up 96ac5961-fb53-49d7-925f-8be1ece073f7 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 up in weight 1 up_from 14 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6800/2431548674,v1:172.21.15.116:6801/2431548674] [v2:172.21.15.116:6802/2431548674,v1:172.21.15.116:6803/2431548674] exists,up 853c8479-ad0a-4af4-996a-646245d17441 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 up in weight 1 up_from 19 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6800/2604165495,v1:172.21.15.171:6801/2604165495] [v2:172.21.15.171:6802/2604165495,v1:172.21.15.171:6803/2604165495] exists,up 284a11b8-2736-4d1c-90b8-59bdb63c3311 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 up in weight 1 up_from 27 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6810/2033336910,v1:172.21.15.44:6811/2033336910] [v2:172.21.15.44:6812/2033336910,v1:172.21.15.44:6813/2033336910] exists,up 895978fc-b2e6-4251-bb79-9bcb77079e00 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6808/3661934334,v1:172.21.15.116:6809/3661934334] [v2:172.21.15.116:6810/3661934334,v1:172.21.15.116:6811/3661934334] exists,up 1e29d022-e84f-43f7-badb-18797f4a3f67 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 up in weight 1 up_from 38 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6808/3188487524,v1:172.21.15.171:6809/3188487524] [v2:172.21.15.171:6810/3188487524,v1:172.21.15.171:6811/3188487524] exists,up 71c826e9-696b-47de-ae4c-023b02ceb6b4 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 up in weight 1 up_from 44 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6818/51483018,v1:172.21.15.44:6819/51483018] [v2:172.21.15.44:6820/51483018,v1:172.21.15.44:6821/51483018] exists,up 275225ee-11a7-4988-851b-7be0b8d4d155 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 up in weight 1 up_from 49 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6816/776412453,v1:172.21.15.116:6817/776412453] [v2:172.21.15.116:6818/776412453,v1:172.21.15.116:6819/776412453] exists,up cbc14308-bca8-4549-8079-6483bda3f101 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 up in weight 1 up_from 55 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6816/2299573831,v1:172.21.15.171:6817/2299573831] [v2:172.21.15.171:6818/2299573831,v1:172.21.15.171:6819/2299573831] exists,up 20dacf62-1c49-4dd9-8d1f-467506b1d1bd 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 up in weight 1 up_from 61 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6826/2869135265,v1:172.21.15.44:6827/2869135265] [v2:172.21.15.44:6828/2869135265,v1:172.21.15.44:6829/2869135265] exists,up 8fa16cab-5114-438f-bb1e-e636366e290d 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 up in weight 1 up_from 66 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6824/1630121926,v1:172.21.15.116:6825/1630121926] [v2:172.21.15.116:6826/1630121926,v1:172.21.15.116:6827/1630121926] exists,up 8cec6245-933e-4d2f-8d4f-5107276fe094 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6824/4225876845,v1:172.21.15.171:6825/4225876845] [v2:172.21.15.171:6826/4225876845,v1:172.21.15.171:6827/4225876845] exists,up f406895e-aab0-4a79-bd14-20a6f0626b29 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.a [2,11,8,4] 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.e [2,4] 2024-07-26T08:37:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.10 [8,11] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.0 [5,9] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.7 [3,9] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.11 [4,7] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.17 [4,8] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.1b [5,9] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.23 [4,8] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.3a [4,6] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.c [9,10] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.f [5,8] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.13 [9,8] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.18 [9,8] 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2545437880 expires 2024-07-27T08:25:12.575949+0000 2024-07-26T08:37:49.845 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/3951671566 expires 2024-07-27T08:23:20.144063+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/1988892875 expires 2024-07-27T08:23:50.924919+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/1840681309 expires 2024-07-27T08:25:12.575949+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/3951671566 expires 2024-07-27T08:23:20.144063+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/475987513 expires 2024-07-27T08:23:20.144063+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/1872480755 expires 2024-07-27T08:23:20.144063+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/1673664031 expires 2024-07-27T08:23:50.924919+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2363513350 expires 2024-07-27T08:23:50.924919+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/1673664031 expires 2024-07-27T08:23:50.924919+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2833279364 expires 2024-07-27T08:23:50.924919+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3302011051 expires 2024-07-27T08:23:20.144063+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/487473 expires 2024-07-27T08:25:12.575949+0000 2024-07-26T08:37:49.846 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3491826252 expires 2024-07-27T08:25:12.575949+0000 2024-07-26T08:37:49.847 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/1840681309 expires 2024-07-27T08:25:12.575949+0000 2024-07-26T08:37:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66b00b3ca0 msgr2=0x7f66b00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66b00b3ca0 0x7f66b00b61b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f66bc005fd0 tx=0x7f66bc00b4a0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c810d650 msgr2=0x7f66c81a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c810d650 0x7f66c81a1760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005e30 tx=0x7f66c00317c0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 shutdown_connections 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66b00b3ca0 0x7f66b00b61b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66c8104b40 0x7f66c819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66c810d650 0x7f66c81a1760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 --2- 172.21.15.44:0/599853586 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66c81041f0 0x7f66c819bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 >> 172.21.15.44:0/599853586 conn(0x7f66c80fdcd0 msgr2=0x7f66c810c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.844+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 shutdown_connections 2024-07-26T08:37:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:49.845+0000 7f66ce000640 1 -- 172.21.15.44:0/599853586 wait complete. 2024-07-26T08:37:50.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:50 smithi044 ceph-mon[32404]: pgmap v457: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:50.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:50 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/599853586' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T08:37:50.747 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-26T08:37:50.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:37:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:50 smithi116 ceph-mon[37097]: pgmap v457: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:50 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/599853586' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T08:37:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:50 smithi171 ceph-mon[37789]: pgmap v457: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:37:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:50 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/599853586' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T08:37:51.257 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:52 smithi044 ceph-mon[32404]: pgmap v458: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:52 smithi116 ceph-mon[37097]: pgmap v458: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:52 smithi171 ceph-mon[37789]: pgmap v458: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.732+0000 7fdd2fe00640 1 -- 172.21.15.44:0/1198329785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd301033c0 msgr2=0x7fdd301037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.732+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/1198329785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd301033c0 0x7fdd301037d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009ca0 tx=0x7fdd2002a8b0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 -- 172.21.15.44:0/1198329785 shutdown_connections 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/1198329785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd30108d00 0x7fdd30109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/1198329785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd30109650 0x7fdd3010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/1198329785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd301033c0 0x7fdd301037d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 -- 172.21.15.44:0/1198329785 >> 172.21.15.44:0/1198329785 conn(0x7fdd300fee10 msgr2=0x7fdd30101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 -- 172.21.15.44:0/1198329785 shutdown_connections 2024-07-26T08:37:53.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.733+0000 7fdd2fe00640 1 -- 172.21.15.44:0/1198329785 wait complete. 2024-07-26T08:37:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.734+0000 7fdd2fe00640 1 Processor -- start 2024-07-26T08:37:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.734+0000 7fdd2fe00640 1 -- start start 2024-07-26T08:37:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 0x7fdd30075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:53.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd30108d00 0x7fdd30075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd30109650 0x7fdd30075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2fe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30113320 con 0x7fdd30108d00 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301131a0 con 0x7fdd301033c0 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2fe00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301134a0 con 0x7fdd30109650 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 0x7fdd30075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 0x7fdd30075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:45074/0 (socket says 172.21.15.44:45074) 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2ea00640 1 -- 172.21.15.44:0/4250238119 learned_addr learned my addr 172.21.15.44:0/4250238119 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:53.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2e000640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd30108d00 0x7fdd30075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.735+0000 7fdd2f400640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd30109650 0x7fdd30075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd2ea00640 1 -- 172.21.15.44:0/4250238119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd30109650 msgr2=0x7fdd30075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd2ea00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd30109650 0x7fdd30075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd2ea00640 1 -- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd30108d00 msgr2=0x7fdd30075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd2ea00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd30108d00 0x7fdd30075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd2ea00640 1 -- 172.21.15.44:0/4250238119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd301a7fd0 con 0x7fdd301033c0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd2ea00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 0x7fdd30075140 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdd20006010 tx=0x7fdd20030cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.736+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20004040 con 0x7fdd301033c0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.737+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd301a8260 con 0x7fdd301033c0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.737+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd20040070 con 0x7fdd301033c0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.737+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20045b50 con 0x7fdd301033c0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.737+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd301a87f0 con 0x7fdd301033c0 2024-07-26T08:37:53.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.738+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd301037d0 con 0x7fdd301033c0 2024-07-26T08:37:53.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.741+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fdd20045cf0 con 0x7fdd301033c0 2024-07-26T08:37:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.742+0000 7fdd27400640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdd140b3c20 0x7fdd140b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:53.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.742+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fdd20105460 con 0x7fdd301033c0 2024-07-26T08:37:53.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.743+0000 7fdd2e000640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdd140b3c20 0x7fdd140b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.743+0000 7fdd2e000640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdd140b3c20 0x7fdd140b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdd30108760 tx=0x7fdd18005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:53.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.743+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdd200cc5a0 con 0x7fdd301033c0 2024-07-26T08:37:53.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.993+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd30076440 con 0x7fdd301033c0 2024-07-26T08:37:53.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.994+0000 7fdd27400640 1 -- 172.21.15.44:0/4250238119 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7fdd200cbea0 con 0x7fdd301033c0 2024-07-26T08:37:53.996 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:53.996 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"btime":"2024-07-26T08:34:37:118441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:34:36.302968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24661,24668,24679,24686,24701]},"id":1}]} 2024-07-26T08:37:53.996 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdd140b3c20 msgr2=0x7fdd140b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdd140b3c20 0x7fdd140b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdd30108760 tx=0x7fdd18005f90 comp rx=0 tx=0).stop 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 msgr2=0x7fdd30075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 0x7fdd30075140 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdd20006010 tx=0x7fdd20030cf0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 shutdown_connections 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fdd140b3c20 0x7fdd140b6130 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdd30109650 0x7fdd30075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd30108d00 0x7fdd30075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 --2- 172.21.15.44:0/4250238119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd301033c0 0x7fdd30075140 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:53.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.997+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 >> 172.21.15.44:0/4250238119 conn(0x7fdd300fee10 msgr2=0x7fdd30101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:54.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.998+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 shutdown_connections 2024-07-26T08:37:54.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:53.998+0000 7fdd2fe00640 1 -- 172.21.15.44:0/4250238119 wait complete. 2024-07-26T08:37:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:54 smithi044 ceph-mon[32404]: pgmap v459: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:54 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4250238119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:54 smithi116 ceph-mon[37097]: pgmap v459: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:54 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4250238119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:54 smithi171 ceph-mon[37789]: pgmap v459: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:37:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:54 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4250238119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:54.944 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:37:55.405 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:37:56.695 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:56 smithi044 ceph-mon[32404]: pgmap v460: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:56 smithi116 ceph-mon[37097]: pgmap v460: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:56 smithi171 ceph-mon[37789]: pgmap v460: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:37:57.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 -- 172.21.15.44:0/2946464252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a810d840 msgr2=0x7f37a810fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:57.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 --2- 172.21.15.44:0/2946464252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a810d840 0x7f37a810fd30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009ca0 tx=0x7f37a002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 -- 172.21.15.44:0/2946464252 shutdown_connections 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 --2- 172.21.15.44:0/2946464252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a810d840 0x7f37a810fd30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 --2- 172.21.15.44:0/2946464252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a8104d30 0x7f37a810d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 --2- 172.21.15.44:0/2946464252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 0x7f37a81047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 -- 172.21.15.44:0/2946464252 >> 172.21.15.44:0/2946464252 conn(0x7f37a8079190 msgr2=0x7f37a80795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.609+0000 7f37aca00640 1 -- 172.21.15.44:0/2946464252 shutdown_connections 2024-07-26T08:37:57.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.610+0000 7f37aca00640 1 -- 172.21.15.44:0/2946464252 wait complete. 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.611+0000 7f37aca00640 1 Processor -- start 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.611+0000 7f37aca00640 1 -- start start 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37aca00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 0x7f37a819c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37aca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a8104d30 0x7f37a819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37aca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a810d840 0x7f37a81a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37aca00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81134b0 con 0x7f37a8104d30 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37aca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8113330 con 0x7f37a810d840 2024-07-26T08:37:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37aca00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8113630 con 0x7f37a81043e0 2024-07-26T08:37:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37a5e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 0x7f37a819c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37a6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a810d840 0x7f37a81a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37a5e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 0x7f37a819c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:45226/0 (socket says 172.21.15.44:45226) 2024-07-26T08:37:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f37a5e00640 1 -- 172.21.15.44:0/433344195 learned_addr learned my addr 172.21.15.44:0/433344195 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:37:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.612+0000 7f379fe00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a8104d30 0x7f37a819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.613+0000 7f37a6800640 1 -- 172.21.15.44:0/433344195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 msgr2=0x7f37a819c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.613+0000 7f37a6800640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 0x7f37a819c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.613+0000 7f37a6800640 1 -- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a8104d30 msgr2=0x7f37a819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.613+0000 7f37a6800640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a8104d30 0x7f37a819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.613+0000 7f37a6800640 1 -- 172.21.15.44:0/433344195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a81a20d0 con 0x7f37a810d840 2024-07-26T08:37:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.613+0000 7f37a6800640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a810d840 0x7f37a81a19a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f37a002ade0 tx=0x7f37a0031660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.614+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0003b70 con 0x7f37a810d840 2024-07-26T08:37:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.614+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a81a2360 con 0x7f37a810d840 2024-07-26T08:37:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.614+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a81a8b20 con 0x7f37a810d840 2024-07-26T08:37:57.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.614+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37a0031910 con 0x7f37a810d840 2024-07-26T08:37:57.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.614+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a002f8c0 con 0x7f37a810d840 2024-07-26T08:37:57.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.615+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3784005290 con 0x7f37a810d840 2024-07-26T08:37:57.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.618+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f37a0003d10 con 0x7f37a810d840 2024-07-26T08:37:57.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.619+0000 7f379ea00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f378c0b3ca0 0x7f378c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:37:57.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.619+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f37a0105420 con 0x7f37a810d840 2024-07-26T08:37:57.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.619+0000 7f37a5e00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f378c0b3ca0 0x7f378c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:37:57.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.620+0000 7f37a5e00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f378c0b3ca0 0x7f378c0b61b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3798009c70 tx=0x7f3798005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:37:57.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.623+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f37a00cc560 con 0x7f37a810d840 2024-07-26T08:37:57.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.864+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3784005580 con 0x7f37a810d840 2024-07-26T08:37:57.867 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.866+0000 7f379ea00640 1 -- 172.21.15.44:0/433344195 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f37a00cbe60 con 0x7f37a810d840 2024-07-26T08:37:57.867 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:37:57.868 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-07-26T08:37:57.868 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"btime":"2024-07-26T08:34:37:118441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:34:36.302968+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24661,24668,24679,24686,24701]},"id":1}]} 2024-07-26T08:37:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f378c0b3ca0 msgr2=0x7f378c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:57.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f378c0b3ca0 0x7f378c0b61b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3798009c70 tx=0x7f3798005f90 comp rx=0 tx=0).stop 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a810d840 msgr2=0x7f37a81a19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a810d840 0x7f37a81a19a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f37a002ade0 tx=0x7f37a0031660 comp rx=0 tx=0).stop 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 shutdown_connections 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f378c0b3ca0 0x7f378c0b61b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37a810d840 0x7f37a81a19a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37a8104d30 0x7f37a819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 --2- 172.21.15.44:0/433344195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37a81043e0 0x7f37a819c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:37:57.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.869+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 >> 172.21.15.44:0/433344195 conn(0x7f37a8079190 msgr2=0x7f37a80fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:37:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.870+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 shutdown_connections 2024-07-26T08:37:57.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:37:57.870+0000 7f37aca00640 1 -- 172.21.15.44:0/433344195 wait complete. 2024-07-26T08:37:58.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:58 smithi044 ceph-mon[32404]: pgmap v461: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:58.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:37:58 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/433344195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:58 smithi116 ceph-mon[37097]: pgmap v461: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:37:58 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/433344195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:58 smithi171 ceph-mon[37789]: pgmap v461: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:37:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:37:58 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/433344195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:37:58.842 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:37:59.312 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:00 smithi044 ceph-mon[32404]: pgmap v462: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:00 smithi116 ceph-mon[37097]: pgmap v462: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:00 smithi171 ceph-mon[37789]: pgmap v462: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:01.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 -- 172.21.15.44:0/3514356511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c1043e0 msgr2=0x7fda9c1047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3514356511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c1043e0 0x7fda9c1047f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009ca0 tx=0x7fda8c02a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 -- 172.21.15.44:0/3514356511 shutdown_connections 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3514356511 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c10d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3514356511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c10d840 0x7fda9c10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3514356511 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c1043e0 0x7fda9c1047f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 -- 172.21.15.44:0/3514356511 >> 172.21.15.44:0/3514356511 conn(0x7fda9c079160 msgr2=0x7fda9c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.698+0000 7fdaa3000640 1 -- 172.21.15.44:0/3514356511 shutdown_connections 2024-07-26T08:38:01.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.699+0000 7fdaa3000640 1 -- 172.21.15.44:0/3514356511 wait complete. 2024-07-26T08:38:01.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 Processor -- start 2024-07-26T08:38:01.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 -- start start 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c1043e0 0x7fda9c19c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c10d840 0x7fda9c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c1134d0 con 0x7fda9c10d840 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113350 con 0x7fda9c1043e0 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.700+0000 7fdaa3000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113650 con 0x7fda9c104d30 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fdaa1000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c10d840 0x7fda9c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c1043e0 0x7fda9c19c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46002/0 (socket says 172.21.15.44:46002) 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fdaa1000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c10d840 0x7fda9c1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51814/0 (socket says 172.21.15.44:51814) 2024-07-26T08:38:01.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c1043e0 0x7fda9c19c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:56886/0 (socket says 172.21.15.44:56886) 2024-07-26T08:38:01.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 -- 172.21.15.44:0/3836479717 learned_addr learned my addr 172.21.15.44:0/3836479717 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:01.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 -- 172.21.15.44:0/3836479717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c1043e0 msgr2=0x7fda9c19c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:01.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c1043e0 0x7fda9c19c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 -- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c10d840 msgr2=0x7fda9c1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:01.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c10d840 0x7fda9c1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.701+0000 7fda9b400640 1 -- 172.21.15.44:0/3836479717 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c1a2050 con 0x7fda9c104d30 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.702+0000 7fda9b400640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c19c680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fda8400e9c0 tx=0x7fda8400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.702+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8400ccc0 con 0x7fda9c104d30 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.702+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda840054d0 con 0x7fda9c104d30 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.702+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84010430 con 0x7fda9c104d30 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.702+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c1a2340 con 0x7fda9c104d30 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.702+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1a8aa0 con 0x7fda9c104d30 2024-07-26T08:38:01.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.703+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c1047f0 con 0x7fda9c104d30 2024-07-26T08:38:01.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.706+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fda84005000 con 0x7fda9c104d30 2024-07-26T08:38:01.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.708+0000 7fda98c00640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fda800b3c20 0x7fda800b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:01.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.708+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fda840e4680 con 0x7fda9c104d30 2024-07-26T08:38:01.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.708+0000 7fda9be00640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fda800b3c20 0x7fda800b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:01.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.709+0000 7fda9be00640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fda800b3c20 0x7fda800b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fda8c005e30 tx=0x7fda8c009360 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:01.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.709+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fda84055600 con 0x7fda9c104d30 2024-07-26T08:38:01.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.944+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda9c1a8d90 con 0x7fda9c104d30 2024-07-26T08:38:01.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.946+0000 7fda98c00640 1 -- 172.21.15.44:0/3836479717 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7fda840052f0 con 0x7fda9c104d30 2024-07-26T08:38:01.948 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:01.949 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:38:01.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.949+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fda800b3c20 msgr2=0x7fda800b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:01.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.949+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fda800b3c20 0x7fda800b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fda8c005e30 tx=0x7fda8c009360 comp rx=0 tx=0).stop 2024-07-26T08:38:01.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.949+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 msgr2=0x7fda9c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:01.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.949+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c19c680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fda8400e9c0 tx=0x7fda8400ee80 comp rx=0 tx=0).stop 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 shutdown_connections 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fda800b3c20 0x7fda800b6130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda9c104d30 0x7fda9c19c680 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda9c10d840 0x7fda9c1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 --2- 172.21.15.44:0/3836479717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda9c1043e0 0x7fda9c19c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 >> 172.21.15.44:0/3836479717 conn(0x7fda9c079160 msgr2=0x7fda9c109630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 shutdown_connections 2024-07-26T08:38:01.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:01.950+0000 7fdaa3000640 1 -- 172.21.15.44:0/3836479717 wait complete. 2024-07-26T08:38:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:02 smithi044 ceph-mon[32404]: pgmap v463: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:02.789 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:02 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3836479717' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:02.792 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:02 smithi116 ceph-mon[37097]: pgmap v463: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:02.793 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:02 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3836479717' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:02 smithi171 ceph-mon[37789]: pgmap v463: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:02 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3836479717' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:02.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T08:33:06.110522+0000', 'modified': '2024-07-26T08:34:36.302968+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24701, 'mds_1': 24686, 'mds_2': 24679, 'mds_3': 24668, 'mds_4': 24661}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24613': {'gid': 24613, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6833/4260884949', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6832', 'nonce': 4260884949}, {'type': 'v1', 'addr': '172.21.15.171:6833', 'nonce': 4260884949}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24620': {'gid': 24620, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6833/2097534847', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6832', 'nonce': 2097534847}, {'type': 'v1', 'addr': '172.21.15.116:6833', 'nonce': 2097534847}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24638': {'gid': 24638, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.44:6837/3938332919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6836', 'nonce': 3938332919}, {'type': 'v1', 'addr': '172.21.15.44:6837', 'nonce': 3938332919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24650': {'gid': 24650, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/1052813979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 1052813979}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 1052813979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24655': {'gid': 24655, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6835/372796843', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6834', 'nonce': 372796843}, {'type': 'v1', 'addr': '172.21.15.171:6835', 'nonce': 372796843}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24661': {'gid': 24661, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.171:6837/3462225872', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6836', 'nonce': 3462225872}, {'type': 'v1', 'addr': '172.21.15.171:6837', 'nonce': 3462225872}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24668': {'gid': 24668, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.116:6837/2715773427', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 2715773427}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2715773427}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24679': {'gid': 24679, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.171:6839/2065417578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6838', 'nonce': 2065417578}, {'type': 'v1', 'addr': '172.21.15.171:6839', 'nonce': 2065417578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24686': {'gid': 24686, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 30, 'addr': '172.21.15.44:6839/1144072288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.44:6838', 'nonce': 1144072288}, {'type': 'v1', 'addr': '172.21.15.44:6839', 'nonce': 1144072288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24701': {'gid': 24701, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6839/3153223603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 3153223603}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 3153223603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24701, 'qdb_cluster': [24661, 24668, 24679, 24686, 24701]} 2024-07-26T08:38:02.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T08:38:02.928 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.c status 2024-07-26T08:38:03.423 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:04 smithi044 ceph-mon[32404]: pgmap v464: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:04 smithi116 ceph-mon[37097]: pgmap v464: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:04 smithi171 ceph-mon[37789]: pgmap v464: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:38:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:38:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.793+0000 7fee6e800640 1 -- 172.21.15.44:0/227215074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68075800 msgr2=0x7fee68075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.793+0000 7fee6e800640 1 --2- 172.21.15.44:0/227215074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68075800 0x7fee68075c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fee54009860 tx=0x7fee5402a340 comp rx=0 tx=0).stop 2024-07-26T08:38:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 -- 172.21.15.44:0/227215074 shutdown_connections 2024-07-26T08:38:05.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 --2- 172.21.15.44:0/227215074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee68113c90 0x7fee68116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 --2- 172.21.15.44:0/227215074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68075800 0x7fee68075c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 --2- 172.21.15.44:0/227215074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68076db0 0x7fee680751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 -- 172.21.15.44:0/227215074 >> 172.21.15.44:0/227215074 conn(0x7fee680fee10 msgr2=0x7fee68101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 -- 172.21.15.44:0/227215074 shutdown_connections 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.794+0000 7fee6e800640 1 -- 172.21.15.44:0/227215074 wait complete. 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.795+0000 7fee6e800640 1 Processor -- start 2024-07-26T08:38:05.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.795+0000 7fee6e800640 1 -- start start 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68075800 0x7fee681a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 0x7fee681a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6e800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee68113c90 0x7fee681a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6e800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fee681198d0 con 0x7fee68075800 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee68119750 con 0x7fee68076db0 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6e800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fee68119a50 con 0x7fee68113c90 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 0x7fee681a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 0x7fee681a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:56916/0 (socket says 172.21.15.44:56916) 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6ca00640 1 -- 172.21.15.44:0/1377149256 learned_addr learned my addr 172.21.15.44:0/1377149256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:05.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.796+0000 7fee6d400640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68075800 0x7fee681a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6de00640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee68113c90 0x7fee681a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6ca00640 1 -- 172.21.15.44:0/1377149256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee68113c90 msgr2=0x7fee681a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6ca00640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee68113c90 0x7fee681a5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6ca00640 1 -- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68075800 msgr2=0x7fee681a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6ca00640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68075800 0x7fee681a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6ca00640 1 -- 172.21.15.44:0/1377149256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee681a6250 con 0x7fee68076db0 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6d400640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68075800 0x7fee681a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.797+0000 7fee6ca00640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 0x7fee681a0880 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee54035040 tx=0x7fee54032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.798+0000 7fee66000640 1 -- 172.21.15.44:0/1377149256 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54037040 con 0x7fee68076db0 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.798+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee681a64e0 con 0x7fee68076db0 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.798+0000 7fee66000640 1 -- 172.21.15.44:0/1377149256 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee54041070 con 0x7fee68076db0 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.798+0000 7fee66000640 1 -- 172.21.15.44:0/1377149256 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54003850 con 0x7fee68076db0 2024-07-26T08:38:05.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.798+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee681acca0 con 0x7fee68076db0 2024-07-26T08:38:05.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.801+0000 7fee76a61740 1 Processor -- start 2024-07-26T08:38:05.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 -- start start 2024-07-26T08:38:05.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b54282a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54282aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54282b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b5427a0950 con 0x55b542464280 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b54278fe50 con 0x55b54245c0a0 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee76a61740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b542793e90 con 0x55b542484f60 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee66000640 1 -- 172.21.15.44:0/1377149256 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fee540039f0 con 0x7fee68076db0 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee6de00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b54282a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee6de00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b54282a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46026/0 (socket says 172.21.15.44:46026) 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee6de00640 1 -- 172.21.15.44:0/8688946 learned_addr learned my addr 172.21.15.44:0/8688946 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:05.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.802+0000 7fee6d400640 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54282aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6ca00640 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54282b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6de00640 1 -- 172.21.15.44:0/8688946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 msgr2=0x55b54282aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6de00640 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54282aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6de00640 1 -- 172.21.15.44:0/8688946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 msgr2=0x55b54282b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6de00640 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54282b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6de00640 1 -- 172.21.15.44:0/8688946 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5427370f0 con 0x55b542484f60 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.803+0000 7fee6de00640 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b54282a760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fee60009fd0 tx=0x7fee600118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6076ccf02d37b891 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.804+0000 7fee5a600640 1 -- 172.21.15.44:0/8688946 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60004d20 con 0x55b542484f60 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.804+0000 7fee5a600640 1 -- 172.21.15.44:0/8688946 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee60004050 con 0x55b542484f60 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.804+0000 7fee5a600640 1 -- 172.21.15.44:0/8688946 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60005280 con 0x55b542484f60 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.804+0000 7fee66000640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee500b3c20 0x7fee500b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.804+0000 7fee66000640 1 -- 172.21.15.44:0/1377149256 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fee54103cb0 con 0x7fee68076db0 2024-07-26T08:38:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee6d400640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee500b3c20 0x7fee500b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 -- 172.21.15.44:0/8688946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 msgr2=0x55b54282a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b54282a760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fee60009fd0 tx=0x7fee600118d0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 -- 172.21.15.44:0/8688946 shutdown_connections 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54282b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54282aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 --2- 172.21.15.44:0/8688946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b54282a760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 -- 172.21.15.44:0/8688946 >> 172.21.15.44:0/8688946 conn(0x55b542479350 msgr2=0x55b542828650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 -- 172.21.15.44:0/8688946 shutdown_connections 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.805+0000 7fee76a61740 1 -- 172.21.15.44:0/8688946 wait complete. 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.806+0000 7fee6d400640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee500b3c20 0x7fee500b6130 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55b5428293e0 tx=0x7fee5c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.806+0000 7fee76a61740 1 -- start start 2024-07-26T08:38:05.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.806+0000 7fee76a61740 1 Processor -- start 2024-07-26T08:38:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b542836220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542464280 0x55b542836760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b542842230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b5425f35f0 con 0x55b542484f60 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b54272c520 con 0x55b54245c0a0 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b542838f40 con 0x55b542464280 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee76a61740 10 client.0 _ll_register_callbacks cb 0x55b54273bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee6de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b542842230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee6ca00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542464280 0x55b542836760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee6de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b542842230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51842/0 (socket says 172.21.15.44:51842) 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee6de00640 1 -- 172.21.15.44:0/2809409646 learned_addr learned my addr 172.21.15.44:0/2809409646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.807+0000 7fee6d400640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b542836220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6ca00640 1 -- 172.21.15.44:0/2809409646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 msgr2=0x55b542836220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6ca00640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b542836220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6ca00640 1 -- 172.21.15.44:0/2809409646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 msgr2=0x55b542842230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6ca00640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b542842230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6ca00640 1 -- 172.21.15.44:0/2809409646 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5424f0540 con 0x55b542464280 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6d400640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b542836220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee6ca00640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542464280 0x55b542836760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee54101350 tx=0x7fee54030630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1d2f04db7f3d215 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee4ea00640 1 -- 172.21.15.44:0/2809409646 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee540c9ec0 con 0x55b542464280 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee4ea00640 1 -- 172.21.15.44:0/2809409646 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fee540d3370 con 0x55b542464280 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.808+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee60004050 con 0x55b542464280 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee4ea00640 1 -- 172.21.15.44:0/2809409646 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee540e4050 con 0x55b542464280 2024-07-26T08:38:05.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 10 client.25300 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b542844e60 con 0x55b542464280 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee4ea00640 1 -- 172.21.15.44:0/2809409646 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee60004050 con 0x55b542464280 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee4ea00640 1 -- 172.21.15.44:0/2809409646 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fee540c60c0 con 0x55b542464280 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 10 client.25300 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 10 client.25300 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 20 client.25300 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b5424438e0 0x55b542848930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 4 client.25300 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:05.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.809+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5427a28a0 con 0x55b5424438e0 2024-07-26T08:38:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.810+0000 7fee6d400640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b5424438e0 0x55b542848930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.810+0000 7fee6d400640 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b5424438e0 0x55b542848930 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ad68160863831c3 server_cookie=95dde0a3dc1940ad in_seq=0 out_seq=0 2024-07-26T08:38:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.810+0000 7fee4ea00640 10 client.25300 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.811+0000 7fee4ea00640 1 -- 172.21.15.44:0/2809409646 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5427a28a0 con 0x55b5424438e0 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.811+0000 7fee4ea00640 10 client.25300 handle_command_reply: tid=0 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 client.25300 shutdown 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542464280 msgr2=0x55b542836760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542464280 0x55b542836760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee54101350 tx=0x7fee54030630 comp rx=0 tx=0).stop 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 shutdown_connections 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542464280 0x55b542836760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b5424438e0 0x55b542848930 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b542842230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 --2- 172.21.15.44:0/2809409646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b542836220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 >> 172.21.15.44:0/2809409646 conn(0x55b542479350 msgr2=0x55b54282d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee4ea00640 0 client.25300 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:38:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 shutdown_connections 2024-07-26T08:38:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 1 -- 172.21.15.44:0/2809409646 wait complete. 2024-07-26T08:38:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.812+0000 7fee76a61740 20 client.25300 trim_cache size 0 max 16384 2024-07-26T08:38:05.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.832+0000 7fee76a61740 1 Processor -- start 2024-07-26T08:38:05.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.832+0000 7fee76a61740 1 -- start start 2024-07-26T08:38:05.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee76a61740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 0x55b542848930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee76a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b542464280 0x55b542836200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee76a61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b5428421a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee76a61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b542846c70 con 0x55b542484f60 2024-07-26T08:38:05.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee76a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b542847340 con 0x55b542464280 2024-07-26T08:38:05.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee76a61740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b542847a10 con 0x55b54245c0a0 2024-07-26T08:38:05.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee6d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 0x55b542848930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee6de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b5428421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee6d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 0x55b542848930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46042/0 (socket says 172.21.15.44:46042) 2024-07-26T08:38:05.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee6de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b5428421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51854/0 (socket says 172.21.15.44:51854) 2024-07-26T08:38:05.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee6de00640 1 -- 172.21.15.44:0/3202448092 learned_addr learned my addr 172.21.15.44:0/3202448092 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:05.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.833+0000 7fee6ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b542464280 0x55b542836200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6ca00640 1 -- 172.21.15.44:0/3202448092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 msgr2=0x55b542848930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6ca00640 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 0x55b542848930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6ca00640 1 -- 172.21.15.44:0/3202448092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 msgr2=0x55b5428421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6ca00640 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b5428421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6ca00640 1 -- 172.21.15.44:0/3202448092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54282e7e0 con 0x55b542464280 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6d400640 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 0x55b542848930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:05.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6de00640 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b5428421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:38:05.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.834+0000 7fee6ca00640 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b542464280 0x55b542836200 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fee540c67d0 tx=0x7fee54005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c18017d1fb3aafad server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.835+0000 7fee59c00640 1 -- 172.21.15.44:0/3202448092 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee540c9ec0 con 0x55b542464280 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.835+0000 7fee59c00640 1 -- 172.21.15.44:0/3202448092 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee540328f0 con 0x55b542464280 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.835+0000 7fee59c00640 1 -- 172.21.15.44:0/3202448092 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee540c6610 con 0x55b542464280 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 -- 172.21.15.44:0/3202448092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b542464280 msgr2=0x55b542836200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b542464280 0x55b542836200 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fee540c67d0 tx=0x7fee54005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 -- 172.21.15.44:0/3202448092 shutdown_connections 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542484f60 0x55b5428421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b542464280 0x55b542836200 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 --2- 172.21.15.44:0/3202448092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b54245c0a0 0x55b542848930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 -- 172.21.15.44:0/3202448092 >> 172.21.15.44:0/3202448092 conn(0x55b542479350 msgr2=0x55b5428348a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 -- 172.21.15.44:0/3202448092 shutdown_connections 2024-07-26T08:38:05.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.836+0000 7fee76a61740 1 -- 172.21.15.44:0/3202448092 wait complete. 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.837+0000 7fee76a61740 1 -- start start 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.837+0000 7fee76a61740 1 Processor -- start 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54284bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54284c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b5428afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b5428487b0 con 0x55b542464280 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5428480e0 con 0x55b54245c0a0 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b542848e80 con 0x55b542484f60 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee76a61740 10 client.0 _ll_register_callbacks cb 0x7fee54009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee6ca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54284c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee6d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54284bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.838+0000 7fee6ca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54284c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51870/0 (socket says 172.21.15.44:51870) 2024-07-26T08:38:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54284bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:56944/0 (socket says 172.21.15.44:56944) 2024-07-26T08:38:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6ca00640 1 -- 172.21.15.44:0/73979914 learned_addr learned my addr 172.21.15.44:0/73979914 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6de00640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b5428afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 -- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 msgr2=0x55b5428afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b5428afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 -- 172.21.15.44:0/73979914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 msgr2=0x55b54284c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54284c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 -- 172.21.15.44:0/73979914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5428237d0 con 0x55b54245c0a0 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6ca00640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54284c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee6d400640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54284bdf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55b542849770 tx=0x7fee5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faff8479ceaa8cdd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.839+0000 7fee4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.840+0000 7fee4ea00640 1 -- 172.21.15.44:0/73979914 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c003d30 con 0x55b54245c0a0 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.840+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee540328f0 con 0x55b54245c0a0 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.840+0000 7fee4ea00640 1 -- 172.21.15.44:0/73979914 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fee5c033470 con 0x55b54245c0a0 2024-07-26T08:38:05.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.840+0000 7fee4ea00640 1 -- 172.21.15.44:0/73979914 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee5c03c050 con 0x55b54245c0a0 2024-07-26T08:38:05.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.840+0000 7fee76a61740 10 client.25307 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:05.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.840+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5428b0f60 con 0x55b54245c0a0 2024-07-26T08:38:05.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee4ea00640 1 -- 172.21.15.44:0/73979914 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee540328f0 con 0x55b54245c0a0 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee4ea00640 1 -- 172.21.15.44:0/73979914 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fee54003ce0 con 0x55b54245c0a0 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee76a61740 10 client.25307 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee76a61740 10 client.25307 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee76a61740 20 client.25307 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee76a61740 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b542418c90 0x55b5428b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee76a61740 4 client.25307 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.841+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "status"}) -- 0x55b54279e340 con 0x55b542418c90 2024-07-26T08:38:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.842+0000 7fee6ca00640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b542418c90 0x55b5428b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.842+0000 7fee6ca00640 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b542418c90 0x55b5428b4800 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fdc817e540152f4b server_cookie=f4fdee09c239ba54 in_seq=0 out_seq=0 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.842+0000 7fee4ea00640 10 client.25307 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.842+0000 7fee4ea00640 1 -- 172.21.15.44:0/73979914 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b54279e340 con 0x55b542418c90 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.842+0000 7fee4ea00640 10 client.25307 handle_command_reply: tid=0 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.842+0000 7fee76a61740 1 client.25307 shutdown 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 msgr2=0x55b54284bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54284bdf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55b542849770 tx=0x7fee5c0046c0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 shutdown_connections 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b542484f60 0x55b5428afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55b542418c90 0x55b5428b4800 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b542464280 0x55b54284c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 --2- 172.21.15.44:0/73979914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b54245c0a0 0x55b54284bdf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 >> 172.21.15.44:0/73979914 conn(0x55b542479350 msgr2=0x55b542834470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee4ea00640 0 client.25307 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:38:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 shutdown_connections 2024-07-26T08:38:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 1 -- 172.21.15.44:0/73979914 wait complete. 2024-07-26T08:38:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.843+0000 7fee76a61740 20 client.25307 trim_cache size 0 max 16384 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 4, 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24661, 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 214.758320916, 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:38:05.847 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 340.54795089499999 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee500b3c20 msgr2=0x7fee500b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee500b3c20 0x7fee500b6130 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55b5428293e0 tx=0x7fee5c009260 comp rx=0 tx=0).stop 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 msgr2=0x7fee681a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 0x7fee681a0880 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee54035040 tx=0x7fee54032d50 comp rx=0 tx=0).stop 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 shutdown_connections 2024-07-26T08:38:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fee500b3c20 0x7fee500b6130 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee68113c90 0x7fee681a5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee68076db0 0x7fee681a0880 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 --2- 172.21.15.44:0/1377149256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fee68075800 0x7fee681a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 >> 172.21.15.44:0/1377149256 conn(0x7fee680fee10 msgr2=0x7fee6810fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 shutdown_connections 2024-07-26T08:38:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:05.844+0000 7fee6e800640 1 -- 172.21.15.44:0/1377149256 wait complete. 2024-07-26T08:38:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:06 smithi044 ceph-mon[32404]: pgmap v465: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:06 smithi116 ceph-mon[37097]: pgmap v465: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:06.798 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.h status 2024-07-26T08:38:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:06 smithi171 ceph-mon[37789]: pgmap v465: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:07.289 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:08.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:08 smithi044 ceph-mon[32404]: pgmap v466: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:08 smithi116 ceph-mon[37097]: pgmap v466: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:08 smithi171 ceph-mon[37789]: pgmap v466: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 -- 172.21.15.44:0/1729103461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 msgr2=0x7f1650107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 --2- 172.21.15.44:0/1729103461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f1650107ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f163c009be0 tx=0x7f163c02a6c0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 -- 172.21.15.44:0/1729103461 shutdown_connections 2024-07-26T08:38:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 --2- 172.21.15.44:0/1729103461 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1650108530 0x7f165010ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 --2- 172.21.15.44:0/1729103461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f1650107ff0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 --2- 172.21.15.44:0/1729103461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f16501022a0 0x7f16501026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 -- 172.21.15.44:0/1729103461 >> 172.21.15.44:0/1729103461 conn(0x7f16500fdcf0 msgr2=0x7f1650100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 -- 172.21.15.44:0/1729103461 shutdown_connections 2024-07-26T08:38:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.592+0000 7f1654e00640 1 -- 172.21.15.44:0/1729103461 wait complete. 2024-07-26T08:38:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 Processor -- start 2024-07-26T08:38:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 -- start start 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16501022a0 0x7f165019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f165019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 0x7f16501a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1650113310 con 0x7f1650108530 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1650113190 con 0x7f1650107be0 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.594+0000 7f1654e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1650113490 con 0x7f16501022a0 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16501022a0 0x7f165019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 0x7f16501a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16501022a0 0x7f165019bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46066/0 (socket says 172.21.15.44:46066) 2024-07-26T08:38:09.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 0x7f16501a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51884/0 (socket says 172.21.15.44:51884) 2024-07-26T08:38:09.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f165019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164f400640 1 -- 172.21.15.44:0/550509314 learned_addr learned my addr 172.21.15.44:0/550509314 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:09.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164ea00640 1 -- 172.21.15.44:0/550509314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16501022a0 msgr2=0x7f165019bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164ea00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16501022a0 0x7f165019bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164ea00640 1 -- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 msgr2=0x7f16501a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164ea00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 0x7f16501a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164ea00640 1 -- 172.21.15.44:0/550509314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16501a1e70 con 0x7f1650107be0 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.595+0000 7f164fe00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 0x7f16501a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.596+0000 7f164ea00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f165019c4a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f163c034040 tx=0x7f163c003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:09.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.596+0000 7f1647e00640 1 -- 172.21.15.44:0/550509314 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c041020 con 0x7f1650107be0 2024-07-26T08:38:09.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.596+0000 7f1647e00640 1 -- 172.21.15.44:0/550509314 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f163c030970 con 0x7f1650107be0 2024-07-26T08:38:09.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.596+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16501a2100 con 0x7f1650107be0 2024-07-26T08:38:09.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.596+0000 7f1647e00640 1 -- 172.21.15.44:0/550509314 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c036040 con 0x7f1650107be0 2024-07-26T08:38:09.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.596+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16501a88c0 con 0x7f1650107be0 2024-07-26T08:38:09.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.600+0000 7f1647e00640 1 -- 172.21.15.44:0/550509314 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f163c044020 con 0x7f1650107be0 2024-07-26T08:38:09.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.600+0000 7f165d075740 1 Processor -- start 2024-07-26T08:38:09.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.600+0000 7f165d075740 1 -- start start 2024-07-26T08:38:09.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f165d075740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a49780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f165d075740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56125667b0a0 0x561256a49cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f165d075740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 0x561256a4a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f165d075740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5612569bf970 con 0x561256683280 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f165d075740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612569aee70 con 0x56125667b0a0 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f165d075740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5612569b2eb0 con 0x5612566a3f60 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f164ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 0x561256a4a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f164fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a49780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f164ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 0x561256a4a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51886/0 (socket says 172.21.15.44:51886) 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f164ea00640 1 -- 172.21.15.44:0/1223052548 learned_addr learned my addr 172.21.15.44:0/1223052548 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:09.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.601+0000 7f164f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56125667b0a0 0x561256a49cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164f400640 1 -- 172.21.15.44:0/1223052548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 msgr2=0x561256a49780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164f400640 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a49780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164f400640 1 -- 172.21.15.44:0/1223052548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 msgr2=0x561256a4a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164f400640 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 0x561256a4a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164f400640 1 -- 172.21.15.44:0/1223052548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561256956110 con 0x56125667b0a0 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164ea00640 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 0x561256a4a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f1647e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f16340b3c20 0x7f16340b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f164f400640 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56125667b0a0 0x561256a49cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1640009cc0 tx=0x7f1640011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f5a4ea08658e86b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.602+0000 7f1647e00640 1 -- 172.21.15.44:0/550509314 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f163c104d20 con 0x7f1650107be0 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.603+0000 7f1644c00640 1 -- 172.21.15.44:0/1223052548 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1640004230 con 0x56125667b0a0 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.603+0000 7f164fe00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f16340b3c20 0x7f16340b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.603+0000 7f1644c00640 1 -- 172.21.15.44:0/1223052548 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16400043d0 con 0x56125667b0a0 2024-07-26T08:38:09.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.603+0000 7f1644c00640 1 -- 172.21.15.44:0/1223052548 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1640004d40 con 0x56125667b0a0 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 -- 172.21.15.44:0/1223052548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56125667b0a0 msgr2=0x561256a49cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f164fe00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f16340b3c20 0x7f16340b6130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1648012640 tx=0x7f1648005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56125667b0a0 0x561256a49cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1640009cc0 tx=0x7f1640011810 comp rx=0 tx=0).stop 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 -- 172.21.15.44:0/1223052548 shutdown_connections 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561256683280 0x561256a4a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56125667b0a0 0x561256a49cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 --2- 172.21.15.44:0/1223052548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a49780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 -- 172.21.15.44:0/1223052548 >> 172.21.15.44:0/1223052548 conn(0x561256698350 msgr2=0x561256a47670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:09.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 -- 172.21.15.44:0/1223052548 shutdown_connections 2024-07-26T08:38:09.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.604+0000 7f165d075740 1 -- 172.21.15.44:0/1223052548 wait complete. 2024-07-26T08:38:09.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.605+0000 7f165d075740 1 -- start start 2024-07-26T08:38:09.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.605+0000 7f165d075740 1 Processor -- start 2024-07-26T08:38:09.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.605+0000 7f165d075740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a55230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.605+0000 7f165d075740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a55770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f165d075740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a61250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f165d075740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5612568125f0 con 0x56125667b0a0 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f165d075740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56125694b540 con 0x561256683280 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f165d075740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561256a57f60 con 0x5612566a3f60 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f165d075740 10 client.0 _ll_register_callbacks cb 0x56125695af10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a55230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a55230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51894/0 (socket says 172.21.15.44:51894) 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164f400640 1 -- 172.21.15.44:0/2226818395 learned_addr learned my addr 172.21.15.44:0/2226818395 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164ea00640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a55770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164fe00640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a61250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164ea00640 1 -- 172.21.15.44:0/2226818395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 msgr2=0x561256a61250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164ea00640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a61250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164ea00640 1 -- 172.21.15.44:0/2226818395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 msgr2=0x561256a55230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164ea00640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a55230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164ea00640 1 -- 172.21.15.44:0/2226818395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125670f540 con 0x561256683280 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.606+0000 7f164f400640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a55230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f164ea00640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a55770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f163c104fc0 tx=0x7f163c030cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=346a207175482e5f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f1639600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f1639600640 1 -- 172.21.15.44:0/2226818395 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c0cb530 con 0x561256683280 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16400043d0 con 0x561256683280 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f1639600640 1 -- 172.21.15.44:0/2226818395 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f163c0cb9c0 con 0x561256683280 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f1639600640 1 -- 172.21.15.44:0/2226818395 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f163c0caed0 con 0x561256683280 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f165d075740 10 client.25319 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.607+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561256a63e60 con 0x561256683280 2024-07-26T08:38:09.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f1639600640 1 -- 172.21.15.44:0/2226818395 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16400043d0 con 0x561256683280 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f1639600640 1 -- 172.21.15.44:0/2226818395 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f163c0d4380 con 0x561256683280 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f165d075740 10 client.25319 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f165d075740 10 client.25319 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f165d075740 20 client.25319 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f165d075740 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5612566628e0 0x561256a67950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f165d075740 4 client.25319 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:09.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612569c18c0 con 0x5612566628e0 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.608+0000 7f164f400640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5612566628e0 0x561256a67950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.609+0000 7f164f400640 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5612566628e0 0x561256a67950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e413d40a134024fa server_cookie=fd99a6946eabfab1 in_seq=0 out_seq=0 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.609+0000 7f1639600640 10 client.25319 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f1639600640 1 -- 172.21.15.44:0/2226818395 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5612569c18c0 con 0x5612566628e0 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f1639600640 10 client.25319 handle_command_reply: tid=0 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 client.25319 shutdown 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 msgr2=0x561256a55770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a55770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f163c104fc0 tx=0x7f163c030cb0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 shutdown_connections 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5612566628e0 0x561256a67950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5612566a3f60 0x561256a61250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a55770 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 --2- 172.21.15.44:0/2226818395 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a55230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f1639600640 0 client.25319 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:38:09.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 >> 172.21.15.44:0/2226818395 conn(0x561256698350 msgr2=0x561256a4c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:09.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.610+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 shutdown_connections 2024-07-26T08:38:09.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.611+0000 7f165d075740 1 -- 172.21.15.44:0/2226818395 wait complete. 2024-07-26T08:38:09.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.611+0000 7f165d075740 20 client.25319 trim_cache size 0 max 16384 2024-07-26T08:38:09.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 Processor -- start 2024-07-26T08:38:09.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 -- start start 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 0x561256a67280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a4f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612566a3f60 0x561256a651b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561256a65c90 con 0x5612566a3f60 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561256a66360 con 0x561256683280 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f165d075740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561256a66a30 con 0x56125667b0a0 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f164f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 0x561256a67280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f164f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 0x561256a67280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46084/0 (socket says 172.21.15.44:46084) 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f164f400640 1 -- 172.21.15.44:0/1196117902 learned_addr learned my addr 172.21.15.44:0/1196117902 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:09.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.631+0000 7f164fe00640 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612566a3f60 0x561256a651b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164ea00640 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a4f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164f400640 1 -- 172.21.15.44:0/1196117902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 msgr2=0x561256a4f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164f400640 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a4f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164f400640 1 -- 172.21.15.44:0/1196117902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612566a3f60 msgr2=0x561256a651b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164f400640 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612566a3f60 0x561256a651b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164f400640 1 -- 172.21.15.44:0/1196117902 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561256a45a10 con 0x56125667b0a0 2024-07-26T08:38:09.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.632+0000 7f164f400640 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 0x561256a67280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16400098b0 tx=0x7f16400047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee606c3cbf4605a3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:09.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.633+0000 7f163be00640 1 -- 172.21.15.44:0/1196117902 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164001e040 con 0x56125667b0a0 2024-07-26T08:38:09.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.633+0000 7f163be00640 1 -- 172.21.15.44:0/1196117902 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f164001f040 con 0x56125667b0a0 2024-07-26T08:38:09.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.633+0000 7f163be00640 1 -- 172.21.15.44:0/1196117902 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164002d510 con 0x56125667b0a0 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.633+0000 7f165d075740 1 -- 172.21.15.44:0/1196117902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 msgr2=0x561256a67280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.633+0000 7f165d075740 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 0x561256a67280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16400098b0 tx=0x7f16400047a0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 -- 172.21.15.44:0/1196117902 shutdown_connections 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612566a3f60 0x561256a651b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561256683280 0x561256a4f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 --2- 172.21.15.44:0/1196117902 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125667b0a0 0x561256a67280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 -- 172.21.15.44:0/1196117902 >> 172.21.15.44:0/1196117902 conn(0x561256698350 msgr2=0x561256a538b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 -- 172.21.15.44:0/1196117902 shutdown_connections 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 -- 172.21.15.44:0/1196117902 wait complete. 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 -- start start 2024-07-26T08:38:09.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.634+0000 7f165d075740 1 Processor -- start 2024-07-26T08:38:09.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561256683280 0x561256a68b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 0x561256a69090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561256a677d0 con 0x56125667b0a0 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561256a67100 con 0x5612566a3f60 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561256a67ea0 con 0x561256683280 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f165d075740 10 client.0 _ll_register_callbacks cb 0x7f163c102250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f164fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 0x561256a69090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f164f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f164fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 0x561256a69090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:57004/0 (socket says 172.21.15.44:57004) 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f164fe00640 1 -- 172.21.15.44:0/2604353357 learned_addr learned my addr 172.21.15.44:0/2604353357 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.635+0000 7f164ea00640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561256683280 0x561256a68b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164fe00640 1 -- 172.21.15.44:0/2604353357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561256683280 msgr2=0x561256a68b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164fe00640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561256683280 0x561256a68b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164fe00640 1 -- 172.21.15.44:0/2604353357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 msgr2=0x561256a701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:09.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164fe00640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164fe00640 1 -- 172.21.15.44:0/2604353357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561256a69700 con 0x5612566a3f60 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164ea00640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561256683280 0x561256a68b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164f400640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f164fe00640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 0x561256a69090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1648003bd0 tx=0x7f16480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f21b06174ff193c5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.636+0000 7f1639600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f1639600640 1 -- 172.21.15.44:0/2604353357 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164802e060 con 0x5612566a3f60 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f1639600640 1 -- 172.21.15.44:0/2604353357 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f164800fbe0 con 0x5612566a3f60 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f164001f040 con 0x5612566a3f60 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f1639600640 1 -- 172.21.15.44:0/2604353357 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1648035050 con 0x5612566a3f60 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f165d075740 10 client.25325 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:09.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561256acfa70 con 0x5612566a3f60 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.637+0000 7f1639600640 1 -- 172.21.15.44:0/2604353357 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f164001f040 con 0x5612566a3f60 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f1639600640 1 -- 172.21.15.44:0/2604353357 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f164801f400 con 0x5612566a3f60 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f165d075740 10 client.25325 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f165d075740 10 client.25325 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f165d075740 20 client.25325 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f165d075740 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x561256637c90 0x561256ad33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f165d075740 4 client.25325 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "status"}) -- 0x5612569bd360 con 0x561256637c90 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.638+0000 7f164f400640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x561256637c90 0x561256ad33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:09.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f164f400640 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x561256637c90 0x561256ad33b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4c570f549e0400d4 server_cookie=4ae2d59d38e269cb in_seq=0 out_seq=0 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f1639600640 10 client.25325 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f1639600640 1 -- 172.21.15.44:0/2604353357 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5612569bd360 con 0x561256637c90 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f1639600640 10 client.25325 handle_command_reply: tid=0 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 client.25325 shutdown 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 msgr2=0x561256a69090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 0x561256a69090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1648003bd0 tx=0x7f16480046c0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 shutdown_connections 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x561256637c90 0x561256ad33b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561256683280 0x561256a68b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612566a3f60 0x561256a69090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f1639600640 0 client.25325 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.639+0000 7f165d075740 1 --2- 172.21.15.44:0/2604353357 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56125667b0a0 0x561256a701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.640+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 >> 172.21.15.44:0/2604353357 conn(0x561256698350 msgr2=0x561256a53480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:09.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.640+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 shutdown_connections 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.640+0000 7f165d075740 1 -- 172.21.15.44:0/2604353357 wait complete. 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.640+0000 7f165d075740 20 client.25325 trim_cache size 0 max 16384 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 3, 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24668, 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:38:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:38:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 219.565978675, 2024-07-26T08:38:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:38:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:38:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:38:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 341.70285343699999 2024-07-26T08:38:09.645 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f16340b3c20 msgr2=0x7f16340b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f16340b3c20 0x7f16340b6130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1648012640 tx=0x7f1648005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 msgr2=0x7f165019c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f165019c4a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f163c034040 tx=0x7f163c003c20 comp rx=0 tx=0).stop 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 shutdown_connections 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f16340b3c20 0x7f16340b6130 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1650108530 0x7f16501a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1650107be0 0x7f165019c4a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 --2- 172.21.15.44:0/550509314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16501022a0 0x7f165019bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 >> 172.21.15.44:0/550509314 conn(0x7f16500fdcf0 msgr2=0x7f16501072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 shutdown_connections 2024-07-26T08:38:09.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:09.641+0000 7f1654e00640 1 -- 172.21.15.44:0/550509314 wait complete. 2024-07-26T08:38:10.657 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.i status 2024-07-26T08:38:10.738 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:10 smithi044 ceph-mon[32404]: pgmap v467: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:10 smithi116 ceph-mon[37097]: pgmap v467: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:10 smithi171 ceph-mon[37789]: pgmap v467: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:11.156 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:12 smithi044 ceph-mon[32404]: pgmap v468: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:12 smithi116 ceph-mon[37097]: pgmap v468: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:12 smithi171 ceph-mon[37789]: pgmap v468: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 -- 172.21.15.44:0/271377697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2801099f0 msgr2=0x7ff280109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 --2- 172.21.15.44:0/271377697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2801099f0 0x7ff280109e00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff270009900 tx=0x7ff27002a400 comp rx=0 tx=0).stop 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 -- 172.21.15.44:0/271377697 shutdown_connections 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 --2- 172.21.15.44:0/271377697 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff28010fc80 0x7ff2801164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 --2- 172.21.15.44:0/271377697 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff28010f330 0x7ff28010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 --2- 172.21.15.44:0/271377697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2801099f0 0x7ff280109e00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 -- 172.21.15.44:0/271377697 >> 172.21.15.44:0/271377697 conn(0x7ff280078ff0 msgr2=0x7ff280107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 -- 172.21.15.44:0/271377697 shutdown_connections 2024-07-26T08:38:13.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.493+0000 7ff285200640 1 -- 172.21.15.44:0/271377697 wait complete. 2024-07-26T08:38:13.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.495+0000 7ff285200640 1 Processor -- start 2024-07-26T08:38:13.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.495+0000 7ff285200640 1 -- start start 2024-07-26T08:38:13.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.495+0000 7ff285200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2801099f0 0x7ff2801a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.495+0000 7ff285200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff28010f330 0x7ff2801a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.495+0000 7ff285200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28010fc80 0x7ff2801a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff285200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff280119940 con 0x7ff28010f330 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff285200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2801197c0 con 0x7ff28010fc80 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff285200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff280119ac0 con 0x7ff2801099f0 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2801099f0 0x7ff2801a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff28010f330 0x7ff2801a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28010fc80 0x7ff2801a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2801099f0 0x7ff2801a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52088/0 (socket says 172.21.15.44:52088) 2024-07-26T08:38:13.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27f400640 1 -- 172.21.15.44:0/3006916253 learned_addr learned my addr 172.21.15.44:0/3006916253 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:13.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27fe00640 1 -- 172.21.15.44:0/3006916253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2801099f0 msgr2=0x7ff2801a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27fe00640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2801099f0 0x7ff2801a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27fe00640 1 -- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff28010f330 msgr2=0x7ff2801a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27fe00640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff28010f330 0x7ff2801a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.496+0000 7ff27fe00640 1 -- 172.21.15.44:0/3006916253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2801a62a0 con 0x7ff28010fc80 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.497+0000 7ff27fe00640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28010fc80 0x7ff2801a5b70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff27800ec50 tx=0x7ff27800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.497+0000 7ff277e00640 1 -- 172.21.15.44:0/3006916253 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278003b70 con 0x7ff28010fc80 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.497+0000 7ff277e00640 1 -- 172.21.15.44:0/3006916253 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff278003d10 con 0x7ff28010fc80 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.497+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2801a6590 con 0x7ff28010fc80 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.497+0000 7ff277e00640 1 -- 172.21.15.44:0/3006916253 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278010830 con 0x7ff28010fc80 2024-07-26T08:38:13.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.498+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2801accf0 con 0x7ff28010fc80 2024-07-26T08:38:13.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.501+0000 7ff28d4ad740 1 Processor -- start 2024-07-26T08:38:13.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.501+0000 7ff277e00640 1 -- 172.21.15.44:0/3006916253 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7ff278003ec0 con 0x7ff28010fc80 2024-07-26T08:38:13.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.501+0000 7ff28d4ad740 1 -- start start 2024-07-26T08:38:13.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee28a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee28aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee28b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564dee200950 con 0x564dedee4f60 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564dee1efe50 con 0x564dedebc0a0 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564dee1f3e90 con 0x564dedec4280 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee28a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff27f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee28aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff27f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee28aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47540/0 (socket says 172.21.15.44:47540) 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee28a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52562/0 (socket says 172.21.15.44:52562) 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff27f400640 1 -- 172.21.15.44:0/2396309779 learned_addr learned my addr 172.21.15.44:0/2396309779 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.502+0000 7ff27ea00640 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee28b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27f400640 1 -- 172.21.15.44:0/2396309779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 msgr2=0x564dee28b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27f400640 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee28b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27f400640 1 -- 172.21.15.44:0/2396309779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 msgr2=0x564dee28a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27f400640 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee28a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27f400640 1 -- 172.21.15.44:0/2396309779 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dee1970f0 con 0x564dedebc0a0 2024-07-26T08:38:13.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27ea00640 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee28b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:13.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff27f400640 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee28aca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff270000c00 tx=0x7ff270031b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bbf4101a38d912a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:13.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.503+0000 7ff277e00640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff2640b3c20 0x7ff2640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.504+0000 7ff274c00640 1 -- 172.21.15.44:0/2396309779 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff270040070 con 0x564dedebc0a0 2024-07-26T08:38:13.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.504+0000 7ff274c00640 1 -- 172.21.15.44:0/2396309779 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff270004030 con 0x564dedebc0a0 2024-07-26T08:38:13.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.504+0000 7ff274c00640 1 -- 172.21.15.44:0/2396309779 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff270035040 con 0x564dedebc0a0 2024-07-26T08:38:13.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.504+0000 7ff277e00640 1 -- 172.21.15.44:0/3006916253 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff2780e4650 con 0x7ff28010fc80 2024-07-26T08:38:13.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.504+0000 7ff27ea00640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff2640b3c20 0x7ff2640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2396309779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 msgr2=0x564dee28aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff27ea00640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff2640b3c20 0x7ff2640b6130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff26800b6c0 tx=0x7ff268014040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee28aca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff270000c00 tx=0x7ff270031b90 comp rx=0 tx=0).stop 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2396309779 shutdown_connections 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee28b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee28aca0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2396309779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee28a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2396309779 >> 172.21.15.44:0/2396309779 conn(0x564deded9350 msgr2=0x564dee288650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2396309779 shutdown_connections 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.505+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2396309779 wait complete. 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 -- start start 2024-07-26T08:38:13.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 Processor -- start 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee296280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedec4280 0x564dee2967c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 0x564dee2a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564dee0535f0 con 0x564dedec4280 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564dee18c520 con 0x564dedebc0a0 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.506+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564dee298ee0 con 0x564dedee4f60 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff28d4ad740 10 client.0 _ll_register_callbacks cb 0x564dee19bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedec4280 0x564dee2967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 0x564dee2a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee296280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 0x564dee2a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52112/0 (socket says 172.21.15.44:52112) 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedec4280 0x564dee2967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52574/0 (socket says 172.21.15.44:52574) 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 -- 172.21.15.44:0/2328625285 learned_addr learned my addr 172.21.15.44:0/2328625285 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 -- 172.21.15.44:0/2328625285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 msgr2=0x564dee296280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee296280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 -- 172.21.15.44:0/2328625285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedec4280 msgr2=0x564dee2967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedec4280 0x564dee2967c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27fe00640 1 -- 172.21.15.44:0/2328625285 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dedf50540 con 0x564dedee4f60 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.507+0000 7ff27f400640 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee296280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff27fe00640 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 0x564dee2a21d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff27800efd0 tx=0x7ff278009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1931d8534a1b9e22 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff26d600640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff26d600640 1 -- 172.21.15.44:0/2328625285 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2780e4070 con 0x564dedee4f60 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff26d600640 1 -- 172.21.15.44:0/2328625285 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff2780a7680 con 0x564dedee4f60 2024-07-26T08:38:13.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff270004030 con 0x564dedee4f60 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff26d600640 1 -- 172.21.15.44:0/2328625285 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2780e4420 con 0x564dedee4f60 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff28d4ad740 10 client.25336 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.508+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564dee2a4e00 con 0x564dedee4f60 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff26d600640 1 -- 172.21.15.44:0/2328625285 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff270004030 con 0x564dedee4f60 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff26d600640 1 -- 172.21.15.44:0/2328625285 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff2780cc170 con 0x564dedee4f60 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff28d4ad740 10 client.25336 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff28d4ad740 10 client.25336 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff28d4ad740 20 client.25336 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dedea38e0 0x564dee2a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff28d4ad740 4 client.25336 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.509+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564dee2028a0 con 0x564dedea38e0 2024-07-26T08:38:13.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.510+0000 7ff27f400640 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dedea38e0 0x564dee2a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.510+0000 7ff27f400640 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dedea38e0 0x564dee2a88b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f1ee619f6ae0a0e7 server_cookie=830d5a99373acd32 in_seq=0 out_seq=0 2024-07-26T08:38:13.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.510+0000 7ff26d600640 10 client.25336 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.511+0000 7ff26d600640 1 -- 172.21.15.44:0/2328625285 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564dee2028a0 con 0x564dedea38e0 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.511+0000 7ff26d600640 10 client.25336 handle_command_reply: tid=0 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.511+0000 7ff28d4ad740 1 client.25336 shutdown 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 msgr2=0x564dee2a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 0x564dee2a21d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff27800efd0 tx=0x7ff278009820 comp rx=0 tx=0).stop 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 shutdown_connections 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dedea38e0 0x564dee2a88b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedee4f60 0x564dee2a21d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedec4280 0x564dee2967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/2328625285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedebc0a0 0x564dee296280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff26d600640 0 client.25336 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 >> 172.21.15.44:0/2328625285 conn(0x564deded9350 msgr2=0x564dee28d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 shutdown_connections 2024-07-26T08:38:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 1 -- 172.21.15.44:0/2328625285 wait complete. 2024-07-26T08:38:13.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.512+0000 7ff28d4ad740 20 client.25336 trim_cache size 0 max 16384 2024-07-26T08:38:13.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.532+0000 7ff28d4ad740 1 Processor -- start 2024-07-26T08:38:13.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.532+0000 7ff28d4ad740 1 -- start start 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 0x564dee2a7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee296260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedee4f60 0x564dee2a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564dee2a6bf0 con 0x564dedebc0a0 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564dee2a72c0 con 0x564dedee4f60 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564dee2a7990 con 0x564dedec4280 2024-07-26T08:38:13.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff27f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 0x564dee2a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff27f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 0x564dee2a7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52576/0 (socket says 172.21.15.44:52576) 2024-07-26T08:38:13.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff27f400640 1 -- 172.21.15.44:0/3996406501 learned_addr learned my addr 172.21.15.44:0/3996406501 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff27ea00640 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee296260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.533+0000 7ff27fe00640 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedee4f60 0x564dee2a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27f400640 1 -- 172.21.15.44:0/3996406501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 msgr2=0x564dee296260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27f400640 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee296260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27f400640 1 -- 172.21.15.44:0/3996406501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedee4f60 msgr2=0x564dee2a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27f400640 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedee4f60 0x564dee2a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27f400640 1 -- 172.21.15.44:0/3996406501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dee2837d0 con 0x564dedebc0a0 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27fe00640 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedee4f60 0x564dee2a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff27f400640 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 0x564dee2a7440 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff270009a20 tx=0x7ff270005160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39e4f4a8e59a5e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff26fe00640 1 -- 172.21.15.44:0/3996406501 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff270040070 con 0x564dedebc0a0 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.534+0000 7ff26fe00640 1 -- 172.21.15.44:0/3996406501 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff270005600 con 0x564dedebc0a0 2024-07-26T08:38:13.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff26fe00640 1 -- 172.21.15.44:0/3996406501 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff270035040 con 0x564dedebc0a0 2024-07-26T08:38:13.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3996406501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 msgr2=0x564dee2a7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 0x564dee2a7440 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff270009a20 tx=0x7ff270005160 comp rx=0 tx=0).stop 2024-07-26T08:38:13.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3996406501 shutdown_connections 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedec4280 0x564dee296260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedee4f60 0x564dee2a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3996406501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedebc0a0 0x564dee2a7440 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.535+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3996406501 >> 172.21.15.44:0/3996406501 conn(0x564deded9350 msgr2=0x564dee294900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.536+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3996406501 shutdown_connections 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.536+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3996406501 wait complete. 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.536+0000 7ff28d4ad740 1 -- start start 2024-07-26T08:38:13.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.536+0000 7ff28d4ad740 1 Processor -- start 2024-07-26T08:38:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 0x564dee2b1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedec4280 0x564dee2a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee2a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564dee2a8730 con 0x564dedee4f60 2024-07-26T08:38:13.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564dee2a8060 con 0x564dedec4280 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564dee2a8e00 con 0x564dedebc0a0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff28d4ad740 10 client.0 _ll_register_callbacks cb 0x7ff2780aa9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff27fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee2a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff27f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 0x564dee2b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedec4280 0x564dee2a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff27f400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 0x564dee2b1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52124/0 (socket says 172.21.15.44:52124) 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.537+0000 7ff27f400640 1 -- 172.21.15.44:0/3413484729 learned_addr learned my addr 172.21.15.44:0/3413484729 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff27f400640 1 -- 172.21.15.44:0/3413484729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedec4280 msgr2=0x564dee2a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff27f400640 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedec4280 0x564dee2a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff27f400640 1 -- 172.21.15.44:0/3413484729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 msgr2=0x564dee2a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff27f400640 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee2a9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff27f400640 1 -- 172.21.15.44:0/3413484729 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dee2aa470 con 0x564dedebc0a0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff27f400640 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 0x564dee2b1050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff27002ab80 tx=0x7ff270031d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69ed319afef5a50 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.538+0000 7ff26d600640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff26d600640 1 -- 172.21.15.44:0/3413484729 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27005e060 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff26d600640 1 -- 172.21.15.44:0/3413484729 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff2700316a0 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff270005600 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff26d600640 1 -- 172.21.15.44:0/3413484729 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff270065050 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff28d4ad740 10 client.25342 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564dee310850 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.539+0000 7ff26d600640 1 -- 172.21.15.44:0/3413484729 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff270005600 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff26d600640 1 -- 172.21.15.44:0/3413484729 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff2700594e0 con 0x564dedebc0a0 2024-07-26T08:38:13.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff28d4ad740 10 client.25342 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:13.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff28d4ad740 10 client.25342 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T08:38:13.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff28d4ad740 20 client.25342 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:13.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dede78c90 0x564dee313a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:13.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff28d4ad740 4 client.25342 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:38:13.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "status"}) -- 0x564dee1fe340 con 0x564dede78c90 2024-07-26T08:38:13.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.540+0000 7ff27ea00640 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dede78c90 0x564dee313a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.541+0000 7ff27ea00640 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dede78c90 0x564dee313a10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=165bed788117a6eb server_cookie=8854621716987708 in_seq=0 out_seq=0 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.541+0000 7ff26d600640 10 client.25342 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff26d600640 1 -- 172.21.15.44:0/3413484729 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564dee1fe340 con 0x564dede78c90 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff26d600640 10 client.25342 handle_command_reply: tid=0 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 client.25342 shutdown 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 msgr2=0x564dee2b1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 0x564dee2b1050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff27002ab80 tx=0x7ff270031d70 comp rx=0 tx=0).stop 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 shutdown_connections 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x564dede78c90 0x564dee313a10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564dedee4f60 0x564dee2a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dedec4280 0x564dee2a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 --2- 172.21.15.44:0/3413484729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564dedebc0a0 0x564dee2b1050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 >> 172.21.15.44:0/3413484729 conn(0x564deded9350 msgr2=0x564dee287e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff26d600640 0 client.25342 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 shutdown_connections 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.542+0000 7ff28d4ad740 1 -- 172.21.15.44:0/3413484729 wait complete. 2024-07-26T08:38:13.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff28d4ad740 20 client.25342 trim_cache size 0 max 16384 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 2, 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24679, 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 224.48661467700001, 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:38:13.547 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 342.85289989500001 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff2640b3c20 msgr2=0x7ff2640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff2640b3c20 0x7ff2640b6130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff26800b6c0 tx=0x7ff268014040 comp rx=0 tx=0).stop 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28010fc80 msgr2=0x7ff2801a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28010fc80 0x7ff2801a5b70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff27800ec50 tx=0x7ff27800c6a0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 shutdown_connections 2024-07-26T08:38:13.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff2640b3c20 0x7ff2640b6130 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28010fc80 0x7ff2801a5b70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff28010f330 0x7ff2801a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 --2- 172.21.15.44:0/3006916253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2801099f0 0x7ff2801a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:13.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.543+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 >> 172.21.15.44:0/3006916253 conn(0x7ff280078ff0 msgr2=0x7ff2800797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:13.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.544+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 shutdown_connections 2024-07-26T08:38:13.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:13.544+0000 7ff285200640 1 -- 172.21.15.44:0/3006916253 wait complete. 2024-07-26T08:38:14.391 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.g status 2024-07-26T08:38:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:14 smithi044 ceph-mon[32404]: pgmap v469: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:14 smithi116 ceph-mon[37097]: pgmap v469: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:14 smithi171 ceph-mon[37789]: pgmap v469: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:14.876 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:16.740 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:16 smithi044 ceph-mon[32404]: pgmap v470: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:16 smithi116 ceph-mon[37097]: pgmap v470: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:16 smithi171 ceph-mon[37789]: pgmap v470: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 -- 172.21.15.44:0/284593051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 msgr2=0x7fd35010fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 --2- 172.21.15.44:0/284593051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 0x7fd35010fd30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd348009900 tx=0x7fd34802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 -- 172.21.15.44:0/284593051 shutdown_connections 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 --2- 172.21.15.44:0/284593051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 0x7fd35010fd30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 --2- 172.21.15.44:0/284593051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd350104ce0 0x7fd35010d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 --2- 172.21.15.44:0/284593051 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd3501042b0 0x7fd3501046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 -- 172.21.15.44:0/284593051 >> 172.21.15.44:0/284593051 conn(0x7fd350079160 msgr2=0x7fd350079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 -- 172.21.15.44:0/284593051 shutdown_connections 2024-07-26T08:38:17.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.306+0000 7fd355200640 1 -- 172.21.15.44:0/284593051 wait complete. 2024-07-26T08:38:17.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.307+0000 7fd355200640 1 Processor -- start 2024-07-26T08:38:17.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.307+0000 7fd355200640 1 -- start start 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd355200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 0x7fd35019c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd355200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd350104ce0 0x7fd35019c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd355200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 0x7fd3501a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd355200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501134d0 con 0x7fd35010d860 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd355200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113350 con 0x7fd3501042b0 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd355200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd350113650 con 0x7fd350104ce0 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd34de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 0x7fd35019c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd34de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 0x7fd35019c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47610/0 (socket says 172.21.15.44:47610) 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd34de00640 1 -- 172.21.15.44:0/514788771 learned_addr learned my addr 172.21.15.44:0/514788771 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:17.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd34d400640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd350104ce0 0x7fd35019c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.308+0000 7fd34e800640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 0x7fd3501a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.309+0000 7fd34de00640 1 -- 172.21.15.44:0/514788771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd350104ce0 msgr2=0x7fd35019c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.309+0000 7fd34de00640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd350104ce0 0x7fd35019c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.309+0000 7fd34de00640 1 -- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 msgr2=0x7fd3501a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.309+0000 7fd34de00640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 0x7fd3501a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.309+0000 7fd34de00640 1 -- 172.21.15.44:0/514788771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3501a1fd0 con 0x7fd3501042b0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.309+0000 7fd34de00640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 0x7fd35019c060 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd33800e9c0 tx=0x7fd33800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.310+0000 7fd346a00640 1 -- 172.21.15.44:0/514788771 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33800cde0 con 0x7fd3501042b0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.310+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3501a22c0 con 0x7fd3501042b0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.310+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501a8a20 con 0x7fd3501042b0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.310+0000 7fd346a00640 1 -- 172.21.15.44:0/514788771 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3380108e0 con 0x7fd3501042b0 2024-07-26T08:38:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.310+0000 7fd346a00640 1 -- 172.21.15.44:0/514788771 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380188a0 con 0x7fd3501042b0 2024-07-26T08:38:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.313+0000 7fd35bdc3740 1 Processor -- start 2024-07-26T08:38:17.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.313+0000 7fd35bdc3740 1 -- start start 2024-07-26T08:38:17.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5eb3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5ae50a0 0x5617b5eb3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5eb3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5617b5e29950 con 0x5617b5b0df60 2024-07-26T08:38:17.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617b5e18e50 con 0x5617b5aed280 2024-07-26T08:38:17.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5617b5e1ce90 con 0x5617b5ae50a0 2024-07-26T08:38:17.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd34e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5eb3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd34e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5eb3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52612/0 (socket says 172.21.15.44:52612) 2024-07-26T08:38:17.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd34e800640 1 -- 172.21.15.44:0/3153996152 learned_addr learned my addr 172.21.15.44:0/3153996152 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:17.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.314+0000 7fd34d400640 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5eb3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34e800640 1 -- 172.21.15.44:0/3153996152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5ae50a0 msgr2=0x5617b5eb3a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:38:17.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34e800640 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5ae50a0 0x5617b5eb3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34e800640 1 -- 172.21.15.44:0/3153996152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 msgr2=0x5617b5eb3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34e800640 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5eb3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34e800640 1 -- 172.21.15.44:0/3153996152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b5dc00f0 con 0x5617b5b0df60 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34d400640 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5eb3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd346a00640 1 -- 172.21.15.44:0/514788771 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fd33801f020 con 0x7fd3501042b0 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.315+0000 7fd34e800640 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5eb3550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd348009900 tx=0x7fd348005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcf30340391eea1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.316+0000 7fd33f400640 1 -- 172.21.15.44:0/3153996152 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348040070 con 0x5617b5b0df60 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.316+0000 7fd33f400640 1 -- 172.21.15.44:0/3153996152 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3480044c0 con 0x5617b5b0df60 2024-07-26T08:38:17.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.316+0000 7fd33f400640 1 -- 172.21.15.44:0/3153996152 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348035040 con 0x5617b5b0df60 2024-07-26T08:38:17.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3153996152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 msgr2=0x5617b5eb3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5eb3550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd348009900 tx=0x7fd348005f30 comp rx=0 tx=0).stop 2024-07-26T08:38:17.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd346a00640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd3340b3ca0 0x7fd3340b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3153996152 shutdown_connections 2024-07-26T08:38:17.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5ae50a0 0x5617b5eb3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd346a00640 1 -- 172.21.15.44:0/514788771 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fd3380e4220 con 0x7fd3501042b0 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5eb3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3153996152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5eb3550 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3153996152 >> 172.21.15.44:0/3153996152 conn(0x5617b5b02350 msgr2=0x5617b5eb1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd34e800640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd3340b3ca0 0x7fd3340b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3153996152 shutdown_connections 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.317+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3153996152 wait complete. 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.318+0000 7fd34e800640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd3340b3ca0 0x7fd3340b61b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3480096c0 tx=0x7fd34802a3f0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:17.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.318+0000 7fd35bdc3740 1 -- start start 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.318+0000 7fd35bdc3740 1 Processor -- start 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 0x5617b5ebf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5ebf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5617b5c7c5f0 con 0x5617b5ae50a0 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617b5db5520 con 0x5617b5aed280 2024-07-26T08:38:17.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ec1cd0 con 0x5617b5b0df60 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd35bdc3740 10 client.0 _ll_register_callbacks cb 0x5617b5dc4ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd34de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 0x5617b5ebf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd34de00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 0x5617b5ebf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52626/0 (socket says 172.21.15.44:52626) 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd34de00640 1 -- 172.21.15.44:0/2839395257 learned_addr learned my addr 172.21.15.44:0/2839395257 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.319+0000 7fd34d400640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5ebf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34e800640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34de00640 1 -- 172.21.15.44:0/2839395257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 msgr2=0x5617b5ecaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34de00640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34de00640 1 -- 172.21.15.44:0/2839395257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 msgr2=0x5617b5ebf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34de00640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5ebf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34de00640 1 -- 172.21.15.44:0/2839395257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b5b79540 con 0x5617b5ae50a0 2024-07-26T08:38:17.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34d400640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5ebf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd34de00640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 0x5617b5ebf080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd3380aac50 tx=0x7fd3380ab480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24541bfd8acb37e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.320+0000 7fd333e00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd333e00640 1 -- 172.21.15.44:0/2839395257 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380b3d70 con 0x5617b5ae50a0 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd333e00640 1 -- 172.21.15.44:0/2839395257 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fd338010a90 con 0x5617b5ae50a0 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5617b5e2b8a0 con 0x5617b5ae50a0 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd333e00640 1 -- 172.21.15.44:0/2839395257 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd3380b38e0 con 0x5617b5ae50a0 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd35bdc3740 10 client.15228 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:17.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617b5ecdc70 con 0x5617b5ae50a0 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd333e00640 1 -- 172.21.15.44:0/2839395257 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5617b5e2b8a0 con 0x5617b5ae50a0 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.321+0000 7fd333e00640 1 -- 172.21.15.44:0/2839395257 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd3380aa0b0 con 0x5617b5ae50a0 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd35bdc3740 10 client.15228 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd35bdc3740 10 client.15228 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd35bdc3740 20 client.15228 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5acc8e0 0x5617b5ed1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd35bdc3740 4 client.15228 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617b5e27340 con 0x5617b5acc8e0 2024-07-26T08:38:17.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.322+0000 7fd34d400640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5acc8e0 0x5617b5ed1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.323+0000 7fd34d400640 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5acc8e0 0x5617b5ed1680 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a1a38e436b366c server_cookie=ed287446c1357547 in_seq=0 out_seq=0 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.323+0000 7fd333e00640 10 client.15228 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.323+0000 7fd333e00640 1 -- 172.21.15.44:0/2839395257 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5617b5e27340 con 0x5617b5acc8e0 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd333e00640 10 client.15228 handle_command_reply: tid=0 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 client.15228 shutdown 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 msgr2=0x5617b5ebf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 0x5617b5ebf080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd3380aac50 tx=0x7fd3380ab480 comp rx=0 tx=0).stop 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 shutdown_connections 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5acc8e0 0x5617b5ed1680 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5aed280 0x5617b5ebf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/2839395257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5ae50a0 0x5617b5ebf080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 >> 172.21.15.44:0/2839395257 conn(0x5617b5b02350 msgr2=0x5617b5eb6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:17.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd333e00640 0 client.15228 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:38:17.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 shutdown_connections 2024-07-26T08:38:17.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 1 -- 172.21.15.44:0/2839395257 wait complete. 2024-07-26T08:38:17.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.324+0000 7fd35bdc3740 20 client.15228 trim_cache size 0 max 16384 2024-07-26T08:38:17.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.344+0000 7fd35bdc3740 1 Processor -- start 2024-07-26T08:38:17.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 -- start start 2024-07-26T08:38:17.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 0x5617b5eba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ecf9c0 con 0x5617b5aed280 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ed0090 con 0x5617b5ae50a0 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ed0760 con 0x5617b5b0df60 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd34d400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 0x5617b5eba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd34e800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.345+0000 7fd34d400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 0x5617b5eba030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52642/0 (socket says 172.21.15.44:52642) 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34d400640 1 -- 172.21.15.44:0/3683572090 learned_addr learned my addr 172.21.15.44:0/3683572090 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:17.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34e800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52156/0 (socket says 172.21.15.44:52156) 2024-07-26T08:38:17.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 -- 172.21.15.44:0/3683572090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 msgr2=0x5617b5ecaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 -- 172.21.15.44:0/3683572090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 msgr2=0x5617b5eba030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 0x5617b5eba030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 -- 172.21.15.44:0/3683572090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b5eb5840 con 0x5617b5ae50a0 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34d400640 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 0x5617b5eba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.346+0000 7fd34de00640 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed0210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd3380e1be0 tx=0x7fd3380095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3263dd65ab29a6d3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.347+0000 7fd33ea00640 1 -- 172.21.15.44:0/3683572090 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380b3d70 con 0x5617b5ae50a0 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.347+0000 7fd33ea00640 1 -- 172.21.15.44:0/3683572090 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd338010a90 con 0x5617b5ae50a0 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.347+0000 7fd33ea00640 1 -- 172.21.15.44:0/3683572090 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380a66b0 con 0x5617b5ae50a0 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3683572090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 msgr2=0x5617b5ed0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed0210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd3380e1be0 tx=0x7fd3380095b0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3683572090 shutdown_connections 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5b0df60 0x5617b5ecaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5aed280 0x5617b5eba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3683572090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed0210 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3683572090 >> 172.21.15.44:0/3683572090 conn(0x5617b5b02350 msgr2=0x5617b5eabd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:17.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3683572090 shutdown_connections 2024-07-26T08:38:17.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.348+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3683572090 wait complete. 2024-07-26T08:38:17.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 -- start start 2024-07-26T08:38:17.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 Processor -- start 2024-07-26T08:38:17.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 0x5617b5eda210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5ed8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ed1500 con 0x5617b5b0df60 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ed0e30 con 0x5617b5ae50a0 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5617b5ed1bd0 con 0x5617b5aed280 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.349+0000 7fd35bdc3740 10 client.0 _ll_register_callbacks cb 0x7fd3380aa4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 0x5617b5eda210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5ed8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 0x5617b5eda210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52172/0 (socket says 172.21.15.44:52172) 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47634/0 (socket says 172.21.15.44:47634) 2024-07-26T08:38:17.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34d400640 1 -- 172.21.15.44:0/3444230232 learned_addr learned my addr 172.21.15.44:0/3444230232 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 -- 172.21.15.44:0/3444230232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 msgr2=0x5617b5eda210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 0x5617b5eda210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 -- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 msgr2=0x5617b5ed8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5ed8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34de00640 1 -- 172.21.15.44:0/3444230232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b5ed92d0 con 0x5617b5ae50a0 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.350+0000 7fd34d400640 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 0x5617b5eda210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd34de00640 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed9c70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5617b5eb1e30 tx=0x7fd33800efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9fdcc4ae2eadec6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:17.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd333e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd333e00640 1 -- 172.21.15.44:0/3444230232 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380b3d70 con 0x5617b5ae50a0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd333e00640 1 -- 172.21.15.44:0/3444230232 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fd3380aa130 con 0x5617b5ae50a0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd338010a90 con 0x5617b5ae50a0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd333e00640 1 -- 172.21.15.44:0/3444230232 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd3380b3a80 con 0x5617b5ae50a0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd35bdc3740 10 client.25361 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617b5f39490 con 0x5617b5ae50a0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.351+0000 7fd333e00640 1 -- 172.21.15.44:0/3444230232 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd338010a90 con 0x5617b5ae50a0 2024-07-26T08:38:17.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd333e00640 1 -- 172.21.15.44:0/3444230232 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd3380a9b20 con 0x5617b5ae50a0 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd35bdc3740 10 client.25361 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd35bdc3740 10 client.25361 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd35bdc3740 20 client.25361 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5aa1c90 0x5617b5f3c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd35bdc3740 4 client.25361 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "status"}) -- 0x5617b5e24a80 con 0x5617b5aa1c90 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.352+0000 7fd34d400640 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5aa1c90 0x5617b5f3c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd34d400640 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5aa1c90 0x5617b5f3c8b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7393d4f40ee309d server_cookie=af44bbcad5fa6e0 in_seq=0 out_seq=0 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd333e00640 10 client.25361 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd333e00640 1 -- 172.21.15.44:0/3444230232 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5617b5e24a80 con 0x5617b5aa1c90 2024-07-26T08:38:17.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd333e00640 10 client.25361 handle_command_reply: tid=0 2024-07-26T08:38:17.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 client.25361 shutdown 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 msgr2=0x5617b5ed9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed9c70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5617b5eb1e30 tx=0x7fd33800efb0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 shutdown_connections 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x5617b5aa1c90 0x5617b5f3c8b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617b5aed280 0x5617b5eda210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5617b5b0df60 0x5617b5ed8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd35bdc3740 1 --2- 172.21.15.44:0/3444230232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617b5ae50a0 0x5617b5ed9c70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.354+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 >> 172.21.15.44:0/3444230232 conn(0x5617b5b02350 msgr2=0x5617b5eba030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.353+0000 7fd333e00640 0 client.25361 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:38:17.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.354+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 shutdown_connections 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.354+0000 7fd35bdc3740 1 -- 172.21.15.44:0/3444230232 wait complete. 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.354+0000 7fd35bdc3740 20 client.25361 trim_cache size 0 max 16384 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 1, 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24686, 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:38:17.362 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 229.309437277, 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 342.17380454300002 2024-07-26T08:38:17.363 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd3340b3ca0 msgr2=0x7fd3340b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd3340b3ca0 0x7fd3340b61b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3480096c0 tx=0x7fd34802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 msgr2=0x7fd35019c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 0x7fd35019c060 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd33800e9c0 tx=0x7fd33800ee80 comp rx=0 tx=0).stop 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 shutdown_connections 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd3340b3ca0 0x7fd3340b61b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd350104ce0 0x7fd35019c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd35010d860 0x7fd3501a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 --2- 172.21.15.44:0/514788771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3501042b0 0x7fd35019c060 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 >> 172.21.15.44:0/514788771 conn(0x7fd350079160 msgr2=0x7fd350109680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 shutdown_connections 2024-07-26T08:38:17.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:17.355+0000 7fd355200640 1 -- 172.21.15.44:0/514788771 wait complete. 2024-07-26T08:38:18.207 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T08:38:18.709 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:18 smithi044 ceph-mon[32404]: pgmap v471: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:18 smithi116 ceph-mon[37097]: pgmap v471: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:18 smithi171 ceph-mon[37789]: pgmap v471: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:20 smithi044 ceph-mon[32404]: pgmap v472: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:20 smithi116 ceph-mon[37097]: pgmap v472: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:20 smithi171 ceph-mon[37789]: pgmap v472: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:21.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 -- 172.21.15.44:0/472513753 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010ba70 msgr2=0x7fbee01122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 --2- 172.21.15.44:0/472513753 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010ba70 0x7fbee01122a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009ca0 tx=0x7fbed802a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 -- 172.21.15.44:0/472513753 shutdown_connections 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 --2- 172.21.15.44:0/472513753 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010ba70 0x7fbee01122a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 --2- 172.21.15.44:0/472513753 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee010b120 0x7fbee010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 --2- 172.21.15.44:0/472513753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee01057e0 0x7fbee0105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 -- 172.21.15.44:0/472513753 >> 172.21.15.44:0/472513753 conn(0x7fbee01011f0 msgr2=0x7fbee0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 -- 172.21.15.44:0/472513753 shutdown_connections 2024-07-26T08:38:21.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.998+0000 7fbee7e00640 1 -- 172.21.15.44:0/472513753 wait complete. 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.999+0000 7fbee7e00640 1 Processor -- start 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:20.999+0000 7fbee7e00640 1 -- start start 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 0x7fbee019c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010b120 0x7fbee019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee010ba70 0x7fbee01a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee7e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0115720 con 0x7fbee01057e0 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbee01155a0 con 0x7fbee010ba70 2024-07-26T08:38:21.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee7e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbee01158a0 con 0x7fbee010b120 2024-07-26T08:38:21.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 0x7fbee019c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 0x7fbee019c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52674/0 (socket says 172.21.15.44:52674) 2024-07-26T08:38:21.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee5400640 1 -- 172.21.15.44:0/1930785377 learned_addr learned my addr 172.21.15.44:0/1930785377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:21.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee4a00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010b120 0x7fbee019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.000+0000 7fbee5e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee010ba70 0x7fbee01a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee5400640 1 -- 172.21.15.44:0/1930785377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010b120 msgr2=0x7fbee019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee5400640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010b120 0x7fbee019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee5400640 1 -- 172.21.15.44:0/1930785377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee010ba70 msgr2=0x7fbee01a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee5400640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee010ba70 0x7fbee01a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee5400640 1 -- 172.21.15.44:0/1930785377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee01a2180 con 0x7fbee01057e0 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee4a00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010b120 0x7fbee019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:21.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee5400640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 0x7fbee019c210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e9f0 tx=0x7fbed400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:21.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbede000640 1 -- 172.21.15.44:0/1930785377 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed400cd80 con 0x7fbee01057e0 2024-07-26T08:38:21.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee01a2470 con 0x7fbee01057e0 2024-07-26T08:38:21.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.001+0000 7fbede000640 1 -- 172.21.15.44:0/1930785377 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed40108e0 con 0x7fbee01057e0 2024-07-26T08:38:21.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.002+0000 7fbede000640 1 -- 172.21.15.44:0/1930785377 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4018980 con 0x7fbee01057e0 2024-07-26T08:38:21.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.002+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01a8bd0 con 0x7fbee01057e0 2024-07-26T08:38:21.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.006+0000 7fbeeeaba740 1 Processor -- start 2024-07-26T08:38:21.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.006+0000 7fbeeeaba740 1 -- start start 2024-07-26T08:38:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 0x5566d7d12550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d12a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d12fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566d7c88950 con 0x5566d794c280 2024-07-26T08:38:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566d7c77e50 con 0x5566d796cf60 2024-07-26T08:38:21.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5566d7c7be90 con 0x5566d79440a0 2024-07-26T08:38:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbee5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 0x5566d7d12550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbee5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 0x5566d7d12550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47658/0 (socket says 172.21.15.44:47658) 2024-07-26T08:38:21.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbee5e00640 1 -- 172.21.15.44:0/2869049523 learned_addr learned my addr 172.21.15.44:0/2869049523 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbede000640 1 -- 172.21.15.44:0/1930785377 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fbed4018b20 con 0x7fbee01057e0 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.007+0000 7fbee4a00640 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d12fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbee5e00640 1 -- 172.21.15.44:0/2869049523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 msgr2=0x5566d7d12a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d12a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbee5e00640 1 -- 172.21.15.44:0/2869049523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 msgr2=0x5566d7d12fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d12fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbee5e00640 1 -- 172.21.15.44:0/2869049523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d7c1f0f0 con 0x5566d796cf60 2024-07-26T08:38:21.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 0x5566d7d12550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbed802f410 tx=0x7fbed802f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=332552cae22a94b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:21.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbed2600640 1 -- 172.21.15.44:0/2869049523 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8031cf0 con 0x5566d796cf60 2024-07-26T08:38:21.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.008+0000 7fbed2600640 1 -- 172.21.15.44:0/2869049523 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed8036030 con 0x5566d796cf60 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.009+0000 7fbed2600640 1 -- 172.21.15.44:0/2869049523 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed80042a0 con 0x5566d796cf60 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.009+0000 7fbede000640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbec80b3c20 0x7fbec80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.009+0000 7fbede000640 1 -- 172.21.15.44:0/1930785377 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fbed40e4510 con 0x7fbee01057e0 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.009+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2869049523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 msgr2=0x5566d7d12550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.009+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 0x5566d7d12550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbed802f410 tx=0x7fbed802f440 comp rx=0 tx=0).stop 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbee5400640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbec80b3c20 0x7fbec80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2869049523 shutdown_connections 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d12a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d12fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2869049523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d796cf60 0x5566d7d12550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2869049523 >> 172.21.15.44:0/2869049523 conn(0x5566d7961350 msgr2=0x5566d7d10440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:21.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2869049523 shutdown_connections 2024-07-26T08:38:21.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2869049523 wait complete. 2024-07-26T08:38:21.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.010+0000 7fbee5400640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbec80b3c20 0x7fbec80b6130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbed40aaf90 tx=0x7fbed4018420 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:21.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 -- start start 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 Processor -- start 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d1e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d1e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d796cf60 0x5566d7d29e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566d7adb5f0 con 0x5566d794c280 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566d7c14520 con 0x5566d79440a0 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.011+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d20b50 con 0x5566d796cf60 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbeeeaba740 10 client.0 _ll_register_callbacks cb 0x5566d7c23ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d1e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee4a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d1e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d1e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47660/0 (socket says 172.21.15.44:47660) 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 -- 172.21.15.44:0/2046778321 learned_addr learned my addr 172.21.15.44:0/2046778321 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d796cf60 0x5566d7d29e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 -- 172.21.15.44:0/2046778321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d796cf60 msgr2=0x5566d7d29e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d796cf60 0x5566d7d29e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 -- 172.21.15.44:0/2046778321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 msgr2=0x5566d7d1e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d1e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee5e00640 1 -- 172.21.15.44:0/2046778321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d79d8540 con 0x5566d79440a0 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.012+0000 7fbee4a00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d1e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbee5e00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d1e020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005c90 tx=0x7fbed8003df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db271bcc5c8a5b0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbec6a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbec6a00640 1 -- 172.21.15.44:0/2046778321 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8059020 con 0x5566d79440a0 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbed8036030 con 0x5566d79440a0 2024-07-26T08:38:21.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbec6a00640 1 -- 172.21.15.44:0/2046778321 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fbed8055a80 con 0x5566d79440a0 2024-07-26T08:38:21.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbec6a00640 1 -- 172.21.15.44:0/2046778321 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbed8031e60 con 0x5566d79440a0 2024-07-26T08:38:21.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbeeeaba740 10 client.25373 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:21.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.013+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566d7d2ca70 con 0x5566d79440a0 2024-07-26T08:38:21.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbec6a00640 1 -- 172.21.15.44:0/2046778321 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed8036030 con 0x5566d79440a0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbec6a00640 1 -- 172.21.15.44:0/2046778321 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbed8043070 con 0x5566d79440a0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbeeeaba740 10 client.25373 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbeeeaba740 10 client.25373 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbeeeaba740 20 client.25373 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d792b8e0 0x5566d7d30110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbeeeaba740 4 client.25373 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.014+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566d7c8a8a0 con 0x5566d792b8e0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.015+0000 7fbee4a00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d792b8e0 0x5566d7d30110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.015+0000 7fbee4a00640 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d792b8e0 0x5566d7d30110 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ece92113c76a5d1 server_cookie=1f6f56638f6486ce in_seq=0 out_seq=0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.015+0000 7fbec6a00640 10 client.25373 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.016+0000 7fbec6a00640 1 -- 172.21.15.44:0/2046778321 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5566d7c8a8a0 con 0x5566d792b8e0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.016+0000 7fbec6a00640 10 client.25373 handle_command_reply: tid=0 2024-07-26T08:38:21.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.016+0000 7fbeeeaba740 1 client.25373 shutdown 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 msgr2=0x5566d7d1e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d1e020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005c90 tx=0x7fbed8003df0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 shutdown_connections 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d792b8e0 0x5566d7d30110 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d796cf60 0x5566d7d29e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d794c280 0x5566d7d1e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/2046778321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d1e020 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbec6a00640 0 client.25373 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 >> 172.21.15.44:0/2046778321 conn(0x5566d7961350 msgr2=0x5566d7d156e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 shutdown_connections 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 1 -- 172.21.15.44:0/2046778321 wait complete. 2024-07-26T08:38:21.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.017+0000 7fbeeeaba740 20 client.25373 trim_cache size 0 max 16384 2024-07-26T08:38:21.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.037+0000 7fbeeeaba740 1 Processor -- start 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.037+0000 7fbeeeaba740 1 -- start start 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d18670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 0x5566d7d18fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d29de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d2e450 con 0x5566d796cf60 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d2eb20 con 0x5566d79440a0 2024-07-26T08:38:21.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d2f1f0 con 0x5566d794c280 2024-07-26T08:38:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbee4a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 0x5566d7d18fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbee5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d18670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbee4a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 0x5566d7d18fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52224/0 (socket says 172.21.15.44:52224) 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.038+0000 7fbee4a00640 1 -- 172.21.15.44:0/932433495 learned_addr learned my addr 172.21.15.44:0/932433495 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.039+0000 7fbee5e00640 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d29de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.039+0000 7fbee4a00640 1 -- 172.21.15.44:0/932433495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 msgr2=0x5566d7d18670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.039+0000 7fbee4a00640 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d18670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.039+0000 7fbee4a00640 1 -- 172.21.15.44:0/932433495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 msgr2=0x5566d7d29de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.039+0000 7fbee4a00640 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d29de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.039+0000 7fbee4a00640 1 -- 172.21.15.44:0/932433495 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d7d153b0 con 0x5566d794c280 2024-07-26T08:38:21.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.040+0000 7fbee4a00640 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 0x5566d7d18fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbecc012700 tx=0x7fbecc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=607ed9414dd8bb03 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:21.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.040+0000 7fbed1c00640 1 -- 172.21.15.44:0/932433495 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc004050 con 0x5566d794c280 2024-07-26T08:38:21.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.040+0000 7fbed1c00640 1 -- 172.21.15.44:0/932433495 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbecc00fd20 con 0x5566d794c280 2024-07-26T08:38:21.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.040+0000 7fbed1c00640 1 -- 172.21.15.44:0/932433495 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc003990 con 0x5566d794c280 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 -- 172.21.15.44:0/932433495 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 msgr2=0x5566d7d18fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 0x5566d7d18fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbecc012700 tx=0x7fbecc005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 -- 172.21.15.44:0/932433495 shutdown_connections 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d794c280 0x5566d7d18fd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d29de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/932433495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d79440a0 0x5566d7d18670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 -- 172.21.15.44:0/932433495 >> 172.21.15.44:0/932433495 conn(0x5566d7961350 msgr2=0x5566d7d0ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.041+0000 7fbeeeaba740 1 -- 172.21.15.44:0/932433495 shutdown_connections 2024-07-26T08:38:21.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.042+0000 7fbeeeaba740 1 -- 172.21.15.44:0/932433495 wait complete. 2024-07-26T08:38:21.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.042+0000 7fbeeeaba740 1 -- start start 2024-07-26T08:38:21.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.042+0000 7fbeeeaba740 1 Processor -- start 2024-07-26T08:38:21.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d319e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 0x5566d7d31fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d36ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d2ff90 con 0x5566d796cf60 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d2f8c0 con 0x5566d794c280 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5566d7d30660 con 0x5566d79440a0 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbeeeaba740 10 client.0 _ll_register_callbacks cb 0x7fbed8031af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbee4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 0x5566d7d31fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbee4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 0x5566d7d31fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47682/0 (socket says 172.21.15.44:47682) 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbee4a00640 1 -- 172.21.15.44:0/221743504 learned_addr learned my addr 172.21.15.44:0/221743504 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:21.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbee5e00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d36ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.043+0000 7fbee5e00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee4a00640 1 -- 172.21.15.44:0/221743504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 msgr2=0x5566d7d319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee4a00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d319e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee4a00640 1 -- 172.21.15.44:0/221743504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 msgr2=0x5566d7d36ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee4a00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d36ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee4a00640 1 -- 172.21.15.44:0/221743504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d7d37510 con 0x5566d794c280 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee5e00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d36ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbee4a00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 0x5566d7d31fa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbecc014010 tx=0x7fbecc009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb6254e67be14b87 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbec6a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.044+0000 7fbec6a00640 1 -- 172.21.15.44:0/221743504 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc012290 con 0x5566d794c280 2024-07-26T08:38:21.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbecc00fd20 con 0x5566d794c280 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbec6a00640 1 -- 172.21.15.44:0/221743504 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fbecc005030 con 0x5566d794c280 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbec6a00640 1 -- 172.21.15.44:0/221743504 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbecc03c050 con 0x5566d794c280 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbeeeaba740 10 client.25379 fetch_fsmap learned FSMap version 29 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566d7d388d0 con 0x5566d794c280 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbec6a00640 1 -- 172.21.15.44:0/221743504 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbecc00fd20 con 0x5566d794c280 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbec6a00640 1 -- 172.21.15.44:0/221743504 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbecc031020 con 0x5566d794c280 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.045+0000 7fbeeeaba740 10 client.25379 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbeeeaba740 10 client.25379 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbeeeaba740 20 client.25379 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d7900c90 0x5566d7d9bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbeeeaba740 4 client.25379 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T08:38:21.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "status"}) -- 0x5566d7c86340 con 0x5566d7900c90 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbee5e00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d7900c90 0x5566d7d9bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbee5e00640 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d7900c90 0x5566d7d9bfe0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdfa7abbb738d078 server_cookie=7d03dd989d8521b5 in_seq=0 out_seq=0 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.046+0000 7fbec6a00640 10 client.25379 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbec6a00640 1 -- 172.21.15.44:0/221743504 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5566d7c86340 con 0x5566d7900c90 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbec6a00640 10 client.25379 handle_command_reply: tid=0 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 client.25379 shutdown 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 msgr2=0x5566d7d31fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 0x5566d7d31fa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbecc014010 tx=0x7fbecc009d70 comp rx=0 tx=0).stop 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 shutdown_connections 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5566d7900c90 0x5566d7d9bfe0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566d796cf60 0x5566d7d36ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d794c280 0x5566d7d31fa0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 --2- 172.21.15.44:0/221743504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5566d79440a0 0x5566d7d319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbec6a00640 0 client.25379 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 >> 172.21.15.44:0/221743504 conn(0x5566d7961350 msgr2=0x5566d7d0ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 shutdown_connections 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.047+0000 7fbeeeaba740 1 -- 172.21.15.44:0/221743504 wait complete. 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.048+0000 7fbeeeaba740 20 client.25379 trim_cache size 0 max 16384 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "10471e4e-4b28-11ef-bcb7-c7b262605968", 2024-07-26T08:38:21.052 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24701, 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 314.883255115, 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 29, 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 91, 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 91, 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 337.81248703799997 2024-07-26T08:38:21.053 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.048+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbec80b3c20 msgr2=0x7fbec80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.048+0000 7fbee7e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbec80b3c20 0x7fbec80b6130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbed40aaf90 tx=0x7fbed4018420 comp rx=0 tx=0).stop 2024-07-26T08:38:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.048+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 msgr2=0x7fbee019c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.048+0000 7fbee7e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 0x7fbee019c210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e9f0 tx=0x7fbed400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 shutdown_connections 2024-07-26T08:38:21.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fbec80b3c20 0x7fbec80b6130 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbee010b120 0x7fbee019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee010ba70 0x7fbee01a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 --2- 172.21.15.44:0/1930785377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbee01057e0 0x7fbee019c210 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 >> 172.21.15.44:0/1930785377 conn(0x7fbee01011f0 msgr2=0x7fbee01096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 shutdown_connections 2024-07-26T08:38:21.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:21.049+0000 7fbee7e00640 1 -- 172.21.15.44:0/1930785377 wait complete. 2024-07-26T08:38:22.033 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T08:38:22.043 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T08:38:22.043 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T08:38:22.044 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T08:38:22.044 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-07-26T08:38:22.045 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T08:38:22.045 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:38:22.045 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-07-26T08:38:22.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 -- 172.21.15.44:0/3935443329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 msgr2=0x7f22b0105b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:22.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 --2- 172.21.15.44:0/3935443329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b0105b30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009900 tx=0x7f22a402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 -- 172.21.15.44:0/3935443329 shutdown_connections 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 --2- 172.21.15.44:0/3935443329 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b9a0 0x7f22b01121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 --2- 172.21.15.44:0/3935443329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b050 0x7f22b010b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 --2- 172.21.15.44:0/3935443329 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b0105b30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 -- 172.21.15.44:0/3935443329 >> 172.21.15.44:0/3935443329 conn(0x7f22b0101120 msgr2=0x7f22b0103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 -- 172.21.15.44:0/3935443329 shutdown_connections 2024-07-26T08:38:22.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.329+0000 7f22b8600640 1 -- 172.21.15.44:0/3935443329 wait complete. 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.331+0000 7f22b8600640 1 Processor -- start 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.331+0000 7f22b8600640 1 -- start start 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b8600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b019c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b8600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b050 0x7f22b019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b8600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 0x7f22b01a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b8600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0115670 con 0x7f22b0105720 2024-07-26T08:38:22.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b8600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22b01154f0 con 0x7f22b010b9a0 2024-07-26T08:38:22.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b8600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f22b01157f0 con 0x7f22b010b050 2024-07-26T08:38:22.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 0x7f22b01a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:22.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b5c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b019c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:22.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b5200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b050 0x7f22b019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:22.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 0x7f22b01a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:58708/0 (socket says 172.21.15.44:58708) 2024-07-26T08:38:22.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.332+0000 7f22b6600640 1 -- 172.21.15.44:0/1217446852 learned_addr learned my addr 172.21.15.44:0/1217446852 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5200640 1 -- 172.21.15.44:0/1217446852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 msgr2=0x7f22b01a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5200640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 0x7f22b01a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5200640 1 -- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 msgr2=0x7f22b019c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5200640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b019c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5200640 1 -- 172.21.15.44:0/1217446852 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b01a2000 con 0x7f22b010b050 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5c00640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b019c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:22.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b6600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 0x7f22b01a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:22.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b5200640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b050 0x7f22b019c670 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f22b006c290 tx=0x7f229c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:22.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c00cd00 con 0x7f22b010b050 2024-07-26T08:38:22.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b01a22f0 con 0x7f22b010b050 2024-07-26T08:38:22.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f229c0035f0 con 0x7f22b010b050 2024-07-26T08:38:22.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c010690 con 0x7f22b010b050 2024-07-26T08:38:22.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.333+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b01a8a80 con 0x7f22b010b050 2024-07-26T08:38:22.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.335+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b0105b30 con 0x7f22b010b050 2024-07-26T08:38:22.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.337+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f229c0037a0 con 0x7f22b010b050 2024-07-26T08:38:22.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.340+0000 7f22ae000640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f22980b3c20 0x7f22980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:22.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.340+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f229c0e45a0 con 0x7f22b010b050 2024-07-26T08:38:22.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.340+0000 7f22b5c00640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f22980b3c20 0x7f22980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:22.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.341+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f229c055600 con 0x7f22b010b050 2024-07-26T08:38:22.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.341+0000 7f22b5c00640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f22980b3c20 0x7f22980b6130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f22a4031440 tx=0x7f22a4032ed0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:22.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:22.617+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f22b002e0c0 con 0x7f22b010b050 2024-07-26T08:38:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:22 smithi044 ceph-mon[32404]: pgmap v473: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:22 smithi116 ceph-mon[37097]: pgmap v473: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:22 smithi171 ceph-mon[37789]: pgmap v473: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:23.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.508+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f229c0a7270 con 0x7f22b010b050 2024-07-26T08:38:23.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.510+0000 7f22ae000640 1 -- 172.21.15.44:0/1217446852 <== mon.1 v2:172.21.15.171:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 module 'snap_schedule' is already enabled v23) ==== 133+0+0 (secure 0 0 0) 0x7f229c0e52c0 con 0x7f22b010b050 2024-07-26T08:38:23.513 INFO:teuthology.orchestra.run.smithi044.stderr:module 'snap_schedule' is already enabled 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.514+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f22980b3c20 msgr2=0x7f22980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.514+0000 7f22b8600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f22980b3c20 0x7f22980b6130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f22a4031440 tx=0x7f22a4032ed0 comp rx=0 tx=0).stop 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.514+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b050 msgr2=0x7f22b019c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.514+0000 7f22b8600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b050 0x7f22b019c670 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f22b006c290 tx=0x7f229c00ee50 comp rx=0 tx=0).stop 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 shutdown_connections 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f22980b3c20 0x7f22980b6130 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f22b010b050 0x7f22b019c670 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22b010b9a0 0x7f22b01a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 --2- 172.21.15.44:0/1217446852 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22b0105720 0x7f22b019c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:23.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 >> 172.21.15.44:0/1217446852 conn(0x7f22b0101120 msgr2=0x7f22b0103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:23.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 shutdown_connections 2024-07-26T08:38:23.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:23.516+0000 7f22b8600640 1 -- 172.21.15.44:0/1217446852 wait complete. 2024-07-26T08:38:23.547 INFO:teuthology.run_tasks:Running task kclient... 2024-07-26T08:38:23.564 INFO:tasks.kclient:Mounting kernel clients... 2024-07-26T08:38:23.564 INFO:tasks.kclient:config is {'client.0': {}} 2024-07-26T08:38:23.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:23.565 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-07-26T08:38:23.618 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-07-26T08:38:23.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:23.620 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T08:38:23.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:23.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-07-26T08:38:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:23 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1217446852' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:38:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:23 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:38:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:23 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1217446852' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:38:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:23 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:38:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:23 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1217446852' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:38:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:23 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T08:38:24.095 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-07-26T08:38:24.095 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T08:38:24.095 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:24.096 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T08:38:24.562 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:24 smithi044 ceph-mon[32404]: pgmap v474: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:24 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T08:38:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:24 smithi044 ceph-mon[32404]: mgrmap e23: x(active, since 13m), standbys: z, y 2024-07-26T08:38:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:24 smithi116 ceph-mon[37097]: pgmap v474: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:24 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T08:38:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:24 smithi116 ceph-mon[37097]: mgrmap e23: x(active, since 13m), standbys: z, y 2024-07-26T08:38:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:24 smithi171 ceph-mon[37789]: pgmap v474: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:38:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:24 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T08:38:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:24 smithi171 ceph-mon[37789]: mgrmap e23: x(active, since 13m), standbys: z, y 2024-07-26T08:38:26.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:26 smithi044 ceph-mon[32404]: pgmap v475: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:26 smithi116 ceph-mon[37097]: pgmap v475: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:26 smithi171 ceph-mon[37789]: pgmap v475: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:27.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 -- 172.21.15.44:0/232688273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810b120 msgr2=0x7f693810b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:27.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 --2- 172.21.15.44:0/232688273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810b120 0x7f693810b530 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6924009900 tx=0x7f692402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 -- 172.21.15.44:0/232688273 shutdown_connections 2024-07-26T08:38:27.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 --2- 172.21.15.44:0/232688273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f693810ba70 0x7f69381122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 --2- 172.21.15.44:0/232688273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810b120 0x7f693810b530 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 --2- 172.21.15.44:0/232688273 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f69381057e0 0x7f6938105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 -- 172.21.15.44:0/232688273 >> 172.21.15.44:0/232688273 conn(0x7f69381011f0 msgr2=0x7f6938103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:27.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.001+0000 7f6940600640 1 -- 172.21.15.44:0/232688273 shutdown_connections 2024-07-26T08:38:27.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.002+0000 7f6940600640 1 -- 172.21.15.44:0/232688273 wait complete. 2024-07-26T08:38:27.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 Processor -- start 2024-07-26T08:38:27.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 -- start start 2024-07-26T08:38:27.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69381057e0 0x7f693819c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:27.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 0x7f693819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:27.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810ba70 0x7f69381a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f69381156e0 con 0x7f693810ba70 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6938115560 con 0x7f69381057e0 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.003+0000 7f6940600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6938115860 con 0x7f693810b120 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 0x7f693819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693e600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810ba70 0x7f69381a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 0x7f693819c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:53364/0 (socket says 172.21.15.44:53364) 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 -- 172.21.15.44:0/4203782821 learned_addr learned my addr 172.21.15.44:0/4203782821 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:27.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693dc00640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69381057e0 0x7f693819c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:27.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 -- 172.21.15.44:0/4203782821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69381057e0 msgr2=0x7f693819c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:27.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69381057e0 0x7f693819c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 -- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810ba70 msgr2=0x7f69381a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:27.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810ba70 0x7f69381a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693d200640 1 -- 172.21.15.44:0/4203782821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69381a2140 con 0x7f693810b120 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.004+0000 7f693e600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810ba70 0x7f69381a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.005+0000 7f693d200640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 0x7f693819c770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f692402a910 tx=0x7f6924031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.005+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692403f030 con 0x7f693810b120 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.005+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69381a23d0 con 0x7f693810b120 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.005+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6924004490 con 0x7f693810b120 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.005+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924035070 con 0x7f693810b120 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.006+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69381a8b90 con 0x7f693810b120 2024-07-26T08:38:27.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.007+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6938105bf0 con 0x7f693810b120 2024-07-26T08:38:27.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.009+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f6924042030 con 0x7f693810b120 2024-07-26T08:38:27.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.011+0000 7f6936000640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69200b3c20 0x7f69200b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:27.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.012+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f6924104440 con 0x7f693810b120 2024-07-26T08:38:27.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.012+0000 7f693dc00640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69200b3c20 0x7f69200b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:27.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.013+0000 7f693dc00640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69200b3c20 0x7f69200b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f692c005fd0 tx=0x7f692c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:27.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.015+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f69240caf30 con 0x7f693810b120 2024-07-26T08:38:27.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.265+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f693810d850 con 0x7f693810b120 2024-07-26T08:38:27.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.266+0000 7f6936000640 1 -- 172.21.15.44:0/4203782821 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+88 (secure 0 0 0) 0x7f69240cf480 con 0x7f693810b120 2024-07-26T08:38:27.267 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T08:38:27.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69200b3c20 msgr2=0x7f69200b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:27.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69200b3c20 0x7f69200b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f692c005fd0 tx=0x7f692c009500 comp rx=0 tx=0).stop 2024-07-26T08:38:27.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 msgr2=0x7f693819c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:27.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 0x7f693819c770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f692402a910 tx=0x7f6924031a20 comp rx=0 tx=0).stop 2024-07-26T08:38:27.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 shutdown_connections 2024-07-26T08:38:27.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f69200b3c20 0x7f69200b6130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f693810b120 0x7f693819c770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f693810ba70 0x7f69381a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 --2- 172.21.15.44:0/4203782821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69381057e0 0x7f693819c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:27.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.269+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 >> 172.21.15.44:0/4203782821 conn(0x7f69381011f0 msgr2=0x7f6938109770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:27.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.270+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 shutdown_connections 2024-07-26T08:38:27.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:27.270+0000 7f6940600640 1 -- 172.21.15.44:0/4203782821 wait complete. 2024-07-26T08:38:27.679 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:27 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/4203782821' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T08:38:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:27 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/4203782821' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T08:38:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:27 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/4203782821' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T08:38:28.203 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T08:38:28.204 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T08:38:28.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:28.204 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-07-26T08:38:28.264 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T08:38:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T08:38:28.264 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-07-26T08:38:28.265 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 41569sec preferred_lft 41569sec 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-07-26T08:38:28.266 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-07-26T08:38:28.267 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T08:38:28.267 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:38:28.267 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:38:28.267 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-07-26T08:38:28.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-07-26T08:38:28.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-07-26T08:38:28.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T08:38:28.268 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:38:28.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:38:28.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:28.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:28.537 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T08:38:28.601 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-07-26T08:38:28.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:28.605 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-07-26T08:38:28.651 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-07-26T08:38:28.651 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T08:38:28.651 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-07-26T08:38:28.651 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-07-26T08:38:28.651 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T08:38:28.652 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:38:28.652 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:38:28.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-07-26T08:38:28.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-07-26T08:38:28.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-07-26T08:38:28.652 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:38:28.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:38:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:28 smithi044 ceph-mon[32404]: pgmap v476: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:28 smithi116 ceph-mon[37097]: pgmap v476: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:28 smithi171 ceph-mon[37789]: pgmap v476: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:38:28.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:28.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:28.861 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-07-26T08:38:28.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:28.910 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-07-26T08:38:28.967 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:38:28.968 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:38:28.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T08:38:28.968 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T08:38:28.968 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:38:29.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:38:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:29.159 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T08:38:29.159 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:38:29.159 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:38:29.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T08:38:29.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T08:38:29.159 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T08:38:29.160 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T08:38:29.160 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T08:38:29.160 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:38:29.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:38:29.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:29.442 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T08:38:29.442 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-07-26T08:38:29.442 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-07-26T08:38:29.442 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T08:38:29.442 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-07-26T08:38:29.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:38:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:29.643 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T08:38:29.644 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:29.687 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T08:38:29.688 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T08:38:29.688 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:29.784 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-07-26T08:38:29.784 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-07-26T08:38:29.784 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/b30a7374-b969-4a02-9f96-a8a2cec18bd0 2024-07-26T08:38:29.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:29.784 DEBUG:teuthology.orchestra.run.smithi044:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/b30a7374-b969-4a02-9f96-a8a2cec18bd0 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-07-26T08:38:29.991 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-07-26T08:38:29.991 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-07-26T08:38:29.991 INFO:teuthology.orchestra.run.smithi044.stdout:invalid new device string format 2024-07-26T08:38:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:30 smithi116 ceph-mon[37097]: pgmap v477: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:30.778 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-07-26T08:38:30.778 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-07-26T08:38:30.778 INFO:teuthology.orchestra.run.smithi044.stdout:invalid new device string format 2024-07-26T08:38:30.778 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: resolved to: "172.21.15.44:3300,172.21.15.116:3300,172.21.15.171:3300" 2024-07-26T08:38:30.778 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.44:3300,172.21.15.116:3300,172.21.15.171:3300:/volumes/qa/sv_1/b30a7374-b969-4a02-9f96-a8a2cec18bd0 2024-07-26T08:38:30.778 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=10471e4e-4b28-11ef-bcb7-c7b262605968" will pass to kernel 2024-07-26T08:38:30.782 INFO:teuthology.orchestra.run.smithi044.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T08:38:30.782 INFO:teuthology.orchestra.run.smithi044.stdout: You just mounted a file system that supports labels which does not 2024-07-26T08:38:30.782 INFO:teuthology.orchestra.run.smithi044.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T08:38:30.782 INFO:teuthology.orchestra.run.smithi044.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T08:38:30.782 INFO:teuthology.orchestra.run.smithi044.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T08:38:30.784 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:30 smithi044 ceph-mon[32404]: pgmap v477: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:30.788 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T08:38:30.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T08:38:30.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:30 smithi171 ceph-mon[37789]: pgmap v477: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:38:30.877 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T08:38:30.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T08:38:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:31.003 INFO:teuthology.orchestra.run.smithi044.stdout:client25387 2024-07-26T08:38:31.003 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:38:31.003 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T08:38:31.058 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T08:38:31.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T08:38:31.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:31.186 INFO:teuthology.orchestra.run.smithi044.stdout:client25387 2024-07-26T08:38:31.186 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T08:38:31.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T08:38:31.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:31.307 INFO:teuthology.orchestra.run.smithi044.stdout:client25387 2024-07-26T08:38:31.307 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968.client25387/status') 2024-07-26T08:38:31.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T08:38:31.447 INFO:teuthology.orchestra.run.smithi044.stdout:instance: client.25387 (3)172.21.15.44:0/1790199663 2024-07-26T08:38:31.447 INFO:teuthology.orchestra.run.smithi044.stdout:blocklisted: false 2024-07-26T08:38:31.448 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records in 2024-07-26T08:38:31.448 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records out 2024-07-26T08:38:31.448 INFO:teuthology.orchestra.run.smithi044.stderr:71 bytes copied, 8.882e-05 s, 799 kB/s 2024-07-26T08:38:31.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T08:38:31.461 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T08:38:31.470 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T08:38:31.470 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:38:31.470 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-07-26T08:38:31.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 -- 172.21.15.44:0/3153843008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 msgr2=0x7fb2e8105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 --2- 172.21.15.44:0/3153843008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e8105950 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009ca0 tx=0x7fb2d802a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 -- 172.21.15.44:0/3153843008 shutdown_connections 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 --2- 172.21.15.44:0/3153843008 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e810ae70 0x7fb2e810b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 --2- 172.21.15.44:0/3153843008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 0x7fb2e8111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 --2- 172.21.15.44:0/3153843008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e8105950 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 -- 172.21.15.44:0/3153843008 >> 172.21.15.44:0/3153843008 conn(0x7fb2e8100f20 msgr2=0x7fb2e8103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 -- 172.21.15.44:0/3153843008 shutdown_connections 2024-07-26T08:38:31.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.755+0000 7fb2e7200640 1 -- 172.21.15.44:0/3153843008 wait complete. 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.756+0000 7fb2e7200640 1 Processor -- start 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.756+0000 7fb2e7200640 1 -- start start 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e7200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e810ae70 0x7fb2e819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e7200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 0x7fb2e81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e7200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8115490 con 0x7fb2e810b7c0 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8115310 con 0x7fb2e8105540 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e7200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8115610 con 0x7fb2e810ae70 2024-07-26T08:38:31.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e819bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54538/0 (socket says 172.21.15.44:54538) 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 -- 172.21.15.44:0/2507294930 learned_addr learned my addr 172.21.15.44:0/2507294930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e6800640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 0x7fb2e81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5400640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e810ae70 0x7fb2e819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 -- 172.21.15.44:0/2507294930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e810ae70 msgr2=0x7fb2e819c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e810ae70 0x7fb2e819c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 -- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 msgr2=0x7fb2e81a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 0x7fb2e81a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e5e00640 1 -- 172.21.15.44:0/2507294930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e81a1e80 con 0x7fb2e8105540 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.757+0000 7fb2e6800640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 0x7fb2e81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:31.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.758+0000 7fb2e5e00640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e819bf50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8033040 tx=0x7fb2d80316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:31.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.758+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8003b70 con 0x7fb2e8105540 2024-07-26T08:38:31.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.758+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d8003d10 con 0x7fb2e8105540 2024-07-26T08:38:31.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.758+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d802f9e0 con 0x7fb2e8105540 2024-07-26T08:38:31.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.758+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e81a2110 con 0x7fb2e8105540 2024-07-26T08:38:31.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.758+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e81a8900 con 0x7fb2e8105540 2024-07-26T08:38:31.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.759+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e8105950 con 0x7fb2e8105540 2024-07-26T08:38:31.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.762+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7fb2d802fba0 con 0x7fb2e8105540 2024-07-26T08:38:31.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.763+0000 7fb2dea00640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb2cc0b3ca0 0x7fb2cc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:31.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.763+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb2d8105790 con 0x7fb2e8105540 2024-07-26T08:38:31.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.764+0000 7fb2e5400640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb2cc0b3ca0 0x7fb2cc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:31.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.764+0000 7fb2e5400640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb2cc0b3ca0 0x7fb2cc0b61b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8104750 tx=0x7fb2d0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:31.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:31.765+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb2d802fe90 con 0x7fb2e8105540 2024-07-26T08:38:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.019+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fb2e819d070 con 0x7fb2e8105540 2024-07-26T08:38:32.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.528+0000 7fb2dea00640 1 -- 172.21.15.44:0/2507294930 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v30) ==== 117+0+0 (secure 0 0 0) 0x7fb2e819d070 con 0x7fb2e8105540 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb2cc0b3ca0 msgr2=0x7fb2cc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb2cc0b3ca0 0x7fb2cc0b61b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8104750 tx=0x7fb2d0005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 msgr2=0x7fb2e819bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e819bf50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8033040 tx=0x7fb2d80316f0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 shutdown_connections 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb2cc0b3ca0 0x7fb2cc0b61b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e810ae70 0x7fb2e819c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb2e810b7c0 0x7fb2e81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 --2- 172.21.15.44:0/2507294930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2e8105540 0x7fb2e819bf50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 >> 172.21.15.44:0/2507294930 conn(0x7fb2e8100f20 msgr2=0x7fb2e8103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:32.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 shutdown_connections 2024-07-26T08:38:32.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.532+0000 7fb2e7200640 1 -- 172.21.15.44:0/2507294930 wait complete. 2024-07-26T08:38:32.566 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-26T08:38:32.578 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-07-26T08:38:32.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T08:38:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:32 smithi171 ceph-mon[37789]: pgmap v478: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:32 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2507294930' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T08:38:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:32 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T08:38:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 -- 172.21.15.44:0/1651394346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 msgr2=0x7ff85c10f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 --2- 172.21.15.44:0/1651394346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 0x7ff85c10f650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff844009860 tx=0x7ff84402a340 comp rx=0 tx=0).stop 2024-07-26T08:38:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 -- 172.21.15.44:0/1651394346 shutdown_connections 2024-07-26T08:38:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 --2- 172.21.15.44:0/1651394346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 0x7ff85c10f650 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 --2- 172.21.15.44:0/1651394346 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 0x7ff85c1163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 --2- 172.21.15.44:0/1651394346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 -- 172.21.15.44:0/1651394346 >> 172.21.15.44:0/1651394346 conn(0x7ff85c078f80 msgr2=0x7ff85c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 -- 172.21.15.44:0/1651394346 shutdown_connections 2024-07-26T08:38:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.944+0000 7ff85b200640 1 -- 172.21.15.44:0/1651394346 wait complete. 2024-07-26T08:38:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.946+0000 7ff85b200640 1 Processor -- start 2024-07-26T08:38:32.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.946+0000 7ff85b200640 1 -- start start 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c1a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85b200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 0x7ff85c1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 0x7ff85c1a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c119850 con 0x7ff85c10fb90 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1196d0 con 0x7ff85c109910 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85b200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1199d0 con 0x7ff85c10f240 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c1a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c1a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54552/0 (socket says 172.21.15.44:54552) 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 -- 172.21.15.44:0/2892132730 learned_addr learned my addr 172.21.15.44:0/2892132730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859400640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 0x7ff85c1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:32.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff85a800640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 0x7ff85c1a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 -- 172.21.15.44:0/2892132730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 msgr2=0x7ff85c1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 0x7ff85c1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 -- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 msgr2=0x7ff85c1a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 0x7ff85c1a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.947+0000 7ff859e00640 1 -- 172.21.15.44:0/2892132730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c1a6180 con 0x7ff85c109910 2024-07-26T08:38:32.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff85a800640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 0x7ff85c1a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff859e00640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c1a0250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00e9f0 tx=0x7ff84c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c00cd80 con 0x7ff85c109910 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85c1a6470 con 0x7ff85c109910 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff84c0108e0 con 0x7ff85c109910 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c018980 con 0x7ff85c109910 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.948+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85c1acc00 con 0x7ff85c109910 2024-07-26T08:38:32.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.949+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85c109d20 con 0x7ff85c109910 2024-07-26T08:38:32.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.952+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7ff84c010430 con 0x7ff85c109910 2024-07-26T08:38:32.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.953+0000 7ff852a00640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff8400b3c20 0x7ff8400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:32.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.954+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff84c0e49f0 con 0x7ff85c109910 2024-07-26T08:38:32.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.954+0000 7ff859400640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff8400b3c20 0x7ff8400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.955+0000 7ff859400640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff8400b3c20 0x7ff8400b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff844005940 tx=0x7ff8440058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:32.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:32.955+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff84c0abb30 con 0x7ff85c109910 2024-07-26T08:38:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:32 smithi044 ceph-mon[32404]: pgmap v478: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:32 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2507294930' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T08:38:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:32 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T08:38:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:32 smithi116 ceph-mon[37097]: pgmap v478: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:32 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2507294930' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T08:38:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:32 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T08:38:33.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.172+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff85c1a1370 con 0x7ff85c109910 2024-07-26T08:38:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.174+0000 7ff852a00640 1 -- 172.21.15.44:0/2892132730 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7ff84c0ab430 con 0x7ff85c109910 2024-07-26T08:38:33.176 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:33.177 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:38:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.177+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff8400b3c20 msgr2=0x7ff8400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.177+0000 7ff85b200640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff8400b3c20 0x7ff8400b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff844005940 tx=0x7ff8440058d0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.177+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 msgr2=0x7ff85c1a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.177+0000 7ff85b200640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c1a0250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00e9f0 tx=0x7ff84c00eeb0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 shutdown_connections 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff8400b3c20 0x7ff8400b6130 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff85c10f240 0x7ff85c1a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff85c10fb90 0x7ff85c1a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 --2- 172.21.15.44:0/2892132730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff85c109910 0x7ff85c1a0250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 >> 172.21.15.44:0/2892132730 conn(0x7ff85c078f80 msgr2=0x7ff85c107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 shutdown_connections 2024-07-26T08:38:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.178+0000 7ff85b200640 1 -- 172.21.15.44:0/2892132730 wait complete. 2024-07-26T08:38:33.206 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T08:27:40.734083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-07-26T08:32:56.237662+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-07-26T08:33:01.125230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-07-26T08:33:15.867271+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-07-26T08:38:33.207 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T08:38:33.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 -- 172.21.15.44:0/3436479716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c075800 msgr2=0x7f571c075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 --2- 172.21.15.44:0/3436479716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c075800 0x7f571c075c10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5704009900 tx=0x7f570402a410 comp rx=0 tx=0).stop 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 -- 172.21.15.44:0/3436479716 shutdown_connections 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 --2- 172.21.15.44:0/3436479716 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c113be0 0x7f571c1160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 --2- 172.21.15.44:0/3436479716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c075800 0x7f571c075c10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 --2- 172.21.15.44:0/3436479716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f571c076dc0 0x7f571c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 -- 172.21.15.44:0/3436479716 >> 172.21.15.44:0/3436479716 conn(0x7f571c0fed80 msgr2=0x7f571c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 -- 172.21.15.44:0/3436479716 shutdown_connections 2024-07-26T08:38:33.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.496+0000 7f571be00640 1 -- 172.21.15.44:0/3436479716 wait complete. 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.497+0000 7f571be00640 1 Processor -- start 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.497+0000 7f571be00640 1 -- start start 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 0x7f571c1a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c076dc0 0x7f571c1a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f571c113be0 0x7f571c1a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1197e0 con 0x7f571c075800 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f571c119660 con 0x7f571c076dc0 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571be00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f571c119960 con 0x7f571c113be0 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 0x7f571c1a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:33.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 0x7f571c1a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49122/0 (socket says 172.21.15.44:49122) 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571aa00640 1 -- 172.21.15.44:0/2400831202 learned_addr learned my addr 172.21.15.44:0/2400831202 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f5712000640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c076dc0 0x7f571c1a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.498+0000 7f571b400640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f571c113be0 0x7f571c1a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f571aa00640 1 -- 172.21.15.44:0/2400831202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f571c113be0 msgr2=0x7f571c1a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f571aa00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f571c113be0 0x7f571c1a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f571aa00640 1 -- 172.21.15.44:0/2400831202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c076dc0 msgr2=0x7f571c1a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f571aa00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c076dc0 0x7f571c1a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f571aa00640 1 -- 172.21.15.44:0/2400831202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c1a6170 con 0x7f571c075800 2024-07-26T08:38:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f571aa00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 0x7f571c1a0240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f570c00e9f0 tx=0x7f570c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.499+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c00cd80 con 0x7f571c075800 2024-07-26T08:38:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.500+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571c1a6460 con 0x7f571c075800 2024-07-26T08:38:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.500+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f570c0108e0 con 0x7f571c075800 2024-07-26T08:38:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.500+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c018980 con 0x7f571c075800 2024-07-26T08:38:33.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.500+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f571c1acbf0 con 0x7f571c075800 2024-07-26T08:38:33.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.501+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f571c075c10 con 0x7f571c075800 2024-07-26T08:38:33.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.504+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f570c010430 con 0x7f571c075800 2024-07-26T08:38:33.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.506+0000 7f5713e00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f57000b3c20 0x7f57000b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:33.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.506+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f570c0e4470 con 0x7f571c075800 2024-07-26T08:38:33.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.506+0000 7f5712000640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f57000b3c20 0x7f57000b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:33.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.507+0000 7f5712000640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f57000b3c20 0x7f57000b6130 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5704031440 tx=0x7f5704009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:33.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.510+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f570c0aaf60 con 0x7f571c075800 2024-07-26T08:38:33.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.725+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f571c112940 con 0x7f571c075800 2024-07-26T08:38:33.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.727+0000 7f5713e00640 1 -- 172.21.15.44:0/2400831202 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f570c0af4b0 con 0x7f571c075800 2024-07-26T08:38:33.728 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-07-26T08:38:33.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f57000b3c20 msgr2=0x7f57000b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f57000b3c20 0x7f57000b6130 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5704031440 tx=0x7f5704009560 comp rx=0 tx=0).stop 2024-07-26T08:38:33.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 msgr2=0x7f571c1a0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:33.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 0x7f571c1a0240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f570c00e9f0 tx=0x7f570c00eeb0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 shutdown_connections 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f57000b3c20 0x7f57000b6130 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f571c113be0 0x7f571c1a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c076dc0 0x7f571c1a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 --2- 172.21.15.44:0/2400831202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f571c075800 0x7f571c1a0240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 >> 172.21.15.44:0/2400831202 conn(0x7f571c0fed80 msgr2=0x7f571c10fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 shutdown_connections 2024-07-26T08:38:33.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:33.730+0000 7f571be00640 1 -- 172.21.15.44:0/2400831202 wait complete. 2024-07-26T08:38:33.758 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-07-26T08:38:33.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:33 smithi044 ceph-mon[32404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T08:38:33.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:33 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:38:33.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:33 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2892132730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:33 smithi171 ceph-mon[37789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T08:38:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:33 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:38:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:33 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2892132730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:33 smithi116 ceph-mon[37097]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T08:38:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:33 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:38:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:33 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2892132730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 -- 172.21.15.44:0/3729221630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e8109990 msgr2=0x7f79e8109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 --2- 172.21.15.44:0/3729221630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e8109990 0x7f79e8109da0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009900 tx=0x7f79d802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 -- 172.21.15.44:0/3729221630 shutdown_connections 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 --2- 172.21.15.44:0/3729221630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e810c980 0x7f79e81163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 --2- 172.21.15.44:0/3729221630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c030 0x7f79e810c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 --2- 172.21.15.44:0/3729221630 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e8109990 0x7f79e8109da0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 -- 172.21.15.44:0/3729221630 >> 172.21.15.44:0/3729221630 conn(0x7f79e80790b0 msgr2=0x7f79e8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 -- 172.21.15.44:0/3729221630 shutdown_connections 2024-07-26T08:38:34.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.042+0000 7f79e7e00640 1 -- 172.21.15.44:0/3729221630 wait complete. 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.043+0000 7f79e7e00640 1 Processor -- start 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 -- start start 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e8109990 0x7f79e81a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e810c030 0x7f79e81a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 0x7f79e81a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8119880 con 0x7f79e8109990 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8119700 con 0x7f79e810c980 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.044+0000 7f79e7e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8119a00 con 0x7f79e810c030 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 0x7f79e81a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79dfe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e810c030 0x7f79e81a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 0x7f79e81a5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54592/0 (socket says 172.21.15.44:54592) 2024-07-26T08:38:34.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 -- 172.21.15.44:0/2015965645 learned_addr learned my addr 172.21.15.44:0/2015965645 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:34.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e6a00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e8109990 0x7f79e81a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 -- 172.21.15.44:0/2015965645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e810c030 msgr2=0x7f79e81a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e810c030 0x7f79e81a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 -- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e8109990 msgr2=0x7f79e81a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e8109990 0x7f79e81a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 -- 172.21.15.44:0/2015965645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e81a6170 con 0x7f79e810c980 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.045+0000 7f79e7400640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 0x7f79e81a5a70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f79e000efd0 tx=0x7f79e000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.046+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0003b70 con 0x7f79e810c980 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.046+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79e0003d10 con 0x7f79e810c980 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.046+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00108a0 con 0x7f79e810c980 2024-07-26T08:38:34.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.046+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e81a6460 con 0x7f79e810c980 2024-07-26T08:38:34.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.046+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e81acbf0 con 0x7f79e810c980 2024-07-26T08:38:34.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.047+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e8109da0 con 0x7f79e810c980 2024-07-26T08:38:34.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.049+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f79e0004020 con 0x7f79e810c980 2024-07-26T08:38:34.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.051+0000 7f79df400640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f79cc0b3ca0 0x7f79cc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.051+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f79e00e4510 con 0x7f79e810c980 2024-07-26T08:38:34.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.051+0000 7f79e6a00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f79cc0b3ca0 0x7f79cc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.052+0000 7f79e6a00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f79cc0b3ca0 0x7f79cc0b61b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f79d8031440 tx=0x7f79d8009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:34.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.055+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f79e00ab650 con 0x7f79e810c980 2024-07-26T08:38:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.269+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f79e802e0c0 con 0x7f79e810c980 2024-07-26T08:38:34.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.270+0000 7f79df400640 1 -- 172.21.15.44:0/2015965645 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f79e00aaf50 con 0x7f79e810c980 2024-07-26T08:38:34.272 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f79cc0b3ca0 msgr2=0x7f79cc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f79cc0b3ca0 0x7f79cc0b61b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f79d8031440 tx=0x7f79d8009560 comp rx=0 tx=0).stop 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 msgr2=0x7f79e81a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 0x7f79e81a5a70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f79e000efd0 tx=0x7f79e000c6a0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 shutdown_connections 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f79cc0b3ca0 0x7f79cc0b61b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f79e810c030 0x7f79e81a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79e810c980 0x7f79e81a5a70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 --2- 172.21.15.44:0/2015965645 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f79e8109990 0x7f79e81a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 >> 172.21.15.44:0/2015965645 conn(0x7f79e80790b0 msgr2=0x7f79e810d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:34.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.273+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 shutdown_connections 2024-07-26T08:38:34.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.274+0000 7f79e7e00640 1 -- 172.21.15.44:0/2015965645 wait complete. 2024-07-26T08:38:34.304 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 -- 172.21.15.44:0/1343454316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 msgr2=0x7f1a70105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 --2- 172.21.15.44:0/1343454316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a70105b10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a60009ca0 tx=0x7f1a6002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 -- 172.21.15.44:0/1343454316 shutdown_connections 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 --2- 172.21.15.44:0/1343454316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b030 0x7f1a7010b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 --2- 172.21.15.44:0/1343454316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b980 0x7f1a701121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 --2- 172.21.15.44:0/1343454316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a70105b10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 -- 172.21.15.44:0/1343454316 >> 172.21.15.44:0/1343454316 conn(0x7f1a70101120 msgr2=0x7f1a70103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:34.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 -- 172.21.15.44:0/1343454316 shutdown_connections 2024-07-26T08:38:34.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.592+0000 7f1a76400640 1 -- 172.21.15.44:0/1343454316 wait complete. 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.593+0000 7f1a76400640 1 Processor -- start 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 -- start start 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a7019c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 0x7f1a7019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b980 0x7f1a701a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1a701156a0 con 0x7f1a7010b030 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a70115520 con 0x7f1a70105700 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a76400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1a70115820 con 0x7f1a7010b980 2024-07-26T08:38:34.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a66a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 0x7f1a7019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a7019c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a66a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 0x7f1a7019c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49152/0 (socket says 172.21.15.44:49152) 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a66a00640 1 -- 172.21.15.44:0/1684222770 learned_addr learned my addr 172.21.15.44:0/1684222770 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.594+0000 7f1a6fe00640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b980 0x7f1a701a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6f400640 1 -- 172.21.15.44:0/1684222770 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b980 msgr2=0x7f1a701a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6f400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b980 0x7f1a701a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6f400640 1 -- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 msgr2=0x7f1a7019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6f400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 0x7f1a7019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6f400640 1 -- 172.21.15.44:0/1684222770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a701a1fb0 con 0x7f1a70105700 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a66a00640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 0x7f1a7019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6fe00640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b980 0x7f1a701a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:34.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.595+0000 7f1a6f400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a7019c080 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1a60000f80 tx=0x7f1a60031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:34.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.596+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60003b70 con 0x7f1a70105700 2024-07-26T08:38:34.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.596+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a60003d10 con 0x7f1a70105700 2024-07-26T08:38:34.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.596+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a701a2240 con 0x7f1a70105700 2024-07-26T08:38:34.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.596+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6002fa30 con 0x7f1a70105700 2024-07-26T08:38:34.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.596+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a701a8a30 con 0x7f1a70105700 2024-07-26T08:38:34.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.597+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a70105b10 con 0x7f1a70105700 2024-07-26T08:38:34.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.599+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f1a60004580 con 0x7f1a70105700 2024-07-26T08:38:34.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.601+0000 7f1a6cc00640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1a540b3ca0 0x7f1a540b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:34.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.601+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f1a601055d0 con 0x7f1a70105700 2024-07-26T08:38:34.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.601+0000 7f1a66a00640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1a540b3ca0 0x7f1a540b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:34.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.602+0000 7f1a66a00640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1a540b3ca0 0x7f1a540b61b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1a7010ab50 tx=0x7f1a58005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:34.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.602+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1a600cc710 con 0x7f1a70105700 2024-07-26T08:38:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:34 smithi044 ceph-mon[32404]: pgmap v479: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-07-26T08:38:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:34 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2400831202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T08:38:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:34 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2015965645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T08:38:34.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.815+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1a7010d760 con 0x7f1a70105700 2024-07-26T08:38:34.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.817+0000 7f1a6cc00640 1 -- 172.21.15.44:0/1684222770 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f1a600cc010 con 0x7f1a70105700 2024-07-26T08:38:34.818 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-07-26T08:38:34.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.819+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1a540b3ca0 msgr2=0x7f1a540b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.819+0000 7f1a76400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1a540b3ca0 0x7f1a540b61b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1a7010ab50 tx=0x7f1a58005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:34.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.819+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 msgr2=0x7f1a7019c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:34.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.819+0000 7f1a76400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a7019c080 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1a60000f80 tx=0x7f1a60031750 comp rx=0 tx=0).stop 2024-07-26T08:38:34.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 shutdown_connections 2024-07-26T08:38:34.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1a540b3ca0 0x7f1a540b61b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1a7010b980 0x7f1a701a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a7010b030 0x7f1a7019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 --2- 172.21.15.44:0/1684222770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a70105700 0x7f1a7019c080 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 >> 172.21.15.44:0/1684222770 conn(0x7f1a70101120 msgr2=0x7f1a70103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 shutdown_connections 2024-07-26T08:38:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:34.820+0000 7f1a76400640 1 -- 172.21.15.44:0/1684222770 wait complete. 2024-07-26T08:38:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:34 smithi171 ceph-mon[37789]: pgmap v479: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-07-26T08:38:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:34 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2400831202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T08:38:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:34 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2015965645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T08:38:34.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-07-26T08:38:34.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:34 smithi116 ceph-mon[37097]: pgmap v479: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-07-26T08:38:34.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:34 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2400831202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T08:38:34.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:34 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2015965645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T08:38:35.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 -- 172.21.15.44:0/675748783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0751e0 msgr2=0x7f8a7c0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:35.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 --2- 172.21.15.44:0/675748783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c0755f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009900 tx=0x7f8a7002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 -- 172.21.15.44:0/675748783 shutdown_connections 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 --2- 172.21.15.44:0/675748783 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c10f870 0x7f8a7c111cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 --2- 172.21.15.44:0/675748783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c075b30 0x7f8a7c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 --2- 172.21.15.44:0/675748783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c0755f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 -- 172.21.15.44:0/675748783 >> 172.21.15.44:0/675748783 conn(0x7f8a7c0fed60 msgr2=0x7f8a7c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 -- 172.21.15.44:0/675748783 shutdown_connections 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.126+0000 7f8a81e00640 1 -- 172.21.15.44:0/675748783 wait complete. 2024-07-26T08:38:35.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.127+0000 7f8a81e00640 1 Processor -- start 2024-07-26T08:38:35.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 -- start start 2024-07-26T08:38:35.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:35.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c075b30 0x7f8a7c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:35.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c10f870 0x7f8a7c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:35.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c115460 con 0x7f8a7c075b30 2024-07-26T08:38:35.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1152e0 con 0x7f8a7c0751e0 2024-07-26T08:38:35.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.128+0000 7f8a81e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1155e0 con 0x7f8a7c10f870 2024-07-26T08:38:35.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:35.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c075b30 0x7f8a7c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:35.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c19be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54626/0 (socket says 172.21.15.44:54626) 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 -- 172.21.15.44:0/511124627 learned_addr learned my addr 172.21.15.44:0/511124627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a81400640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c10f870 0x7f8a7c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 -- 172.21.15.44:0/511124627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c10f870 msgr2=0x7f8a7c1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c10f870 0x7f8a7c1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 -- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c075b30 msgr2=0x7f8a7c19c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c075b30 0x7f8a7c19c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a80a00640 1 -- 172.21.15.44:0/511124627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c1a1db0 con 0x7f8a7c0751e0 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.129+0000 7f8a7be00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c075b30 0x7f8a7c19c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.130+0000 7f8a80a00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c19be80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a70005940 tx=0x7f8a700040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.130+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7003f030 con 0x7f8a7c0751e0 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.130+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a700044c0 con 0x7f8a7c0751e0 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.130+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7c1a2040 con 0x7f8a7c0751e0 2024-07-26T08:38:35.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.130+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70035070 con 0x7f8a7c0751e0 2024-07-26T08:38:35.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.130+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7c1a8830 con 0x7f8a7c0751e0 2024-07-26T08:38:35.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.132+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a7c0766f0 con 0x7f8a7c0751e0 2024-07-26T08:38:35.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.134+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f8a70042030 con 0x7f8a7c0751e0 2024-07-26T08:38:35.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.135+0000 7f8a79600640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8a640b3c20 0x7f8a640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:35.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.136+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8a701046f0 con 0x7f8a7c0751e0 2024-07-26T08:38:35.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.136+0000 7f8a7be00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8a640b3c20 0x7f8a640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:35.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.137+0000 7f8a7be00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8a640b3c20 0x7f8a640b6130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0ffd90 tx=0x7f8a68005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:35.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.137+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8a700cb830 con 0x7f8a7c0751e0 2024-07-26T08:38:35.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.350+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8a7c1a8a00 con 0x7f8a7c0751e0 2024-07-26T08:38:35.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.351+0000 7f8a79600640 1 -- 172.21.15.44:0/511124627 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f8a700cb130 con 0x7f8a7c0751e0 2024-07-26T08:38:35.352 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-07-26T08:38:35.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8a640b3c20 msgr2=0x7f8a640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8a640b3c20 0x7f8a640b6130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0ffd90 tx=0x7f8a68005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 msgr2=0x7f8a7c19be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c19be80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a70005940 tx=0x7f8a700040c0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 shutdown_connections 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8a640b3c20 0x7f8a640b6130 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c10f870 0x7f8a7c1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a7c075b30 0x7f8a7c19c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 --2- 172.21.15.44:0/511124627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7c0751e0 0x7f8a7c19be80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 >> 172.21.15.44:0/511124627 conn(0x7f8a7c0fed60 msgr2=0x7f8a7c109410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.354+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 shutdown_connections 2024-07-26T08:38:35.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:35.355+0000 7f8a81e00640 1 -- 172.21.15.44:0/511124627 wait complete. 2024-07-26T08:38:35.387 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-07-26T08:38:35.388 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:38:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:35 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/1684222770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T08:38:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:35 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/511124627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T08:38:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:35 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/1684222770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T08:38:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:35 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/511124627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T08:38:35.848 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:35 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/1684222770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T08:38:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:35 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/511124627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T08:38:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:36 smithi044 ceph-mon[32404]: pgmap v480: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T08:38:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:36 smithi171 ceph-mon[37789]: pgmap v480: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T08:38:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:36 smithi116 ceph-mon[37097]: pgmap v480: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.274+0000 7f91c7200640 1 -- 172.21.15.44:0/3726480872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075ab0 msgr2=0x7f91c810f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.274+0000 7f91c7200640 1 --2- 172.21.15.44:0/3726480872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075ab0 0x7f91c810f460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009900 tx=0x7f91b402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.274+0000 7f91c7200640 1 -- 172.21.15.44:0/3726480872 shutdown_connections 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.274+0000 7f91c7200640 1 --2- 172.21.15.44:0/3726480872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c810f9a0 0x7f91c8111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.275+0000 7f91c7200640 1 --2- 172.21.15.44:0/3726480872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075ab0 0x7f91c810f460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.275+0000 7f91c7200640 1 --2- 172.21.15.44:0/3726480872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c8075160 0x7f91c8075570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.275+0000 7f91c7200640 1 -- 172.21.15.44:0/3726480872 >> 172.21.15.44:0/3726480872 conn(0x7f91c80fee10 msgr2=0x7f91c8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.275+0000 7f91c7200640 1 -- 172.21.15.44:0/3726480872 shutdown_connections 2024-07-26T08:38:38.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.275+0000 7f91c7200640 1 -- 172.21.15.44:0/3726480872 wait complete. 2024-07-26T08:38:38.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.276+0000 7f91c7200640 1 Processor -- start 2024-07-26T08:38:38.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.276+0000 7f91c7200640 1 -- start start 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c7200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075160 0x7f91c819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 0x7f91c819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c7200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c810f9a0 0x7f91c81a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c7200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f91c81154c0 con 0x7f91c8075160 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91c8115340 con 0x7f91c8075ab0 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c7200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f91c8115640 con 0x7f91c810f9a0 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 0x7f91c819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c5e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075160 0x7f91c819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 0x7f91c819c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54648/0 (socket says 172.21.15.44:54648) 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.277+0000 7f91c5400640 1 -- 172.21.15.44:0/875220808 learned_addr learned my addr 172.21.15.44:0/875220808 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:38.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c6800640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c810f9a0 0x7f91c81a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c5400640 1 -- 172.21.15.44:0/875220808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c810f9a0 msgr2=0x7f91c81a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c5400640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c810f9a0 0x7f91c81a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c5400640 1 -- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075160 msgr2=0x7f91c819bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:38.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c5400640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075160 0x7f91c819bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c5400640 1 -- 172.21.15.44:0/875220808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c81a1ea0 con 0x7f91c8075ab0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.278+0000 7f91c5400640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 0x7f91c819c4d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f91b4006010 tx=0x7f91b4004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.279+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b403f030 con 0x7f91c8075ab0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.279+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c81a2130 con 0x7f91c8075ab0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.279+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b4003aa0 con 0x7f91c8075ab0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.279+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b4035070 con 0x7f91c8075ab0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.279+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c81a88f0 con 0x7f91c8075ab0 2024-07-26T08:38:38.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.280+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c8075570 con 0x7f91c8075ab0 2024-07-26T08:38:38.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.283+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f91b4042030 con 0x7f91c8075ab0 2024-07-26T08:38:38.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.284+0000 7f91bea00640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91ac0b3c20 0x7f91ac0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.284+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f91b4104940 con 0x7f91c8075ab0 2024-07-26T08:38:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.285+0000 7f91c5e00640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91ac0b3c20 0x7f91ac0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:38.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.285+0000 7f91c5e00640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91ac0b3c20 0x7f91ac0b6130 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009bd0 tx=0x7f91b8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:38.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.286+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f91b40cb430 con 0x7f91c8075ab0 2024-07-26T08:38:38.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.529+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91c810ab20 con 0x7f91c8075ab0 2024-07-26T08:38:38.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.531+0000 7f91bea00640 1 -- 172.21.15.44:0/875220808 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7f91b40cf980 con 0x7f91c8075ab0 2024-07-26T08:38:38.533 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:38.534 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 30 2024-07-26T08:38:38.534 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":30,"btime":"2024-07-26T08:38:32:520674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:38:32.022353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24701,24686,24679,24668,24661]},"id":1}]} 2024-07-26T08:38:38.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91ac0b3c20 msgr2=0x7f91ac0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:38.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91ac0b3c20 0x7f91ac0b6130 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009bd0 tx=0x7f91b8005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 msgr2=0x7f91c819c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 0x7f91c819c4d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f91b4006010 tx=0x7f91b4004060 comp rx=0 tx=0).stop 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 shutdown_connections 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f91ac0b3c20 0x7f91ac0b6130 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91c810f9a0 0x7f91c81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91c8075ab0 0x7f91c819c4d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 --2- 172.21.15.44:0/875220808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f91c8075160 0x7f91c819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.535+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 >> 172.21.15.44:0/875220808 conn(0x7f91c80fee10 msgr2=0x7f91c80ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:38.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.536+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 shutdown_connections 2024-07-26T08:38:38.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:38.536+0000 7f91c7200640 1 -- 172.21.15.44:0/875220808 wait complete. 2024-07-26T08:38:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:38 smithi171 ceph-mon[37789]: pgmap v481: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T08:38:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:38 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/875220808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:38:38.943 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:38 smithi044 ceph-mon[32404]: pgmap v481: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T08:38:38.944 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:38 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/875220808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:38:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:38 smithi116 ceph-mon[37097]: pgmap v481: 129 pgs: 129 active+clean; 650 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T08:38:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:38 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/875220808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:38:39.621 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-07-26T08:38:39.621 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-07-26T08:38:39.622 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T08:38:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.919+0000 7f58d7200640 1 -- 172.21.15.44:0/2423820203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d810ae40 msgr2=0x7f58d810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.919+0000 7f58d7200640 1 --2- 172.21.15.44:0/2423820203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d810ae40 0x7f58d810b250 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009900 tx=0x7f58c002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 -- 172.21.15.44:0/2423820203 shutdown_connections 2024-07-26T08:38:39.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 --2- 172.21.15.44:0/2423820203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d810ae40 0x7f58d810b250 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 --2- 172.21.15.44:0/2423820203 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d810b790 0x7f58d8111fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 --2- 172.21.15.44:0/2423820203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d8105510 0x7f58d8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 -- 172.21.15.44:0/2423820203 >> 172.21.15.44:0/2423820203 conn(0x7f58d8100f90 msgr2=0x7f58d81033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 -- 172.21.15.44:0/2423820203 shutdown_connections 2024-07-26T08:38:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.920+0000 7f58d7200640 1 -- 172.21.15.44:0/2423820203 wait complete. 2024-07-26T08:38:39.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.921+0000 7f58d7200640 1 Processor -- start 2024-07-26T08:38:39.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.921+0000 7f58d7200640 1 -- start start 2024-07-26T08:38:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d7200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 0x7f58d819bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d810ae40 0x7f58d819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d7200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d810b790 0x7f58d81a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d7200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8115400 con 0x7f58d810b790 2024-07-26T08:38:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8115280 con 0x7f58d810ae40 2024-07-26T08:38:39.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d7200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8115580 con 0x7f58d8105510 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d5e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 0x7f58d819bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d6800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d810b790 0x7f58d81a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d5e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 0x7f58d819bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:42606/0 (socket says 172.21.15.44:42606) 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58cd400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d810ae40 0x7f58d819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.922+0000 7f58d5e00640 1 -- 172.21.15.44:0/2735841129 learned_addr learned my addr 172.21.15.44:0/2735841129 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d5e00640 1 -- 172.21.15.44:0/2735841129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d810ae40 msgr2=0x7f58d819c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d5e00640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d810ae40 0x7f58d819c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d5e00640 1 -- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d810b790 msgr2=0x7f58d81a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d5e00640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d810b790 0x7f58d81a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d5e00640 1 -- 172.21.15.44:0/2735841129 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d81a1e10 con 0x7f58d8105510 2024-07-26T08:38:39.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d5e00640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 0x7f58d819bee0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58c800ea60 tx=0x7f58c800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c800cde0 con 0x7f58d8105510 2024-07-26T08:38:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d81a2100 con 0x7f58d8105510 2024-07-26T08:38:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58c80035f0 con 0x7f58d8105510 2024-07-26T08:38:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.923+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8010690 con 0x7f58d8105510 2024-07-26T08:38:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.924+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d81a8890 con 0x7f58d8105510 2024-07-26T08:38:39.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.925+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d8105920 con 0x7f58d8105510 2024-07-26T08:38:39.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.927+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f58c80037a0 con 0x7f58d8105510 2024-07-26T08:38:39.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.929+0000 7f58cf400640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f58bc0b3c20 0x7f58bc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:39.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.929+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f58c80e4530 con 0x7f58d8105510 2024-07-26T08:38:39.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.929+0000 7f58cd400640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f58bc0b3c20 0x7f58bc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:39.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.930+0000 7f58cd400640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f58bc0b3c20 0x7f58bc0b6130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009230 tx=0x7f58c0031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:39.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:39.930+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f58c80ab670 con 0x7f58d8105510 2024-07-26T08:38:40.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.138+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58d8063230 con 0x7f58bc0b3c20 2024-07-26T08:38:40.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.150+0000 7f58cf400640 1 -- 172.21.15.44:0/2735841129 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340810 (secure 0 0 0) 0x7f58d8063230 con 0x7f58bc0b3c20 2024-07-26T08:38:40.157 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:40.160 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:38:40.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.157+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f58bc0b3c20 msgr2=0x7f58bc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.157+0000 7f58d7200640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f58bc0b3c20 0x7f58bc0b6130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009230 tx=0x7f58c0031400 comp rx=0 tx=0).stop 2024-07-26T08:38:40.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.157+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 msgr2=0x7f58d819bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.157+0000 7f58d7200640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 0x7f58d819bee0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f58c800ea60 tx=0x7f58c800ef20 comp rx=0 tx=0).stop 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 shutdown_connections 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f58bc0b3c20 0x7f58bc0b6130 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58d810b790 0x7f58d81a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58d810ae40 0x7f58d819c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 --2- 172.21.15.44:0/2735841129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f58d8105510 0x7f58d819bee0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 >> 172.21.15.44:0/2735841129 conn(0x7f58d8100f90 msgr2=0x7f58d81033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 shutdown_connections 2024-07-26T08:38:40.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.158+0000 7f58d7200640 1 -- 172.21.15.44:0/2735841129 wait complete. 2024-07-26T08:38:40.190 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-07-26T08:38:38.918170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671358,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2578,"num_read_kb":1408,"num_write":394,"num_write_kb":2319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":826,"ondisk_log_size":826,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365740,"kb_used_data":12428,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576084,"statfs":{"total":1151940427776,"available":1151565910016,"internally_reserved":0,"allocated":12726272,"data_stored":8458724,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":73,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006184"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":305,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.693455+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:38:34.693455+0000","last_peered":"2024-07-26T08:38:34.693455+0000","last_clean":"2024-07-26T08:38:34.693455+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:38:34.693455+0000","last_undegraded":"2024-07-26T08:38:34.693455+0000","last_fullsized":"2024-07-26T08:38:34.693455+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":245,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746837+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:36:12.746837+0000","last_peered":"2024-07-26T08:36:12.746837+0000","last_clean":"2024-07-26T08:36:12.746837+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:36:12.746837+0000","last_undegraded":"2024-07-26T08:36:12.746837+0000","last_fullsized":"2024-07-26T08:36:12.746837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:45.161843+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:35:45.161843+0000","last_peered":"2024-07-26T08:35:45.161843+0000","last_clean":"2024-07-26T08:35:45.161843+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:35:45.161843+0000","last_undegraded":"2024-07-26T08:35:45.161843+0000","last_fullsized":"2024-07-26T08:35:45.161843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:57.270075+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:57.270075+0000","last_peered":"2024-07-26T08:34:57.270075+0000","last_clean":"2024-07-26T08:34:57.270075+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:57.270075+0000","last_undegraded":"2024-07-26T08:34:57.270075+0000","last_fullsized":"2024-07-26T08:34:57.270075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741337+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:36:12.741337+0000","last_peered":"2024-07-26T08:36:12.741337+0000","last_clean":"2024-07-26T08:36:12.741337+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:36:12.741337+0000","last_undegraded":"2024-07-26T08:36:12.741337+0000","last_fullsized":"2024-07-26T08:36:12.741337+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:43.579910+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:35:43.579910+0000","last_peered":"2024-07-26T08:35:43.579910+0000","last_clean":"2024-07-26T08:35:43.579910+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:35:43.579910+0000","last_undegraded":"2024-07-26T08:35:43.579910+0000","last_fullsized":"2024-07-26T08:35:43.579910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:30.595827+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:35:30.595827+0000","last_peered":"2024-07-26T08:35:30.595827+0000","last_clean":"2024-07-26T08:35:30.595827+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:35:30.595827+0000","last_undegraded":"2024-07-26T08:35:30.595827+0000","last_fullsized":"2024-07-26T08:35:30.595827+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741380+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:36:12.741380+0000","last_peered":"2024-07-26T08:36:12.741380+0000","last_clean":"2024-07-26T08:36:12.741380+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:36:12.741380+0000","last_undegraded":"2024-07-26T08:36:12.741380+0000","last_fullsized":"2024-07-26T08:36:12.741380+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:48.580551+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:35:48.580551+0000","last_peered":"2024-07-26T08:35:48.580551+0000","last_clean":"2024-07-26T08:35:48.580551+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:35:48.580551+0000","last_undegraded":"2024-07-26T08:35:48.580551+0000","last_fullsized":"2024-07-26T08:35:48.580551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:30.595776+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:35:30.595776+0000","last_peered":"2024-07-26T08:35:30.595776+0000","last_clean":"2024-07-26T08:35:30.595776+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:35:30.595776+0000","last_undegraded":"2024-07-26T08:35:30.595776+0000","last_fullsized":"2024-07-26T08:35:30.595776+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734806+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:36:12.734806+0000","last_peered":"2024-07-26T08:36:12.734806+0000","last_clean":"2024-07-26T08:36:12.734806+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:36:12.734806+0000","last_undegraded":"2024-07-26T08:36:12.734806+0000","last_fullsized":"2024-07-26T08:36:12.734806+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:48.580530+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:35:48.580530+0000","last_peered":"2024-07-26T08:35:48.580530+0000","last_clean":"2024-07-26T08:35:48.580530+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:35:48.580530+0000","last_undegraded":"2024-07-26T08:35:48.580530+0000","last_fullsized":"2024-07-26T08:35:48.580530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742218+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:36:12.742218+0000","last_peered":"2024-07-26T08:36:12.742218+0000","last_clean":"2024-07-26T08:36:12.742218+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:36:12.742218+0000","last_undegraded":"2024-07-26T08:36:12.742218+0000","last_fullsized":"2024-07-26T08:36:12.742218+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:15.594066+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:35:15.594066+0000","last_peered":"2024-07-26T08:35:15.594066+0000","last_clean":"2024-07-26T08:35:15.594066+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:35:15.594066+0000","last_undegraded":"2024-07-26T08:35:15.594066+0000","last_fullsized":"2024-07-26T08:35:15.594066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578665+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:35:33.578665+0000","last_peered":"2024-07-26T08:35:33.578665+0000","last_clean":"2024-07-26T08:35:33.578665+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:35:33.578665+0000","last_undegraded":"2024-07-26T08:35:33.578665+0000","last_fullsized":"2024-07-26T08:35:33.578665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744388+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:36:12.744388+0000","last_peered":"2024-07-26T08:36:12.744388+0000","last_clean":"2024-07-26T08:36:12.744388+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:36:12.744388+0000","last_undegraded":"2024-07-26T08:36:12.744388+0000","last_fullsized":"2024-07-26T08:36:12.744388+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320730+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:35:26.320730+0000","last_peered":"2024-07-26T08:35:26.320730+0000","last_clean":"2024-07-26T08:35:26.320730+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:35:26.320730+0000","last_undegraded":"2024-07-26T08:35:26.320730+0000","last_fullsized":"2024-07-26T08:35:26.320730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578615+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:35:33.578615+0000","last_peered":"2024-07-26T08:35:33.578615+0000","last_clean":"2024-07-26T08:35:33.578615+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:35:33.578615+0000","last_undegraded":"2024-07-26T08:35:33.578615+0000","last_fullsized":"2024-07-26T08:35:33.578615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740319+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:36:12.740319+0000","last_peered":"2024-07-26T08:36:12.740319+0000","last_clean":"2024-07-26T08:36:12.740319+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:36:12.740319+0000","last_undegraded":"2024-07-26T08:36:12.740319+0000","last_fullsized":"2024-07-26T08:36:12.740319+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.272221+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:35:07.272221+0000","last_peered":"2024-07-26T08:35:07.272221+0000","last_clean":"2024-07-26T08:35:07.272221+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:35:07.272221+0000","last_undegraded":"2024-07-26T08:35:07.272221+0000","last_fullsized":"2024-07-26T08:35:07.272221+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:36.345867+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:35:36.345867+0000","last_peered":"2024-07-26T08:35:36.345867+0000","last_clean":"2024-07-26T08:35:36.345867+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:35:36.345867+0000","last_undegraded":"2024-07-26T08:35:36.345867+0000","last_fullsized":"2024-07-26T08:35:36.345867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.712601+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:36:12.712601+0000","last_peered":"2024-07-26T08:36:12.712601+0000","last_clean":"2024-07-26T08:36:12.712601+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:36:12.712601+0000","last_undegraded":"2024-07-26T08:36:12.712601+0000","last_fullsized":"2024-07-26T08:36:12.712601+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.548672+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:35:07.548672+0000","last_peered":"2024-07-26T08:35:07.548672+0000","last_clean":"2024-07-26T08:35:07.548672+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:35:07.548672+0000","last_undegraded":"2024-07-26T08:35:07.548672+0000","last_fullsized":"2024-07-26T08:35:07.548672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578686+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:35:33.578686+0000","last_peered":"2024-07-26T08:35:33.578686+0000","last_clean":"2024-07-26T08:35:33.578686+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:35:33.578686+0000","last_undegraded":"2024-07-26T08:35:33.578686+0000","last_fullsized":"2024-07-26T08:35:33.578686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.747319+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:36:12.747319+0000","last_peered":"2024-07-26T08:36:12.747319+0000","last_clean":"2024-07-26T08:36:12.747319+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:36:12.747319+0000","last_undegraded":"2024-07-26T08:36:12.747319+0000","last_fullsized":"2024-07-26T08:36:12.747319+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:08.575512+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:35:08.575512+0000","last_peered":"2024-07-26T08:35:08.575512+0000","last_clean":"2024-07-26T08:35:08.575512+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:35:08.575512+0000","last_undegraded":"2024-07-26T08:35:08.575512+0000","last_fullsized":"2024-07-26T08:35:08.575512+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:25.159325+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:35:25.159325+0000","last_peered":"2024-07-26T08:35:25.159325+0000","last_clean":"2024-07-26T08:35:25.159325+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:35:25.159325+0000","last_undegraded":"2024-07-26T08:35:25.159325+0000","last_fullsized":"2024-07-26T08:35:25.159325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728535+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:36:12.728535+0000","last_peered":"2024-07-26T08:36:12.728535+0000","last_clean":"2024-07-26T08:36:12.728535+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:36:12.728535+0000","last_undegraded":"2024-07-26T08:36:12.728535+0000","last_fullsized":"2024-07-26T08:36:12.728535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'18","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:37:21.364570+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:37:21.364570+0000","last_peered":"2024-07-26T08:37:21.364570+0000","last_clean":"2024-07-26T08:37:21.364570+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:37:21.364570+0000","last_undegraded":"2024-07-26T08:37:21.364570+0000","last_fullsized":"2024-07-26T08:37:21.364570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:17.274168+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:35:17.274168+0000","last_peered":"2024-07-26T08:35:17.274168+0000","last_clean":"2024-07-26T08:35:17.274168+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:35:17.274168+0000","last_undegraded":"2024-07-26T08:35:17.274168+0000","last_fullsized":"2024-07-26T08:35:17.274168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734147+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:36:12.734147+0000","last_peered":"2024-07-26T08:36:12.734147+0000","last_clean":"2024-07-26T08:36:12.734147+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:36:12.734147+0000","last_undegraded":"2024-07-26T08:36:12.734147+0000","last_fullsized":"2024-07-26T08:36:12.734147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:05.592648+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:35:05.592648+0000","last_peered":"2024-07-26T08:35:05.592648+0000","last_clean":"2024-07-26T08:35:05.592648+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:35:05.592648+0000","last_undegraded":"2024-07-26T08:35:05.592648+0000","last_fullsized":"2024-07-26T08:35:05.592648+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320776+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:35:26.320776+0000","last_peered":"2024-07-26T08:35:26.320776+0000","last_clean":"2024-07-26T08:35:26.320776+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:35:26.320776+0000","last_undegraded":"2024-07-26T08:35:26.320776+0000","last_fullsized":"2024-07-26T08:35:26.320776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744052+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:36:12.744052+0000","last_peered":"2024-07-26T08:36:12.744052+0000","last_clean":"2024-07-26T08:36:12.744052+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:36:12.744052+0000","last_undegraded":"2024-07-26T08:36:12.744052+0000","last_fullsized":"2024-07-26T08:36:12.744052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:06.339791+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:35:06.339791+0000","last_peered":"2024-07-26T08:35:06.339791+0000","last_clean":"2024-07-26T08:35:06.339791+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:35:06.339791+0000","last_undegraded":"2024-07-26T08:35:06.339791+0000","last_fullsized":"2024-07-26T08:35:06.339791+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:25.595196+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:35:25.595196+0000","last_peered":"2024-07-26T08:35:25.595196+0000","last_clean":"2024-07-26T08:35:25.595196+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:35:25.595196+0000","last_undegraded":"2024-07-26T08:35:25.595196+0000","last_fullsized":"2024-07-26T08:35:25.595196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740048+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:36:12.740048+0000","last_peered":"2024-07-26T08:36:12.740048+0000","last_clean":"2024-07-26T08:36:12.740048+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:36:12.740048+0000","last_undegraded":"2024-07-26T08:36:12.740048+0000","last_fullsized":"2024-07-26T08:36:12.740048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:35.696701+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:38:35.696701+0000","last_peered":"2024-07-26T08:38:35.696701+0000","last_clean":"2024-07-26T08:38:35.696701+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:38:35.696701+0000","last_undegraded":"2024-07-26T08:38:35.696701+0000","last_fullsized":"2024-07-26T08:38:35.696701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":246,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.566680+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:35:26.566680+0000","last_peered":"2024-07-26T08:35:26.566680+0000","last_clean":"2024-07-26T08:35:26.566680+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:35:26.566680+0000","last_undegraded":"2024-07-26T08:35:26.566680+0000","last_fullsized":"2024-07-26T08:35:26.566680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729599+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:36:12.729599+0000","last_peered":"2024-07-26T08:36:12.729599+0000","last_clean":"2024-07-26T08:36:12.729599+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:36:12.729599+0000","last_undegraded":"2024-07-26T08:36:12.729599+0000","last_fullsized":"2024-07-26T08:36:12.729599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.779037+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:38:34.779037+0000","last_peered":"2024-07-26T08:38:34.779037+0000","last_clean":"2024-07-26T08:38:34.779037+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:38:34.779037+0000","last_undegraded":"2024-07-26T08:38:34.779037+0000","last_fullsized":"2024-07-26T08:38:34.779037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":245,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.567573+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:35:31.567573+0000","last_peered":"2024-07-26T08:35:31.567573+0000","last_clean":"2024-07-26T08:35:31.567573+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:35:31.567573+0000","last_undegraded":"2024-07-26T08:35:31.567573+0000","last_fullsized":"2024-07-26T08:35:31.567573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733996+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:36:12.733996+0000","last_peered":"2024-07-26T08:36:12.733996+0000","last_clean":"2024-07-26T08:36:12.733996+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:36:12.733996+0000","last_undegraded":"2024-07-26T08:36:12.733996+0000","last_fullsized":"2024-07-26T08:36:12.733996+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:57.422284+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:57.422284+0000","last_peered":"2024-07-26T08:34:57.422284+0000","last_clean":"2024-07-26T08:34:57.422284+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:57.422284+0000","last_undegraded":"2024-07-26T08:34:57.422284+0000","last_fullsized":"2024-07-26T08:34:57.422284+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406163+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:35:31.406163+0000","last_peered":"2024-07-26T08:35:31.406163+0000","last_clean":"2024-07-26T08:35:31.406163+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:35:31.406163+0000","last_undegraded":"2024-07-26T08:35:31.406163+0000","last_fullsized":"2024-07-26T08:35:31.406163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745276+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:36:12.745276+0000","last_peered":"2024-07-26T08:36:12.745276+0000","last_clean":"2024-07-26T08:36:12.745276+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:36:12.745276+0000","last_undegraded":"2024-07-26T08:36:12.745276+0000","last_fullsized":"2024-07-26T08:36:12.745276+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:03.574928+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:35:03.574928+0000","last_peered":"2024-07-26T08:35:03.574928+0000","last_clean":"2024-07-26T08:35:03.574928+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:35:03.574928+0000","last_undegraded":"2024-07-26T08:35:03.574928+0000","last_fullsized":"2024-07-26T08:35:03.574928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:07.430772+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:36:07.430772+0000","last_peered":"2024-07-26T08:36:07.430772+0000","last_clean":"2024-07-26T08:36:07.430772+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:36:07.430772+0000","last_undegraded":"2024-07-26T08:36:07.430772+0000","last_fullsized":"2024-07-26T08:36:07.430772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:22.425289+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:35:22.425289+0000","last_peered":"2024-07-26T08:35:22.425289+0000","last_clean":"2024-07-26T08:35:22.425289+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:35:22.425289+0000","last_undegraded":"2024-07-26T08:35:22.425289+0000","last_fullsized":"2024-07-26T08:35:22.425289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742063+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:36:12.742063+0000","last_peered":"2024-07-26T08:36:12.742063+0000","last_clean":"2024-07-26T08:36:12.742063+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:36:12.742063+0000","last_undegraded":"2024-07-26T08:36:12.742063+0000","last_fullsized":"2024-07-26T08:36:12.742063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":301,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:33.496342+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:38:33.496342+0000","last_peered":"2024-07-26T08:38:33.496342+0000","last_clean":"2024-07-26T08:38:33.496342+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:38:33.496342+0000","last_undegraded":"2024-07-26T08:38:33.496342+0000","last_fullsized":"2024-07-26T08:38:33.496342+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":247,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:27.551514+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:35:27.551514+0000","last_peered":"2024-07-26T08:35:27.551514+0000","last_clean":"2024-07-26T08:35:27.551514+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:35:27.551514+0000","last_undegraded":"2024-07-26T08:35:27.551514+0000","last_fullsized":"2024-07-26T08:35:27.551514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744223+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:36:12.744223+0000","last_peered":"2024-07-26T08:36:12.744223+0000","last_clean":"2024-07-26T08:36:12.744223+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:36:12.744223+0000","last_undegraded":"2024-07-26T08:36:12.744223+0000","last_fullsized":"2024-07-26T08:36:12.744223+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:16.404127+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:35:16.404127+0000","last_peered":"2024-07-26T08:35:16.404127+0000","last_clean":"2024-07-26T08:35:16.404127+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:35:16.404127+0000","last_undegraded":"2024-07-26T08:35:16.404127+0000","last_fullsized":"2024-07-26T08:35:16.404127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:32.864361+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:35:32.864361+0000","last_peered":"2024-07-26T08:35:32.864361+0000","last_clean":"2024-07-26T08:35:32.864361+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:35:32.864361+0000","last_undegraded":"2024-07-26T08:35:32.864361+0000","last_fullsized":"2024-07-26T08:35:32.864361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727635+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:36:12.727635+0000","last_peered":"2024-07-26T08:36:12.727635+0000","last_clean":"2024-07-26T08:36:12.727635+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:36:12.727635+0000","last_undegraded":"2024-07-26T08:36:12.727635+0000","last_fullsized":"2024-07-26T08:36:12.727635+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:16.565096+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:35:16.565096+0000","last_peered":"2024-07-26T08:35:16.565096+0000","last_clean":"2024-07-26T08:35:16.565096+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:35:16.565096+0000","last_undegraded":"2024-07-26T08:35:16.565096+0000","last_fullsized":"2024-07-26T08:35:16.565096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406097+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:35:31.406097+0000","last_peered":"2024-07-26T08:35:31.406097+0000","last_clean":"2024-07-26T08:35:31.406097+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:35:31.406097+0000","last_undegraded":"2024-07-26T08:35:31.406097+0000","last_fullsized":"2024-07-26T08:35:31.406097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.722492+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:36:12.722492+0000","last_peered":"2024-07-26T08:36:12.722492+0000","last_clean":"2024-07-26T08:36:12.722492+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:36:12.722492+0000","last_undegraded":"2024-07-26T08:36:12.722492+0000","last_fullsized":"2024-07-26T08:36:12.722492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:15.594107+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:35:15.594107+0000","last_peered":"2024-07-26T08:35:15.594107+0000","last_clean":"2024-07-26T08:35:15.594107+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:35:15.594107+0000","last_undegraded":"2024-07-26T08:35:15.594107+0000","last_fullsized":"2024-07-26T08:35:15.594107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406127+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:35:31.406127+0000","last_peered":"2024-07-26T08:35:31.406127+0000","last_clean":"2024-07-26T08:35:31.406127+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:35:31.406127+0000","last_undegraded":"2024-07-26T08:35:31.406127+0000","last_fullsized":"2024-07-26T08:35:31.406127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733854+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:36:12.733854+0000","last_peered":"2024-07-26T08:36:12.733854+0000","last_clean":"2024-07-26T08:36:12.733854+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:36:12.733854+0000","last_undegraded":"2024-07-26T08:36:12.733854+0000","last_fullsized":"2024-07-26T08:36:12.733854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108992+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:32.108992+0000","last_peered":"2024-07-26T08:34:32.108992+0000","last_clean":"2024-07-26T08:34:32.108992+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:32.108992+0000","last_undegraded":"2024-07-26T08:34:32.108992+0000","last_fullsized":"2024-07-26T08:34:32.108992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:46.570284+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:35:46.570284+0000","last_peered":"2024-07-26T08:35:46.570284+0000","last_clean":"2024-07-26T08:35:46.570284+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:35:46.570284+0000","last_undegraded":"2024-07-26T08:35:46.570284+0000","last_fullsized":"2024-07-26T08:35:46.570284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.731664+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:36:12.731664+0000","last_peered":"2024-07-26T08:36:12.731664+0000","last_clean":"2024-07-26T08:36:12.731664+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:36:12.731664+0000","last_undegraded":"2024-07-26T08:36:12.731664+0000","last_fullsized":"2024-07-26T08:36:12.731664+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'8","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:36.784851+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:38:36.784851+0000","last_peered":"2024-07-26T08:38:36.784851+0000","last_clean":"2024-07-26T08:38:36.784851+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:38:36.784851+0000","last_undegraded":"2024-07-26T08:38:36.784851+0000","last_fullsized":"2024-07-26T08:38:36.784851+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1427,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:46.348069+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:35:46.348069+0000","last_peered":"2024-07-26T08:35:46.348069+0000","last_clean":"2024-07-26T08:35:46.348069+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:35:46.348069+0000","last_undegraded":"2024-07-26T08:35:46.348069+0000","last_fullsized":"2024-07-26T08:35:46.348069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734066+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:36:12.734066+0000","last_peered":"2024-07-26T08:36:12.734066+0000","last_clean":"2024-07-26T08:36:12.734066+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:36:12.734066+0000","last_undegraded":"2024-07-26T08:36:12.734066+0000","last_fullsized":"2024-07-26T08:36:12.734066+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:21.404762+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:35:21.404762+0000","last_peered":"2024-07-26T08:35:21.404762+0000","last_clean":"2024-07-26T08:35:21.404762+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:35:21.404762+0000","last_undegraded":"2024-07-26T08:35:21.404762+0000","last_fullsized":"2024-07-26T08:35:21.404762+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.322567+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:35:41.322567+0000","last_peered":"2024-07-26T08:35:41.322567+0000","last_clean":"2024-07-26T08:35:41.322567+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:35:41.322567+0000","last_undegraded":"2024-07-26T08:35:41.322567+0000","last_fullsized":"2024-07-26T08:35:41.322567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741015+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:36:12.741015+0000","last_peered":"2024-07-26T08:36:12.741015+0000","last_clean":"2024-07-26T08:36:12.741015+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:36:12.741015+0000","last_undegraded":"2024-07-26T08:36:12.741015+0000","last_fullsized":"2024-07-26T08:36:12.741015+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:20.158778+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:35:20.158778+0000","last_peered":"2024-07-26T08:35:20.158778+0000","last_clean":"2024-07-26T08:35:20.158778+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:35:20.158778+0000","last_undegraded":"2024-07-26T08:35:20.158778+0000","last_fullsized":"2024-07-26T08:35:20.158778+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:45.923456+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:35:45.923456+0000","last_peered":"2024-07-26T08:35:45.923456+0000","last_clean":"2024-07-26T08:35:45.923456+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:35:45.923456+0000","last_undegraded":"2024-07-26T08:35:45.923456+0000","last_fullsized":"2024-07-26T08:35:45.923456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.717243+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:36:12.717243+0000","last_peered":"2024-07-26T08:36:12.717243+0000","last_clean":"2024-07-26T08:36:12.717243+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:36:12.717243+0000","last_undegraded":"2024-07-26T08:36:12.717243+0000","last_fullsized":"2024-07-26T08:36:12.717243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":595,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:37.502224+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:38:37.502224+0000","last_peered":"2024-07-26T08:38:37.502224+0000","last_clean":"2024-07-26T08:38:37.502224+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:38:37.502224+0000","last_undegraded":"2024-07-26T08:38:37.502224+0000","last_fullsized":"2024-07-26T08:38:37.502224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72963,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":504,"num_read_kb":324,"num_write":46,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.346929+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:35:41.346929+0000","last_peered":"2024-07-26T08:35:41.346929+0000","last_clean":"2024-07-26T08:35:41.346929+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:35:41.346929+0000","last_undegraded":"2024-07-26T08:35:41.346929+0000","last_fullsized":"2024-07-26T08:35:41.346929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738676+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:36:12.738676+0000","last_peered":"2024-07-26T08:36:12.738676+0000","last_clean":"2024-07-26T08:36:12.738676+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:36:12.738676+0000","last_undegraded":"2024-07-26T08:36:12.738676+0000","last_fullsized":"2024-07-26T08:36:12.738676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.405488+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:35:26.405488+0000","last_peered":"2024-07-26T08:35:26.405488+0000","last_clean":"2024-07-26T08:35:26.405488+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:35:26.405488+0000","last_undegraded":"2024-07-26T08:35:26.405488+0000","last_fullsized":"2024-07-26T08:35:26.405488+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:38.579362+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:35:38.579362+0000","last_peered":"2024-07-26T08:35:38.579362+0000","last_clean":"2024-07-26T08:35:38.579362+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:35:38.579362+0000","last_undegraded":"2024-07-26T08:35:38.579362+0000","last_fullsized":"2024-07-26T08:35:38.579362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740282+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:36:12.740282+0000","last_peered":"2024-07-26T08:36:12.740282+0000","last_clean":"2024-07-26T08:36:12.740282+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:36:12.740282+0000","last_undegraded":"2024-07-26T08:36:12.740282+0000","last_fullsized":"2024-07-26T08:36:12.740282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:22.425306+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:35:22.425306+0000","last_peered":"2024-07-26T08:35:22.425306+0000","last_clean":"2024-07-26T08:35:22.425306+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:35:22.425306+0000","last_undegraded":"2024-07-26T08:35:22.425306+0000","last_fullsized":"2024-07-26T08:35:22.425306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.346745+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:35:41.346745+0000","last_peered":"2024-07-26T08:35:41.346745+0000","last_clean":"2024-07-26T08:35:41.346745+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:35:41.346745+0000","last_undegraded":"2024-07-26T08:35:41.346745+0000","last_fullsized":"2024-07-26T08:35:41.346745+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741135+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:36:12.741135+0000","last_peered":"2024-07-26T08:36:12.741135+0000","last_clean":"2024-07-26T08:36:12.741135+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:36:12.741135+0000","last_undegraded":"2024-07-26T08:36:12.741135+0000","last_fullsized":"2024-07-26T08:36:12.741135+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":544,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:37.501908+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:38:37.501908+0000","last_peered":"2024-07-26T08:38:37.501908+0000","last_clean":"2024-07-26T08:38:37.501908+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:38:37.501908+0000","last_undegraded":"2024-07-26T08:38:37.501908+0000","last_fullsized":"2024-07-26T08:38:37.501908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":492,"num_read_kb":492,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:37.278601+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:35:37.278601+0000","last_peered":"2024-07-26T08:35:37.278601+0000","last_clean":"2024-07-26T08:35:37.278601+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:35:37.278601+0000","last_undegraded":"2024-07-26T08:35:37.278601+0000","last_fullsized":"2024-07-26T08:35:37.278601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:17.861977+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:35:17.861977+0000","last_peered":"2024-07-26T08:35:17.861977+0000","last_clean":"2024-07-26T08:35:17.861977+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:35:17.861977+0000","last_undegraded":"2024-07-26T08:35:17.861977+0000","last_fullsized":"2024-07-26T08:35:17.861977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746413+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:36:12.746413+0000","last_peered":"2024-07-26T08:36:12.746413+0000","last_clean":"2024-07-26T08:36:12.746413+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:36:12.746413+0000","last_undegraded":"2024-07-26T08:36:12.746413+0000","last_fullsized":"2024-07-26T08:36:12.746413+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.405429+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:35:26.405429+0000","last_peered":"2024-07-26T08:35:26.405429+0000","last_clean":"2024-07-26T08:35:26.405429+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:35:26.405429+0000","last_undegraded":"2024-07-26T08:35:26.405429+0000","last_fullsized":"2024-07-26T08:35:26.405429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320795+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:35:26.320795+0000","last_peered":"2024-07-26T08:35:26.320795+0000","last_clean":"2024-07-26T08:35:26.320795+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:35:26.320795+0000","last_undegraded":"2024-07-26T08:35:26.320795+0000","last_fullsized":"2024-07-26T08:35:26.320795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741858+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:36:12.741858+0000","last_peered":"2024-07-26T08:36:12.741858+0000","last_clean":"2024-07-26T08:36:12.741858+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:36:12.741858+0000","last_undegraded":"2024-07-26T08:36:12.741858+0000","last_fullsized":"2024-07-26T08:36:12.741858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.566623+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:35:26.566623+0000","last_peered":"2024-07-26T08:35:26.566623+0000","last_clean":"2024-07-26T08:35:26.566623+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:35:26.566623+0000","last_undegraded":"2024-07-26T08:35:26.566623+0000","last_fullsized":"2024-07-26T08:35:26.566623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:27.846852+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:35:27.846852+0000","last_peered":"2024-07-26T08:35:27.846852+0000","last_clean":"2024-07-26T08:35:27.846852+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:35:27.846852+0000","last_undegraded":"2024-07-26T08:35:27.846852+0000","last_fullsized":"2024-07-26T08:35:27.846852+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740942+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:36:12.740942+0000","last_peered":"2024-07-26T08:36:12.740942+0000","last_clean":"2024-07-26T08:36:12.740942+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:36:12.740942+0000","last_undegraded":"2024-07-26T08:36:12.740942+0000","last_fullsized":"2024-07-26T08:36:12.740942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:50.162384+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:35:50.162384+0000","last_peered":"2024-07-26T08:35:50.162384+0000","last_clean":"2024-07-26T08:35:50.162384+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:35:50.162384+0000","last_undegraded":"2024-07-26T08:35:50.162384+0000","last_fullsized":"2024-07-26T08:35:50.162384+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.497991+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:38:34.497991+0000","last_peered":"2024-07-26T08:38:34.497991+0000","last_clean":"2024-07-26T08:38:34.497991+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:38:34.497991+0000","last_undegraded":"2024-07-26T08:38:34.497991+0000","last_fullsized":"2024-07-26T08:38:34.497991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":246,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727350+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:36:12.727350+0000","last_peered":"2024-07-26T08:36:12.727350+0000","last_clean":"2024-07-26T08:36:12.727350+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:36:12.727350+0000","last_undegraded":"2024-07-26T08:36:12.727350+0000","last_fullsized":"2024-07-26T08:36:12.727350+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:50.162409+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:35:50.162409+0000","last_peered":"2024-07-26T08:35:50.162409+0000","last_clean":"2024-07-26T08:35:50.162409+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:35:50.162409+0000","last_undegraded":"2024-07-26T08:35:50.162409+0000","last_fullsized":"2024-07-26T08:35:50.162409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723745+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:36:12.723745+0000","last_peered":"2024-07-26T08:36:12.723745+0000","last_clean":"2024-07-26T08:36:12.723745+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:36:12.723745+0000","last_undegraded":"2024-07-26T08:36:12.723745+0000","last_fullsized":"2024-07-26T08:36:12.723745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.722200+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:36:12.722200+0000","last_peered":"2024-07-26T08:36:12.722200+0000","last_clean":"2024-07-26T08:36:12.722200+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:36:12.722200+0000","last_undegraded":"2024-07-26T08:36:12.722200+0000","last_fullsized":"2024-07-26T08:36:12.722200+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737483+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:36:12.737483+0000","last_peered":"2024-07-26T08:36:12.737483+0000","last_clean":"2024-07-26T08:36:12.737483+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:36:12.737483+0000","last_undegraded":"2024-07-26T08:36:12.737483+0000","last_fullsized":"2024-07-26T08:36:12.737483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729462+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:36:12.729462+0000","last_peered":"2024-07-26T08:36:12.729462+0000","last_clean":"2024-07-26T08:36:12.729462+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:36:12.729462+0000","last_undegraded":"2024-07-26T08:36:12.729462+0000","last_fullsized":"2024-07-26T08:36:12.729462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742274+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:36:12.742274+0000","last_peered":"2024-07-26T08:36:12.742274+0000","last_clean":"2024-07-26T08:36:12.742274+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:36:12.742274+0000","last_undegraded":"2024-07-26T08:36:12.742274+0000","last_fullsized":"2024-07-26T08:36:12.742274+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718573+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:36:12.718573+0000","last_peered":"2024-07-26T08:36:12.718573+0000","last_clean":"2024-07-26T08:36:12.718573+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:36:12.718573+0000","last_undegraded":"2024-07-26T08:36:12.718573+0000","last_fullsized":"2024-07-26T08:36:12.718573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.732735+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:36:12.732735+0000","last_peered":"2024-07-26T08:36:12.732735+0000","last_clean":"2024-07-26T08:36:12.732735+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:36:12.732735+0000","last_undegraded":"2024-07-26T08:36:12.732735+0000","last_fullsized":"2024-07-26T08:36:12.732735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745906+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:36:12.745906+0000","last_peered":"2024-07-26T08:36:12.745906+0000","last_clean":"2024-07-26T08:36:12.745906+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:36:12.745906+0000","last_undegraded":"2024-07-26T08:36:12.745906+0000","last_fullsized":"2024-07-26T08:36:12.745906+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746340+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:36:12.746340+0000","last_peered":"2024-07-26T08:36:12.746340+0000","last_clean":"2024-07-26T08:36:12.746340+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:36:12.746340+0000","last_undegraded":"2024-07-26T08:36:12.746340+0000","last_fullsized":"2024-07-26T08:36:12.746340+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742178+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:36:12.742178+0000","last_peered":"2024-07-26T08:36:12.742178+0000","last_clean":"2024-07-26T08:36:12.742178+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:36:12.742178+0000","last_undegraded":"2024-07-26T08:36:12.742178+0000","last_fullsized":"2024-07-26T08:36:12.742178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746534+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:36:12.746534+0000","last_peered":"2024-07-26T08:36:12.746534+0000","last_clean":"2024-07-26T08:36:12.746534+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:36:12.746534+0000","last_undegraded":"2024-07-26T08:36:12.746534+0000","last_fullsized":"2024-07-26T08:36:12.746534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742408+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:36:12.742408+0000","last_peered":"2024-07-26T08:36:12.742408+0000","last_clean":"2024-07-26T08:36:12.742408+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:36:12.742408+0000","last_undegraded":"2024-07-26T08:36:12.742408+0000","last_fullsized":"2024-07-26T08:36:12.742408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723201+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:36:12.723201+0000","last_peered":"2024-07-26T08:36:12.723201+0000","last_clean":"2024-07-26T08:36:12.723201+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:36:12.723201+0000","last_undegraded":"2024-07-26T08:36:12.723201+0000","last_fullsized":"2024-07-26T08:36:12.723201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736113+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:36:12.736113+0000","last_peered":"2024-07-26T08:36:12.736113+0000","last_clean":"2024-07-26T08:36:12.736113+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:36:12.736113+0000","last_undegraded":"2024-07-26T08:36:12.736113+0000","last_fullsized":"2024-07-26T08:36:12.736113+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733731+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:36:12.733731+0000","last_peered":"2024-07-26T08:36:12.733731+0000","last_clean":"2024-07-26T08:36:12.733731+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:36:12.733731+0000","last_undegraded":"2024-07-26T08:36:12.733731+0000","last_fullsized":"2024-07-26T08:36:12.733731+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740824+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:36:12.740824+0000","last_peered":"2024-07-26T08:36:12.740824+0000","last_clean":"2024-07-26T08:36:12.740824+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:36:12.740824+0000","last_undegraded":"2024-07-26T08:36:12.740824+0000","last_fullsized":"2024-07-26T08:36:12.740824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743134+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:36:12.743134+0000","last_peered":"2024-07-26T08:36:12.743134+0000","last_clean":"2024-07-26T08:36:12.743134+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:36:12.743134+0000","last_undegraded":"2024-07-26T08:36:12.743134+0000","last_fullsized":"2024-07-26T08:36:12.743134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739206+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:36:12.739206+0000","last_peered":"2024-07-26T08:36:12.739206+0000","last_clean":"2024-07-26T08:36:12.739206+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:36:12.739206+0000","last_undegraded":"2024-07-26T08:36:12.739206+0000","last_fullsized":"2024-07-26T08:36:12.739206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738943+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:36:12.738943+0000","last_peered":"2024-07-26T08:36:12.738943+0000","last_clean":"2024-07-26T08:36:12.738943+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:36:12.738943+0000","last_undegraded":"2024-07-26T08:36:12.738943+0000","last_fullsized":"2024-07-26T08:36:12.738943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729478+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:36:12.729478+0000","last_peered":"2024-07-26T08:36:12.729478+0000","last_clean":"2024-07-26T08:36:12.729478+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:36:12.729478+0000","last_undegraded":"2024-07-26T08:36:12.729478+0000","last_fullsized":"2024-07-26T08:36:12.729478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719269+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:36:12.719269+0000","last_peered":"2024-07-26T08:36:12.719269+0000","last_clean":"2024-07-26T08:36:12.719269+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:36:12.719269+0000","last_undegraded":"2024-07-26T08:36:12.719269+0000","last_fullsized":"2024-07-26T08:36:12.719269+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742197+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:36:12.742197+0000","last_peered":"2024-07-26T08:36:12.742197+0000","last_clean":"2024-07-26T08:36:12.742197+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:36:12.742197+0000","last_undegraded":"2024-07-26T08:36:12.742197+0000","last_fullsized":"2024-07-26T08:36:12.742197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721942+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:36:12.721942+0000","last_peered":"2024-07-26T08:36:12.721942+0000","last_clean":"2024-07-26T08:36:12.721942+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:36:12.721942+0000","last_undegraded":"2024-07-26T08:36:12.721942+0000","last_fullsized":"2024-07-26T08:36:12.721942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740367+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:36:12.740367+0000","last_peered":"2024-07-26T08:36:12.740367+0000","last_clean":"2024-07-26T08:36:12.740367+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:36:12.740367+0000","last_undegraded":"2024-07-26T08:36:12.740367+0000","last_fullsized":"2024-07-26T08:36:12.740367+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746129+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:36:12.746129+0000","last_peered":"2024-07-26T08:36:12.746129+0000","last_clean":"2024-07-26T08:36:12.746129+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:36:12.746129+0000","last_undegraded":"2024-07-26T08:36:12.746129+0000","last_fullsized":"2024-07-26T08:36:12.746129+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739206+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:36:12.739206+0000","last_peered":"2024-07-26T08:36:12.739206+0000","last_clean":"2024-07-26T08:36:12.739206+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:36:12.739206+0000","last_undegraded":"2024-07-26T08:36:12.739206+0000","last_fullsized":"2024-07-26T08:36:12.739206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728644+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:36:12.728644+0000","last_peered":"2024-07-26T08:36:12.728644+0000","last_clean":"2024-07-26T08:36:12.728644+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:36:12.728644+0000","last_undegraded":"2024-07-26T08:36:12.728644+0000","last_fullsized":"2024-07-26T08:36:12.728644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743895+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:36:12.743895+0000","last_peered":"2024-07-26T08:36:12.743895+0000","last_clean":"2024-07-26T08:36:12.743895+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:36:12.743895+0000","last_undegraded":"2024-07-26T08:36:12.743895+0000","last_fullsized":"2024-07-26T08:36:12.743895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719735+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:36:12.719735+0000","last_peered":"2024-07-26T08:36:12.719735+0000","last_clean":"2024-07-26T08:36:12.719735+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:36:12.719735+0000","last_undegraded":"2024-07-26T08:36:12.719735+0000","last_fullsized":"2024-07-26T08:36:12.719735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736112+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:36:12.736112+0000","last_peered":"2024-07-26T08:36:12.736112+0000","last_clean":"2024-07-26T08:36:12.736112+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:36:12.736112+0000","last_undegraded":"2024-07-26T08:36:12.736112+0000","last_fullsized":"2024-07-26T08:36:12.736112+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739102+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:36:12.739102+0000","last_peered":"2024-07-26T08:36:12.739102+0000","last_clean":"2024-07-26T08:36:12.739102+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:36:12.739102+0000","last_undegraded":"2024-07-26T08:36:12.739102+0000","last_fullsized":"2024-07-26T08:36:12.739102+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740316+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:36:12.740316+0000","last_peered":"2024-07-26T08:36:12.740316+0000","last_clean":"2024-07-26T08:36:12.740316+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:36:12.740316+0000","last_undegraded":"2024-07-26T08:36:12.740316+0000","last_fullsized":"2024-07-26T08:36:12.740316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80756,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2497,"num_read_kb":1339,"num_write":269,"num_write_kb":167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":242268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645397,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540314,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841625,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1175552,"data_stored":1204384,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005150,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201381,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397611,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136923,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561134,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757365,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920890,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540017,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378759,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":609849,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542283,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":609097,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":537092,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":75467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:38:40.197 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T08:38:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.513+0000 7f395a400640 1 -- 172.21.15.44:0/41306651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3954108c50 msgr2=0x7f3954109060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.513+0000 7f395a400640 1 --2- 172.21.15.44:0/41306651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3954108c50 0x7f3954109060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f393c009ca0 tx=0x7f393c02a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 -- 172.21.15.44:0/41306651 shutdown_connections 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 --2- 172.21.15.44:0/41306651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39541095a0 0x7f395410fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 --2- 172.21.15.44:0/41306651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3954108c50 0x7f3954109060 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 --2- 172.21.15.44:0/41306651 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3954103320 0x7f3954103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 -- 172.21.15.44:0/41306651 >> 172.21.15.44:0/41306651 conn(0x7f39540fed80 msgr2=0x7f39541011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 -- 172.21.15.44:0/41306651 shutdown_connections 2024-07-26T08:38:40.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.514+0000 7f395a400640 1 -- 172.21.15.44:0/41306651 wait complete. 2024-07-26T08:38:40.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 Processor -- start 2024-07-26T08:38:40.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 -- start start 2024-07-26T08:38:40.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 0x7f3954075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:40.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3954108c50 0x7f3954075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39541095a0 0x7f3954075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3954113210 con 0x7f3954103320 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3954113090 con 0x7f39541095a0 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.515+0000 7f395a400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3954113390 con 0x7f3954108c50 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 0x7f3954075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 0x7f3954075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49238/0 (socket says 172.21.15.44:49238) 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 -- 172.21.15.44:0/1678472301 learned_addr learned my addr 172.21.15.44:0/1678472301 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3953e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3954108c50 0x7f3954075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959a00640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39541095a0 0x7f3954075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 -- 172.21.15.44:0/1678472301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3954108c50 msgr2=0x7f3954075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3954108c50 0x7f3954075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 -- 172.21.15.44:0/1678472301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39541095a0 msgr2=0x7f3954075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39541095a0 0x7f3954075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.516+0000 7f3959000640 1 -- 172.21.15.44:0/1678472301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39541a7f80 con 0x7f3954103320 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.517+0000 7f3959000640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 0x7f3954075140 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f394400e9f0 tx=0x7f394400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.517+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394400cd80 con 0x7f3954103320 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.517+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39440108e0 con 0x7f3954103320 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.517+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39541a8270 con 0x7f3954103320 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.517+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944018980 con 0x7f3954103320 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.517+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39541a87e0 con 0x7f3954103320 2024-07-26T08:38:40.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.518+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3954103730 con 0x7f3954103320 2024-07-26T08:38:40.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.521+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f3944010430 con 0x7f3954103320 2024-07-26T08:38:40.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.522+0000 7f3951600640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39380b3cd0 0x7f39380b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:40.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.523+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f39440e4870 con 0x7f3954103320 2024-07-26T08:38:40.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.523+0000 7f3953e00640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39380b3cd0 0x7f39380b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:40.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.524+0000 7f3953e00640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39380b3cd0 0x7f39380b61e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f393c000f80 tx=0x7f393c009360 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:40.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.525+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f39440ab360 con 0x7f3954103320 2024-07-26T08:38:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.738+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3954063230 con 0x7f39380b3cd0 2024-07-26T08:38:40.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.750+0000 7f3951600640 1 -- 172.21.15.44:0/1678472301 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340810 (secure 0 0 0) 0x7f3954063230 con 0x7f39380b3cd0 2024-07-26T08:38:40.753 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:40.758 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:38:40.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.758+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39380b3cd0 msgr2=0x7f39380b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.758+0000 7f395a400640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39380b3cd0 0x7f39380b61e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f393c000f80 tx=0x7f393c009360 comp rx=0 tx=0).stop 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.758+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 msgr2=0x7f3954075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.758+0000 7f395a400640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 0x7f3954075140 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f394400e9f0 tx=0x7f394400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 shutdown_connections 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f39380b3cd0 0x7f39380b61e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3954108c50 0x7f3954075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39541095a0 0x7f3954075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 --2- 172.21.15.44:0/1678472301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3954103320 0x7f3954075140 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 >> 172.21.15.44:0/1678472301 conn(0x7f39540fed80 msgr2=0x7f39541011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 shutdown_connections 2024-07-26T08:38:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:40.759+0000 7f395a400640 1 -- 172.21.15.44:0/1678472301 wait complete. 2024-07-26T08:38:40.789 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-07-26T08:38:38.918170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671358,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2578,"num_read_kb":1408,"num_write":394,"num_write_kb":2319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":826,"ondisk_log_size":826,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365740,"kb_used_data":12428,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576084,"statfs":{"total":1151940427776,"available":1151565910016,"internally_reserved":0,"allocated":12726272,"data_stored":8458724,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":73,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006184"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":305,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.693455+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:38:34.693455+0000","last_peered":"2024-07-26T08:38:34.693455+0000","last_clean":"2024-07-26T08:38:34.693455+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:38:34.693455+0000","last_undegraded":"2024-07-26T08:38:34.693455+0000","last_fullsized":"2024-07-26T08:38:34.693455+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":245,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746837+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:36:12.746837+0000","last_peered":"2024-07-26T08:36:12.746837+0000","last_clean":"2024-07-26T08:36:12.746837+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:36:12.746837+0000","last_undegraded":"2024-07-26T08:36:12.746837+0000","last_fullsized":"2024-07-26T08:36:12.746837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:45.161843+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:35:45.161843+0000","last_peered":"2024-07-26T08:35:45.161843+0000","last_clean":"2024-07-26T08:35:45.161843+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:35:45.161843+0000","last_undegraded":"2024-07-26T08:35:45.161843+0000","last_fullsized":"2024-07-26T08:35:45.161843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:57.270075+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:57.270075+0000","last_peered":"2024-07-26T08:34:57.270075+0000","last_clean":"2024-07-26T08:34:57.270075+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:57.270075+0000","last_undegraded":"2024-07-26T08:34:57.270075+0000","last_fullsized":"2024-07-26T08:34:57.270075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741337+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:36:12.741337+0000","last_peered":"2024-07-26T08:36:12.741337+0000","last_clean":"2024-07-26T08:36:12.741337+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:36:12.741337+0000","last_undegraded":"2024-07-26T08:36:12.741337+0000","last_fullsized":"2024-07-26T08:36:12.741337+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:43.579910+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:35:43.579910+0000","last_peered":"2024-07-26T08:35:43.579910+0000","last_clean":"2024-07-26T08:35:43.579910+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:35:43.579910+0000","last_undegraded":"2024-07-26T08:35:43.579910+0000","last_fullsized":"2024-07-26T08:35:43.579910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:30.595827+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:35:30.595827+0000","last_peered":"2024-07-26T08:35:30.595827+0000","last_clean":"2024-07-26T08:35:30.595827+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:35:30.595827+0000","last_undegraded":"2024-07-26T08:35:30.595827+0000","last_fullsized":"2024-07-26T08:35:30.595827+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741380+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:36:12.741380+0000","last_peered":"2024-07-26T08:36:12.741380+0000","last_clean":"2024-07-26T08:36:12.741380+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:36:12.741380+0000","last_undegraded":"2024-07-26T08:36:12.741380+0000","last_fullsized":"2024-07-26T08:36:12.741380+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:48.580551+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:35:48.580551+0000","last_peered":"2024-07-26T08:35:48.580551+0000","last_clean":"2024-07-26T08:35:48.580551+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:35:48.580551+0000","last_undegraded":"2024-07-26T08:35:48.580551+0000","last_fullsized":"2024-07-26T08:35:48.580551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:30.595776+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:35:30.595776+0000","last_peered":"2024-07-26T08:35:30.595776+0000","last_clean":"2024-07-26T08:35:30.595776+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:35:30.595776+0000","last_undegraded":"2024-07-26T08:35:30.595776+0000","last_fullsized":"2024-07-26T08:35:30.595776+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734806+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:36:12.734806+0000","last_peered":"2024-07-26T08:36:12.734806+0000","last_clean":"2024-07-26T08:36:12.734806+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:36:12.734806+0000","last_undegraded":"2024-07-26T08:36:12.734806+0000","last_fullsized":"2024-07-26T08:36:12.734806+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:48.580530+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:35:48.580530+0000","last_peered":"2024-07-26T08:35:48.580530+0000","last_clean":"2024-07-26T08:35:48.580530+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:35:48.580530+0000","last_undegraded":"2024-07-26T08:35:48.580530+0000","last_fullsized":"2024-07-26T08:35:48.580530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742218+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:36:12.742218+0000","last_peered":"2024-07-26T08:36:12.742218+0000","last_clean":"2024-07-26T08:36:12.742218+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:36:12.742218+0000","last_undegraded":"2024-07-26T08:36:12.742218+0000","last_fullsized":"2024-07-26T08:36:12.742218+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:15.594066+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:35:15.594066+0000","last_peered":"2024-07-26T08:35:15.594066+0000","last_clean":"2024-07-26T08:35:15.594066+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:35:15.594066+0000","last_undegraded":"2024-07-26T08:35:15.594066+0000","last_fullsized":"2024-07-26T08:35:15.594066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578665+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:35:33.578665+0000","last_peered":"2024-07-26T08:35:33.578665+0000","last_clean":"2024-07-26T08:35:33.578665+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:35:33.578665+0000","last_undegraded":"2024-07-26T08:35:33.578665+0000","last_fullsized":"2024-07-26T08:35:33.578665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744388+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:36:12.744388+0000","last_peered":"2024-07-26T08:36:12.744388+0000","last_clean":"2024-07-26T08:36:12.744388+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:36:12.744388+0000","last_undegraded":"2024-07-26T08:36:12.744388+0000","last_fullsized":"2024-07-26T08:36:12.744388+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320730+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:35:26.320730+0000","last_peered":"2024-07-26T08:35:26.320730+0000","last_clean":"2024-07-26T08:35:26.320730+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:35:26.320730+0000","last_undegraded":"2024-07-26T08:35:26.320730+0000","last_fullsized":"2024-07-26T08:35:26.320730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578615+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:35:33.578615+0000","last_peered":"2024-07-26T08:35:33.578615+0000","last_clean":"2024-07-26T08:35:33.578615+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:35:33.578615+0000","last_undegraded":"2024-07-26T08:35:33.578615+0000","last_fullsized":"2024-07-26T08:35:33.578615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740319+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:36:12.740319+0000","last_peered":"2024-07-26T08:36:12.740319+0000","last_clean":"2024-07-26T08:36:12.740319+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:36:12.740319+0000","last_undegraded":"2024-07-26T08:36:12.740319+0000","last_fullsized":"2024-07-26T08:36:12.740319+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.272221+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:35:07.272221+0000","last_peered":"2024-07-26T08:35:07.272221+0000","last_clean":"2024-07-26T08:35:07.272221+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:35:07.272221+0000","last_undegraded":"2024-07-26T08:35:07.272221+0000","last_fullsized":"2024-07-26T08:35:07.272221+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:36.345867+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:35:36.345867+0000","last_peered":"2024-07-26T08:35:36.345867+0000","last_clean":"2024-07-26T08:35:36.345867+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:35:36.345867+0000","last_undegraded":"2024-07-26T08:35:36.345867+0000","last_fullsized":"2024-07-26T08:35:36.345867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.712601+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:36:12.712601+0000","last_peered":"2024-07-26T08:36:12.712601+0000","last_clean":"2024-07-26T08:36:12.712601+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:36:12.712601+0000","last_undegraded":"2024-07-26T08:36:12.712601+0000","last_fullsized":"2024-07-26T08:36:12.712601+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.548672+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:35:07.548672+0000","last_peered":"2024-07-26T08:35:07.548672+0000","last_clean":"2024-07-26T08:35:07.548672+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:35:07.548672+0000","last_undegraded":"2024-07-26T08:35:07.548672+0000","last_fullsized":"2024-07-26T08:35:07.548672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578686+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:35:33.578686+0000","last_peered":"2024-07-26T08:35:33.578686+0000","last_clean":"2024-07-26T08:35:33.578686+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:35:33.578686+0000","last_undegraded":"2024-07-26T08:35:33.578686+0000","last_fullsized":"2024-07-26T08:35:33.578686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.747319+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:36:12.747319+0000","last_peered":"2024-07-26T08:36:12.747319+0000","last_clean":"2024-07-26T08:36:12.747319+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:36:12.747319+0000","last_undegraded":"2024-07-26T08:36:12.747319+0000","last_fullsized":"2024-07-26T08:36:12.747319+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:08.575512+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:35:08.575512+0000","last_peered":"2024-07-26T08:35:08.575512+0000","last_clean":"2024-07-26T08:35:08.575512+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:35:08.575512+0000","last_undegraded":"2024-07-26T08:35:08.575512+0000","last_fullsized":"2024-07-26T08:35:08.575512+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:25.159325+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:35:25.159325+0000","last_peered":"2024-07-26T08:35:25.159325+0000","last_clean":"2024-07-26T08:35:25.159325+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:35:25.159325+0000","last_undegraded":"2024-07-26T08:35:25.159325+0000","last_fullsized":"2024-07-26T08:35:25.159325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728535+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:36:12.728535+0000","last_peered":"2024-07-26T08:36:12.728535+0000","last_clean":"2024-07-26T08:36:12.728535+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:36:12.728535+0000","last_undegraded":"2024-07-26T08:36:12.728535+0000","last_fullsized":"2024-07-26T08:36:12.728535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'18","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:37:21.364570+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:37:21.364570+0000","last_peered":"2024-07-26T08:37:21.364570+0000","last_clean":"2024-07-26T08:37:21.364570+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:37:21.364570+0000","last_undegraded":"2024-07-26T08:37:21.364570+0000","last_fullsized":"2024-07-26T08:37:21.364570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:17.274168+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:35:17.274168+0000","last_peered":"2024-07-26T08:35:17.274168+0000","last_clean":"2024-07-26T08:35:17.274168+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:35:17.274168+0000","last_undegraded":"2024-07-26T08:35:17.274168+0000","last_fullsized":"2024-07-26T08:35:17.274168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734147+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:36:12.734147+0000","last_peered":"2024-07-26T08:36:12.734147+0000","last_clean":"2024-07-26T08:36:12.734147+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:36:12.734147+0000","last_undegraded":"2024-07-26T08:36:12.734147+0000","last_fullsized":"2024-07-26T08:36:12.734147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:05.592648+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:35:05.592648+0000","last_peered":"2024-07-26T08:35:05.592648+0000","last_clean":"2024-07-26T08:35:05.592648+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:35:05.592648+0000","last_undegraded":"2024-07-26T08:35:05.592648+0000","last_fullsized":"2024-07-26T08:35:05.592648+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320776+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:35:26.320776+0000","last_peered":"2024-07-26T08:35:26.320776+0000","last_clean":"2024-07-26T08:35:26.320776+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:35:26.320776+0000","last_undegraded":"2024-07-26T08:35:26.320776+0000","last_fullsized":"2024-07-26T08:35:26.320776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744052+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:36:12.744052+0000","last_peered":"2024-07-26T08:36:12.744052+0000","last_clean":"2024-07-26T08:36:12.744052+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:36:12.744052+0000","last_undegraded":"2024-07-26T08:36:12.744052+0000","last_fullsized":"2024-07-26T08:36:12.744052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:06.339791+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:35:06.339791+0000","last_peered":"2024-07-26T08:35:06.339791+0000","last_clean":"2024-07-26T08:35:06.339791+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:35:06.339791+0000","last_undegraded":"2024-07-26T08:35:06.339791+0000","last_fullsized":"2024-07-26T08:35:06.339791+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:25.595196+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:35:25.595196+0000","last_peered":"2024-07-26T08:35:25.595196+0000","last_clean":"2024-07-26T08:35:25.595196+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:35:25.595196+0000","last_undegraded":"2024-07-26T08:35:25.595196+0000","last_fullsized":"2024-07-26T08:35:25.595196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740048+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:36:12.740048+0000","last_peered":"2024-07-26T08:36:12.740048+0000","last_clean":"2024-07-26T08:36:12.740048+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:36:12.740048+0000","last_undegraded":"2024-07-26T08:36:12.740048+0000","last_fullsized":"2024-07-26T08:36:12.740048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:35.696701+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:38:35.696701+0000","last_peered":"2024-07-26T08:38:35.696701+0000","last_clean":"2024-07-26T08:38:35.696701+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:38:35.696701+0000","last_undegraded":"2024-07-26T08:38:35.696701+0000","last_fullsized":"2024-07-26T08:38:35.696701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":246,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.566680+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:35:26.566680+0000","last_peered":"2024-07-26T08:35:26.566680+0000","last_clean":"2024-07-26T08:35:26.566680+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:35:26.566680+0000","last_undegraded":"2024-07-26T08:35:26.566680+0000","last_fullsized":"2024-07-26T08:35:26.566680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729599+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:36:12.729599+0000","last_peered":"2024-07-26T08:36:12.729599+0000","last_clean":"2024-07-26T08:36:12.729599+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:36:12.729599+0000","last_undegraded":"2024-07-26T08:36:12.729599+0000","last_fullsized":"2024-07-26T08:36:12.729599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.779037+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:38:34.779037+0000","last_peered":"2024-07-26T08:38:34.779037+0000","last_clean":"2024-07-26T08:38:34.779037+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:38:34.779037+0000","last_undegraded":"2024-07-26T08:38:34.779037+0000","last_fullsized":"2024-07-26T08:38:34.779037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":245,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.567573+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:35:31.567573+0000","last_peered":"2024-07-26T08:35:31.567573+0000","last_clean":"2024-07-26T08:35:31.567573+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:35:31.567573+0000","last_undegraded":"2024-07-26T08:35:31.567573+0000","last_fullsized":"2024-07-26T08:35:31.567573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733996+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:36:12.733996+0000","last_peered":"2024-07-26T08:36:12.733996+0000","last_clean":"2024-07-26T08:36:12.733996+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:36:12.733996+0000","last_undegraded":"2024-07-26T08:36:12.733996+0000","last_fullsized":"2024-07-26T08:36:12.733996+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:57.422284+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:57.422284+0000","last_peered":"2024-07-26T08:34:57.422284+0000","last_clean":"2024-07-26T08:34:57.422284+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:57.422284+0000","last_undegraded":"2024-07-26T08:34:57.422284+0000","last_fullsized":"2024-07-26T08:34:57.422284+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406163+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:35:31.406163+0000","last_peered":"2024-07-26T08:35:31.406163+0000","last_clean":"2024-07-26T08:35:31.406163+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:35:31.406163+0000","last_undegraded":"2024-07-26T08:35:31.406163+0000","last_fullsized":"2024-07-26T08:35:31.406163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745276+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:36:12.745276+0000","last_peered":"2024-07-26T08:36:12.745276+0000","last_clean":"2024-07-26T08:36:12.745276+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:36:12.745276+0000","last_undegraded":"2024-07-26T08:36:12.745276+0000","last_fullsized":"2024-07-26T08:36:12.745276+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:03.574928+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:35:03.574928+0000","last_peered":"2024-07-26T08:35:03.574928+0000","last_clean":"2024-07-26T08:35:03.574928+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:35:03.574928+0000","last_undegraded":"2024-07-26T08:35:03.574928+0000","last_fullsized":"2024-07-26T08:35:03.574928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:07.430772+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:36:07.430772+0000","last_peered":"2024-07-26T08:36:07.430772+0000","last_clean":"2024-07-26T08:36:07.430772+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:36:07.430772+0000","last_undegraded":"2024-07-26T08:36:07.430772+0000","last_fullsized":"2024-07-26T08:36:07.430772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:22.425289+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:35:22.425289+0000","last_peered":"2024-07-26T08:35:22.425289+0000","last_clean":"2024-07-26T08:35:22.425289+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:35:22.425289+0000","last_undegraded":"2024-07-26T08:35:22.425289+0000","last_fullsized":"2024-07-26T08:35:22.425289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742063+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:36:12.742063+0000","last_peered":"2024-07-26T08:36:12.742063+0000","last_clean":"2024-07-26T08:36:12.742063+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:36:12.742063+0000","last_undegraded":"2024-07-26T08:36:12.742063+0000","last_fullsized":"2024-07-26T08:36:12.742063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":301,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:33.496342+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:38:33.496342+0000","last_peered":"2024-07-26T08:38:33.496342+0000","last_clean":"2024-07-26T08:38:33.496342+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:38:33.496342+0000","last_undegraded":"2024-07-26T08:38:33.496342+0000","last_fullsized":"2024-07-26T08:38:33.496342+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":247,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:27.551514+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:35:27.551514+0000","last_peered":"2024-07-26T08:35:27.551514+0000","last_clean":"2024-07-26T08:35:27.551514+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:35:27.551514+0000","last_undegraded":"2024-07-26T08:35:27.551514+0000","last_fullsized":"2024-07-26T08:35:27.551514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744223+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:36:12.744223+0000","last_peered":"2024-07-26T08:36:12.744223+0000","last_clean":"2024-07-26T08:36:12.744223+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:36:12.744223+0000","last_undegraded":"2024-07-26T08:36:12.744223+0000","last_fullsized":"2024-07-26T08:36:12.744223+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:16.404127+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:35:16.404127+0000","last_peered":"2024-07-26T08:35:16.404127+0000","last_clean":"2024-07-26T08:35:16.404127+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:35:16.404127+0000","last_undegraded":"2024-07-26T08:35:16.404127+0000","last_fullsized":"2024-07-26T08:35:16.404127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:32.864361+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:35:32.864361+0000","last_peered":"2024-07-26T08:35:32.864361+0000","last_clean":"2024-07-26T08:35:32.864361+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:35:32.864361+0000","last_undegraded":"2024-07-26T08:35:32.864361+0000","last_fullsized":"2024-07-26T08:35:32.864361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727635+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:36:12.727635+0000","last_peered":"2024-07-26T08:36:12.727635+0000","last_clean":"2024-07-26T08:36:12.727635+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:36:12.727635+0000","last_undegraded":"2024-07-26T08:36:12.727635+0000","last_fullsized":"2024-07-26T08:36:12.727635+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:16.565096+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:35:16.565096+0000","last_peered":"2024-07-26T08:35:16.565096+0000","last_clean":"2024-07-26T08:35:16.565096+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:35:16.565096+0000","last_undegraded":"2024-07-26T08:35:16.565096+0000","last_fullsized":"2024-07-26T08:35:16.565096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406097+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:35:31.406097+0000","last_peered":"2024-07-26T08:35:31.406097+0000","last_clean":"2024-07-26T08:35:31.406097+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:35:31.406097+0000","last_undegraded":"2024-07-26T08:35:31.406097+0000","last_fullsized":"2024-07-26T08:35:31.406097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.722492+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:36:12.722492+0000","last_peered":"2024-07-26T08:36:12.722492+0000","last_clean":"2024-07-26T08:36:12.722492+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:36:12.722492+0000","last_undegraded":"2024-07-26T08:36:12.722492+0000","last_fullsized":"2024-07-26T08:36:12.722492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:15.594107+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:35:15.594107+0000","last_peered":"2024-07-26T08:35:15.594107+0000","last_clean":"2024-07-26T08:35:15.594107+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:35:15.594107+0000","last_undegraded":"2024-07-26T08:35:15.594107+0000","last_fullsized":"2024-07-26T08:35:15.594107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406127+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:35:31.406127+0000","last_peered":"2024-07-26T08:35:31.406127+0000","last_clean":"2024-07-26T08:35:31.406127+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:35:31.406127+0000","last_undegraded":"2024-07-26T08:35:31.406127+0000","last_fullsized":"2024-07-26T08:35:31.406127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733854+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:36:12.733854+0000","last_peered":"2024-07-26T08:36:12.733854+0000","last_clean":"2024-07-26T08:36:12.733854+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:36:12.733854+0000","last_undegraded":"2024-07-26T08:36:12.733854+0000","last_fullsized":"2024-07-26T08:36:12.733854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108992+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:32.108992+0000","last_peered":"2024-07-26T08:34:32.108992+0000","last_clean":"2024-07-26T08:34:32.108992+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:32.108992+0000","last_undegraded":"2024-07-26T08:34:32.108992+0000","last_fullsized":"2024-07-26T08:34:32.108992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:46.570284+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:35:46.570284+0000","last_peered":"2024-07-26T08:35:46.570284+0000","last_clean":"2024-07-26T08:35:46.570284+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:35:46.570284+0000","last_undegraded":"2024-07-26T08:35:46.570284+0000","last_fullsized":"2024-07-26T08:35:46.570284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.731664+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:36:12.731664+0000","last_peered":"2024-07-26T08:36:12.731664+0000","last_clean":"2024-07-26T08:36:12.731664+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:36:12.731664+0000","last_undegraded":"2024-07-26T08:36:12.731664+0000","last_fullsized":"2024-07-26T08:36:12.731664+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'8","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:36.784851+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:38:36.784851+0000","last_peered":"2024-07-26T08:38:36.784851+0000","last_clean":"2024-07-26T08:38:36.784851+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:38:36.784851+0000","last_undegraded":"2024-07-26T08:38:36.784851+0000","last_fullsized":"2024-07-26T08:38:36.784851+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1427,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:46.348069+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:35:46.348069+0000","last_peered":"2024-07-26T08:35:46.348069+0000","last_clean":"2024-07-26T08:35:46.348069+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:35:46.348069+0000","last_undegraded":"2024-07-26T08:35:46.348069+0000","last_fullsized":"2024-07-26T08:35:46.348069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734066+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:36:12.734066+0000","last_peered":"2024-07-26T08:36:12.734066+0000","last_clean":"2024-07-26T08:36:12.734066+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:36:12.734066+0000","last_undegraded":"2024-07-26T08:36:12.734066+0000","last_fullsized":"2024-07-26T08:36:12.734066+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:21.404762+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:35:21.404762+0000","last_peered":"2024-07-26T08:35:21.404762+0000","last_clean":"2024-07-26T08:35:21.404762+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:35:21.404762+0000","last_undegraded":"2024-07-26T08:35:21.404762+0000","last_fullsized":"2024-07-26T08:35:21.404762+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.322567+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:35:41.322567+0000","last_peered":"2024-07-26T08:35:41.322567+0000","last_clean":"2024-07-26T08:35:41.322567+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:35:41.322567+0000","last_undegraded":"2024-07-26T08:35:41.322567+0000","last_fullsized":"2024-07-26T08:35:41.322567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741015+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:36:12.741015+0000","last_peered":"2024-07-26T08:36:12.741015+0000","last_clean":"2024-07-26T08:36:12.741015+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:36:12.741015+0000","last_undegraded":"2024-07-26T08:36:12.741015+0000","last_fullsized":"2024-07-26T08:36:12.741015+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:20.158778+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:35:20.158778+0000","last_peered":"2024-07-26T08:35:20.158778+0000","last_clean":"2024-07-26T08:35:20.158778+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:35:20.158778+0000","last_undegraded":"2024-07-26T08:35:20.158778+0000","last_fullsized":"2024-07-26T08:35:20.158778+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:45.923456+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:35:45.923456+0000","last_peered":"2024-07-26T08:35:45.923456+0000","last_clean":"2024-07-26T08:35:45.923456+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:35:45.923456+0000","last_undegraded":"2024-07-26T08:35:45.923456+0000","last_fullsized":"2024-07-26T08:35:45.923456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.717243+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:36:12.717243+0000","last_peered":"2024-07-26T08:36:12.717243+0000","last_clean":"2024-07-26T08:36:12.717243+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:36:12.717243+0000","last_undegraded":"2024-07-26T08:36:12.717243+0000","last_fullsized":"2024-07-26T08:36:12.717243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":595,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:37.502224+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:38:37.502224+0000","last_peered":"2024-07-26T08:38:37.502224+0000","last_clean":"2024-07-26T08:38:37.502224+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:38:37.502224+0000","last_undegraded":"2024-07-26T08:38:37.502224+0000","last_fullsized":"2024-07-26T08:38:37.502224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72963,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":504,"num_read_kb":324,"num_write":46,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.346929+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:35:41.346929+0000","last_peered":"2024-07-26T08:35:41.346929+0000","last_clean":"2024-07-26T08:35:41.346929+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:35:41.346929+0000","last_undegraded":"2024-07-26T08:35:41.346929+0000","last_fullsized":"2024-07-26T08:35:41.346929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738676+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:36:12.738676+0000","last_peered":"2024-07-26T08:36:12.738676+0000","last_clean":"2024-07-26T08:36:12.738676+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:36:12.738676+0000","last_undegraded":"2024-07-26T08:36:12.738676+0000","last_fullsized":"2024-07-26T08:36:12.738676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.405488+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:35:26.405488+0000","last_peered":"2024-07-26T08:35:26.405488+0000","last_clean":"2024-07-26T08:35:26.405488+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:35:26.405488+0000","last_undegraded":"2024-07-26T08:35:26.405488+0000","last_fullsized":"2024-07-26T08:35:26.405488+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:38.579362+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:35:38.579362+0000","last_peered":"2024-07-26T08:35:38.579362+0000","last_clean":"2024-07-26T08:35:38.579362+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:35:38.579362+0000","last_undegraded":"2024-07-26T08:35:38.579362+0000","last_fullsized":"2024-07-26T08:35:38.579362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740282+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:36:12.740282+0000","last_peered":"2024-07-26T08:36:12.740282+0000","last_clean":"2024-07-26T08:36:12.740282+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:36:12.740282+0000","last_undegraded":"2024-07-26T08:36:12.740282+0000","last_fullsized":"2024-07-26T08:36:12.740282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:22.425306+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:35:22.425306+0000","last_peered":"2024-07-26T08:35:22.425306+0000","last_clean":"2024-07-26T08:35:22.425306+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:35:22.425306+0000","last_undegraded":"2024-07-26T08:35:22.425306+0000","last_fullsized":"2024-07-26T08:35:22.425306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.346745+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:35:41.346745+0000","last_peered":"2024-07-26T08:35:41.346745+0000","last_clean":"2024-07-26T08:35:41.346745+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:35:41.346745+0000","last_undegraded":"2024-07-26T08:35:41.346745+0000","last_fullsized":"2024-07-26T08:35:41.346745+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741135+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:36:12.741135+0000","last_peered":"2024-07-26T08:36:12.741135+0000","last_clean":"2024-07-26T08:36:12.741135+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:36:12.741135+0000","last_undegraded":"2024-07-26T08:36:12.741135+0000","last_fullsized":"2024-07-26T08:36:12.741135+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":544,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:37.501908+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:38:37.501908+0000","last_peered":"2024-07-26T08:38:37.501908+0000","last_clean":"2024-07-26T08:38:37.501908+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:38:37.501908+0000","last_undegraded":"2024-07-26T08:38:37.501908+0000","last_fullsized":"2024-07-26T08:38:37.501908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":492,"num_read_kb":492,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:37.278601+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:35:37.278601+0000","last_peered":"2024-07-26T08:35:37.278601+0000","last_clean":"2024-07-26T08:35:37.278601+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:35:37.278601+0000","last_undegraded":"2024-07-26T08:35:37.278601+0000","last_fullsized":"2024-07-26T08:35:37.278601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:17.861977+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:35:17.861977+0000","last_peered":"2024-07-26T08:35:17.861977+0000","last_clean":"2024-07-26T08:35:17.861977+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:35:17.861977+0000","last_undegraded":"2024-07-26T08:35:17.861977+0000","last_fullsized":"2024-07-26T08:35:17.861977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746413+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:36:12.746413+0000","last_peered":"2024-07-26T08:36:12.746413+0000","last_clean":"2024-07-26T08:36:12.746413+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:36:12.746413+0000","last_undegraded":"2024-07-26T08:36:12.746413+0000","last_fullsized":"2024-07-26T08:36:12.746413+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.405429+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:35:26.405429+0000","last_peered":"2024-07-26T08:35:26.405429+0000","last_clean":"2024-07-26T08:35:26.405429+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:35:26.405429+0000","last_undegraded":"2024-07-26T08:35:26.405429+0000","last_fullsized":"2024-07-26T08:35:26.405429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320795+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:35:26.320795+0000","last_peered":"2024-07-26T08:35:26.320795+0000","last_clean":"2024-07-26T08:35:26.320795+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:35:26.320795+0000","last_undegraded":"2024-07-26T08:35:26.320795+0000","last_fullsized":"2024-07-26T08:35:26.320795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741858+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:36:12.741858+0000","last_peered":"2024-07-26T08:36:12.741858+0000","last_clean":"2024-07-26T08:36:12.741858+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:36:12.741858+0000","last_undegraded":"2024-07-26T08:36:12.741858+0000","last_fullsized":"2024-07-26T08:36:12.741858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.566623+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:35:26.566623+0000","last_peered":"2024-07-26T08:35:26.566623+0000","last_clean":"2024-07-26T08:35:26.566623+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:35:26.566623+0000","last_undegraded":"2024-07-26T08:35:26.566623+0000","last_fullsized":"2024-07-26T08:35:26.566623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:27.846852+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:35:27.846852+0000","last_peered":"2024-07-26T08:35:27.846852+0000","last_clean":"2024-07-26T08:35:27.846852+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:35:27.846852+0000","last_undegraded":"2024-07-26T08:35:27.846852+0000","last_fullsized":"2024-07-26T08:35:27.846852+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740942+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:36:12.740942+0000","last_peered":"2024-07-26T08:36:12.740942+0000","last_clean":"2024-07-26T08:36:12.740942+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:36:12.740942+0000","last_undegraded":"2024-07-26T08:36:12.740942+0000","last_fullsized":"2024-07-26T08:36:12.740942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:50.162384+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:35:50.162384+0000","last_peered":"2024-07-26T08:35:50.162384+0000","last_clean":"2024-07-26T08:35:50.162384+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:35:50.162384+0000","last_undegraded":"2024-07-26T08:35:50.162384+0000","last_fullsized":"2024-07-26T08:35:50.162384+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.497991+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:38:34.497991+0000","last_peered":"2024-07-26T08:38:34.497991+0000","last_clean":"2024-07-26T08:38:34.497991+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:38:34.497991+0000","last_undegraded":"2024-07-26T08:38:34.497991+0000","last_fullsized":"2024-07-26T08:38:34.497991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":246,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727350+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:36:12.727350+0000","last_peered":"2024-07-26T08:36:12.727350+0000","last_clean":"2024-07-26T08:36:12.727350+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:36:12.727350+0000","last_undegraded":"2024-07-26T08:36:12.727350+0000","last_fullsized":"2024-07-26T08:36:12.727350+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:50.162409+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:35:50.162409+0000","last_peered":"2024-07-26T08:35:50.162409+0000","last_clean":"2024-07-26T08:35:50.162409+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:35:50.162409+0000","last_undegraded":"2024-07-26T08:35:50.162409+0000","last_fullsized":"2024-07-26T08:35:50.162409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723745+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:36:12.723745+0000","last_peered":"2024-07-26T08:36:12.723745+0000","last_clean":"2024-07-26T08:36:12.723745+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:36:12.723745+0000","last_undegraded":"2024-07-26T08:36:12.723745+0000","last_fullsized":"2024-07-26T08:36:12.723745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.722200+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:36:12.722200+0000","last_peered":"2024-07-26T08:36:12.722200+0000","last_clean":"2024-07-26T08:36:12.722200+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:36:12.722200+0000","last_undegraded":"2024-07-26T08:36:12.722200+0000","last_fullsized":"2024-07-26T08:36:12.722200+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737483+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:36:12.737483+0000","last_peered":"2024-07-26T08:36:12.737483+0000","last_clean":"2024-07-26T08:36:12.737483+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:36:12.737483+0000","last_undegraded":"2024-07-26T08:36:12.737483+0000","last_fullsized":"2024-07-26T08:36:12.737483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729462+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:36:12.729462+0000","last_peered":"2024-07-26T08:36:12.729462+0000","last_clean":"2024-07-26T08:36:12.729462+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:36:12.729462+0000","last_undegraded":"2024-07-26T08:36:12.729462+0000","last_fullsized":"2024-07-26T08:36:12.729462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742274+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:36:12.742274+0000","last_peered":"2024-07-26T08:36:12.742274+0000","last_clean":"2024-07-26T08:36:12.742274+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:36:12.742274+0000","last_undegraded":"2024-07-26T08:36:12.742274+0000","last_fullsized":"2024-07-26T08:36:12.742274+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718573+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:36:12.718573+0000","last_peered":"2024-07-26T08:36:12.718573+0000","last_clean":"2024-07-26T08:36:12.718573+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:36:12.718573+0000","last_undegraded":"2024-07-26T08:36:12.718573+0000","last_fullsized":"2024-07-26T08:36:12.718573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.732735+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:36:12.732735+0000","last_peered":"2024-07-26T08:36:12.732735+0000","last_clean":"2024-07-26T08:36:12.732735+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:36:12.732735+0000","last_undegraded":"2024-07-26T08:36:12.732735+0000","last_fullsized":"2024-07-26T08:36:12.732735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745906+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:36:12.745906+0000","last_peered":"2024-07-26T08:36:12.745906+0000","last_clean":"2024-07-26T08:36:12.745906+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:36:12.745906+0000","last_undegraded":"2024-07-26T08:36:12.745906+0000","last_fullsized":"2024-07-26T08:36:12.745906+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746340+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:36:12.746340+0000","last_peered":"2024-07-26T08:36:12.746340+0000","last_clean":"2024-07-26T08:36:12.746340+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:36:12.746340+0000","last_undegraded":"2024-07-26T08:36:12.746340+0000","last_fullsized":"2024-07-26T08:36:12.746340+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742178+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:36:12.742178+0000","last_peered":"2024-07-26T08:36:12.742178+0000","last_clean":"2024-07-26T08:36:12.742178+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:36:12.742178+0000","last_undegraded":"2024-07-26T08:36:12.742178+0000","last_fullsized":"2024-07-26T08:36:12.742178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746534+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:36:12.746534+0000","last_peered":"2024-07-26T08:36:12.746534+0000","last_clean":"2024-07-26T08:36:12.746534+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:36:12.746534+0000","last_undegraded":"2024-07-26T08:36:12.746534+0000","last_fullsized":"2024-07-26T08:36:12.746534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742408+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:36:12.742408+0000","last_peered":"2024-07-26T08:36:12.742408+0000","last_clean":"2024-07-26T08:36:12.742408+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:36:12.742408+0000","last_undegraded":"2024-07-26T08:36:12.742408+0000","last_fullsized":"2024-07-26T08:36:12.742408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723201+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:36:12.723201+0000","last_peered":"2024-07-26T08:36:12.723201+0000","last_clean":"2024-07-26T08:36:12.723201+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:36:12.723201+0000","last_undegraded":"2024-07-26T08:36:12.723201+0000","last_fullsized":"2024-07-26T08:36:12.723201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736113+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:36:12.736113+0000","last_peered":"2024-07-26T08:36:12.736113+0000","last_clean":"2024-07-26T08:36:12.736113+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:36:12.736113+0000","last_undegraded":"2024-07-26T08:36:12.736113+0000","last_fullsized":"2024-07-26T08:36:12.736113+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733731+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:36:12.733731+0000","last_peered":"2024-07-26T08:36:12.733731+0000","last_clean":"2024-07-26T08:36:12.733731+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:36:12.733731+0000","last_undegraded":"2024-07-26T08:36:12.733731+0000","last_fullsized":"2024-07-26T08:36:12.733731+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740824+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:36:12.740824+0000","last_peered":"2024-07-26T08:36:12.740824+0000","last_clean":"2024-07-26T08:36:12.740824+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:36:12.740824+0000","last_undegraded":"2024-07-26T08:36:12.740824+0000","last_fullsized":"2024-07-26T08:36:12.740824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743134+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:36:12.743134+0000","last_peered":"2024-07-26T08:36:12.743134+0000","last_clean":"2024-07-26T08:36:12.743134+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:36:12.743134+0000","last_undegraded":"2024-07-26T08:36:12.743134+0000","last_fullsized":"2024-07-26T08:36:12.743134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739206+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:36:12.739206+0000","last_peered":"2024-07-26T08:36:12.739206+0000","last_clean":"2024-07-26T08:36:12.739206+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:36:12.739206+0000","last_undegraded":"2024-07-26T08:36:12.739206+0000","last_fullsized":"2024-07-26T08:36:12.739206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738943+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:36:12.738943+0000","last_peered":"2024-07-26T08:36:12.738943+0000","last_clean":"2024-07-26T08:36:12.738943+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:36:12.738943+0000","last_undegraded":"2024-07-26T08:36:12.738943+0000","last_fullsized":"2024-07-26T08:36:12.738943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729478+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:36:12.729478+0000","last_peered":"2024-07-26T08:36:12.729478+0000","last_clean":"2024-07-26T08:36:12.729478+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:36:12.729478+0000","last_undegraded":"2024-07-26T08:36:12.729478+0000","last_fullsized":"2024-07-26T08:36:12.729478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719269+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:36:12.719269+0000","last_peered":"2024-07-26T08:36:12.719269+0000","last_clean":"2024-07-26T08:36:12.719269+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:36:12.719269+0000","last_undegraded":"2024-07-26T08:36:12.719269+0000","last_fullsized":"2024-07-26T08:36:12.719269+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742197+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:36:12.742197+0000","last_peered":"2024-07-26T08:36:12.742197+0000","last_clean":"2024-07-26T08:36:12.742197+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:36:12.742197+0000","last_undegraded":"2024-07-26T08:36:12.742197+0000","last_fullsized":"2024-07-26T08:36:12.742197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721942+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:36:12.721942+0000","last_peered":"2024-07-26T08:36:12.721942+0000","last_clean":"2024-07-26T08:36:12.721942+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:36:12.721942+0000","last_undegraded":"2024-07-26T08:36:12.721942+0000","last_fullsized":"2024-07-26T08:36:12.721942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740367+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:36:12.740367+0000","last_peered":"2024-07-26T08:36:12.740367+0000","last_clean":"2024-07-26T08:36:12.740367+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:36:12.740367+0000","last_undegraded":"2024-07-26T08:36:12.740367+0000","last_fullsized":"2024-07-26T08:36:12.740367+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746129+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:36:12.746129+0000","last_peered":"2024-07-26T08:36:12.746129+0000","last_clean":"2024-07-26T08:36:12.746129+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:36:12.746129+0000","last_undegraded":"2024-07-26T08:36:12.746129+0000","last_fullsized":"2024-07-26T08:36:12.746129+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739206+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:36:12.739206+0000","last_peered":"2024-07-26T08:36:12.739206+0000","last_clean":"2024-07-26T08:36:12.739206+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:36:12.739206+0000","last_undegraded":"2024-07-26T08:36:12.739206+0000","last_fullsized":"2024-07-26T08:36:12.739206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728644+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:36:12.728644+0000","last_peered":"2024-07-26T08:36:12.728644+0000","last_clean":"2024-07-26T08:36:12.728644+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:36:12.728644+0000","last_undegraded":"2024-07-26T08:36:12.728644+0000","last_fullsized":"2024-07-26T08:36:12.728644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743895+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:36:12.743895+0000","last_peered":"2024-07-26T08:36:12.743895+0000","last_clean":"2024-07-26T08:36:12.743895+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:36:12.743895+0000","last_undegraded":"2024-07-26T08:36:12.743895+0000","last_fullsized":"2024-07-26T08:36:12.743895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719735+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:36:12.719735+0000","last_peered":"2024-07-26T08:36:12.719735+0000","last_clean":"2024-07-26T08:36:12.719735+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:36:12.719735+0000","last_undegraded":"2024-07-26T08:36:12.719735+0000","last_fullsized":"2024-07-26T08:36:12.719735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736112+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:36:12.736112+0000","last_peered":"2024-07-26T08:36:12.736112+0000","last_clean":"2024-07-26T08:36:12.736112+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:36:12.736112+0000","last_undegraded":"2024-07-26T08:36:12.736112+0000","last_fullsized":"2024-07-26T08:36:12.736112+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739102+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:36:12.739102+0000","last_peered":"2024-07-26T08:36:12.739102+0000","last_clean":"2024-07-26T08:36:12.739102+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:36:12.739102+0000","last_undegraded":"2024-07-26T08:36:12.739102+0000","last_fullsized":"2024-07-26T08:36:12.739102+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740316+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:36:12.740316+0000","last_peered":"2024-07-26T08:36:12.740316+0000","last_clean":"2024-07-26T08:36:12.740316+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:36:12.740316+0000","last_undegraded":"2024-07-26T08:36:12.740316+0000","last_fullsized":"2024-07-26T08:36:12.740316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80756,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2497,"num_read_kb":1339,"num_write":269,"num_write_kb":167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":242268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645397,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540314,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841625,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1175552,"data_stored":1204384,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005150,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201381,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397611,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136923,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561134,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757365,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920890,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540017,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378759,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":609849,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542283,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":609097,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":537092,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":75467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:38:40.794 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-07-26T08:38:40.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T08:38:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:40 smithi171 ceph-mon[37789]: pgmap v482: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T08:38:40.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:40 smithi044 ceph-mon[32404]: pgmap v482: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T08:38:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:40 smithi116 ceph-mon[37097]: pgmap v482: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T08:38:41.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 -- 172.21.15.44:0/2746558197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410b7b0 msgr2=0x7ff704111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:41.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 --2- 172.21.15.44:0/2746558197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410b7b0 0x7ff704111fe0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009ca0 tx=0x7ff6fc02a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 -- 172.21.15.44:0/2746558197 shutdown_connections 2024-07-26T08:38:41.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 --2- 172.21.15.44:0/2746558197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70410ae60 0x7ff70410b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 --2- 172.21.15.44:0/2746558197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410b7b0 0x7ff704111fe0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 --2- 172.21.15.44:0/2746558197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff704105530 0x7ff704105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.116+0000 7ff708a00640 1 -- 172.21.15.44:0/2746558197 >> 172.21.15.44:0/2746558197 conn(0x7ff704100f90 msgr2=0x7ff7041033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:41.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.117+0000 7ff708a00640 1 -- 172.21.15.44:0/2746558197 shutdown_connections 2024-07-26T08:38:41.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.117+0000 7ff708a00640 1 -- 172.21.15.44:0/2746558197 wait complete. 2024-07-26T08:38:41.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 Processor -- start 2024-07-26T08:38:41.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 -- start start 2024-07-26T08:38:41.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704105530 0x7ff70419be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:41.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 0x7ff70419c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:41.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70410b7b0 0x7ff7041a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041154d0 con 0x7ff70410ae60 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff704115350 con 0x7ff70410b7b0 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.118+0000 7ff708a00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff704115650 con 0x7ff704105530 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff6faa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 0x7ff70419c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff6faa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 0x7ff70419c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49258/0 (socket says 172.21.15.44:49258) 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704105530 0x7ff70419be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff6faa00640 1 -- 172.21.15.44:0/2372493929 learned_addr learned my addr 172.21.15.44:0/2372493929 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703e00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70410b7b0 0x7ff7041a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703400640 1 -- 172.21.15.44:0/2372493929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70410b7b0 msgr2=0x7ff7041a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703400640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70410b7b0 0x7ff7041a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703400640 1 -- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 msgr2=0x7ff70419c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703400640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 0x7ff70419c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff703400640 1 -- 172.21.15.44:0/2372493929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7041a1d40 con 0x7ff704105530 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.119+0000 7ff6faa00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 0x7ff70419c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.120+0000 7ff703400640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704105530 0x7ff70419be10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400e9f0 tx=0x7ff6f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.120+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f400ccc0 con 0x7ff704105530 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.120+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6f40108e0 con 0x7ff704105530 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.120+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4018880 con 0x7ff704105530 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.120+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7041a2030 con 0x7ff704105530 2024-07-26T08:38:41.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.120+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7041a87c0 con 0x7ff704105530 2024-07-26T08:38:41.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.121+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff704105940 con 0x7ff704105530 2024-07-26T08:38:41.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.124+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7ff6f4010430 con 0x7ff704105530 2024-07-26T08:38:41.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.125+0000 7ff700c00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff6e80b3c20 0x7ff6e80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:41.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.125+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ff6f40e48e0 con 0x7ff704105530 2024-07-26T08:38:41.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.126+0000 7ff6faa00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff6e80b3c20 0x7ff6e80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:41.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.126+0000 7ff6faa00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff6e80b3c20 0x7ff6e80b6130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff70406a370 tx=0x7ff6ec005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:41.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.127+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff6f40ab3d0 con 0x7ff704105530 2024-07-26T08:38:41.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.340+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff704063230 con 0x7ff6e80b3c20 2024-07-26T08:38:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.347+0000 7ff700c00640 1 -- 172.21.15.44:0/2372493929 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340810 (secure 0 0 0) 0x7ff704063230 con 0x7ff6e80b3c20 2024-07-26T08:38:41.351 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-07-26T08:38:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff6e80b3c20 msgr2=0x7ff6e80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff6e80b3c20 0x7ff6e80b6130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff70406a370 tx=0x7ff6ec005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704105530 msgr2=0x7ff70419be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704105530 0x7ff70419be10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400e9f0 tx=0x7ff6f400eeb0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 shutdown_connections 2024-07-26T08:38:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ff6e80b3c20 0x7ff6e80b6130 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff70410b7b0 0x7ff7041a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff70410ae60 0x7ff70419c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 --2- 172.21.15.44:0/2372493929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704105530 0x7ff70419be10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 >> 172.21.15.44:0/2372493929 conn(0x7ff704100f90 msgr2=0x7ff7041033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.355+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 shutdown_connections 2024-07-26T08:38:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:41.356+0000 7ff708a00640 1 -- 172.21.15.44:0/2372493929 wait complete. 2024-07-26T08:38:41.384 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-07-26T08:38:40.919450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671358,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2583,"num_read_kb":1408,"num_write":394,"num_write_kb":2319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":826,"ondisk_log_size":826,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365740,"kb_used_data":12428,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576084,"statfs":{"total":1151940427776,"available":1151565910016,"internally_reserved":0,"allocated":12726272,"data_stored":8458724,"data_compressed":502245,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19139,"internal_metadata":361674045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":108,"num_read_kb":58,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006296"},"pg_stats":[{"pgid":"2.1f","version":"91'13","reported_seq":305,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.693455+0000","last_change":"2024-07-26T08:32:58.095306+0000","last_active":"2024-07-26T08:38:34.693455+0000","last_peered":"2024-07-26T08:38:34.693455+0000","last_clean":"2024-07-26T08:38:34.693455+0000","last_became_active":"2024-07-26T08:32:58.094833+0000","last_became_peered":"2024-07-26T08:32:58.094833+0000","last_unstale":"2024-07-26T08:38:34.693455+0000","last_undegraded":"2024-07-26T08:38:34.693455+0000","last_fullsized":"2024-07-26T08:38:34.693455+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:02.643314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":245,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746837+0000","last_change":"2024-07-26T08:33:03.150185+0000","last_active":"2024-07-26T08:36:12.746837+0000","last_peered":"2024-07-26T08:36:12.746837+0000","last_clean":"2024-07-26T08:36:12.746837+0000","last_became_active":"2024-07-26T08:33:03.149785+0000","last_became_peered":"2024-07-26T08:33:03.149785+0000","last_unstale":"2024-07-26T08:36:12.746837+0000","last_undegraded":"2024-07-26T08:36:12.746837+0000","last_fullsized":"2024-07-26T08:36:12.746837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:34:57.980033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:45.161843+0000","last_change":"2024-07-26T08:33:17.890624+0000","last_active":"2024-07-26T08:35:45.161843+0000","last_peered":"2024-07-26T08:35:45.161843+0000","last_clean":"2024-07-26T08:35:45.161843+0000","last_became_active":"2024-07-26T08:33:17.890143+0000","last_became_peered":"2024-07-26T08:33:17.890143+0000","last_unstale":"2024-07-26T08:35:45.161843+0000","last_undegraded":"2024-07-26T08:35:45.161843+0000","last_fullsized":"2024-07-26T08:35:45.161843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:45.081192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:57.270075+0000","last_change":"2024-07-26T08:32:58.233587+0000","last_active":"2024-07-26T08:34:57.270075+0000","last_peered":"2024-07-26T08:34:57.270075+0000","last_clean":"2024-07-26T08:34:57.270075+0000","last_became_active":"2024-07-26T08:32:58.229734+0000","last_became_peered":"2024-07-26T08:32:58.229734+0000","last_unstale":"2024-07-26T08:34:57.270075+0000","last_undegraded":"2024-07-26T08:34:57.270075+0000","last_fullsized":"2024-07-26T08:34:57.270075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:33.776874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741337+0000","last_change":"2024-07-26T08:33:03.166670+0000","last_active":"2024-07-26T08:36:12.741337+0000","last_peered":"2024-07-26T08:36:12.741337+0000","last_clean":"2024-07-26T08:36:12.741337+0000","last_became_active":"2024-07-26T08:33:03.166106+0000","last_became_peered":"2024-07-26T08:33:03.166106+0000","last_unstale":"2024-07-26T08:36:12.741337+0000","last_undegraded":"2024-07-26T08:36:12.741337+0000","last_fullsized":"2024-07-26T08:36:12.741337+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:27.889829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:43.579910+0000","last_change":"2024-07-26T08:34:14.593108+0000","last_active":"2024-07-26T08:35:43.579910+0000","last_peered":"2024-07-26T08:35:43.579910+0000","last_clean":"2024-07-26T08:35:43.579910+0000","last_became_active":"2024-07-26T08:34:14.592702+0000","last_became_peered":"2024-07-26T08:34:14.592702+0000","last_unstale":"2024-07-26T08:35:43.579910+0000","last_undegraded":"2024-07-26T08:35:43.579910+0000","last_fullsized":"2024-07-26T08:35:43.579910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:58.734636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:30.595827+0000","last_change":"2024-07-26T08:32:58.093403+0000","last_active":"2024-07-26T08:35:30.595827+0000","last_peered":"2024-07-26T08:35:30.595827+0000","last_clean":"2024-07-26T08:35:30.595827+0000","last_became_active":"2024-07-26T08:32:58.093004+0000","last_became_peered":"2024-07-26T08:32:58.093004+0000","last_unstale":"2024-07-26T08:35:30.595827+0000","last_undegraded":"2024-07-26T08:35:30.595827+0000","last_fullsized":"2024-07-26T08:35:30.595827+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:38:19.667582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741380+0000","last_change":"2024-07-26T08:33:03.141522+0000","last_active":"2024-07-26T08:36:12.741380+0000","last_peered":"2024-07-26T08:36:12.741380+0000","last_clean":"2024-07-26T08:36:12.741380+0000","last_became_active":"2024-07-26T08:33:03.140783+0000","last_became_peered":"2024-07-26T08:33:03.140783+0000","last_unstale":"2024-07-26T08:36:12.741380+0000","last_undegraded":"2024-07-26T08:36:12.741380+0000","last_fullsized":"2024-07-26T08:36:12.741380+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:51.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:48.580551+0000","last_change":"2024-07-26T08:33:17.888474+0000","last_active":"2024-07-26T08:35:48.580551+0000","last_peered":"2024-07-26T08:35:48.580551+0000","last_clean":"2024-07-26T08:35:48.580551+0000","last_became_active":"2024-07-26T08:33:17.887916+0000","last_became_peered":"2024-07-26T08:33:17.887916+0000","last_unstale":"2024-07-26T08:35:48.580551+0000","last_undegraded":"2024-07-26T08:35:48.580551+0000","last_fullsized":"2024-07-26T08:35:48.580551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:59:17.928875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:30.595776+0000","last_change":"2024-07-26T08:32:58.091883+0000","last_active":"2024-07-26T08:35:30.595776+0000","last_peered":"2024-07-26T08:35:30.595776+0000","last_clean":"2024-07-26T08:35:30.595776+0000","last_became_active":"2024-07-26T08:32:58.091314+0000","last_became_peered":"2024-07-26T08:32:58.091314+0000","last_unstale":"2024-07-26T08:35:30.595776+0000","last_undegraded":"2024-07-26T08:35:30.595776+0000","last_fullsized":"2024-07-26T08:35:30.595776+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:02:06.624624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734806+0000","last_change":"2024-07-26T08:33:03.143508+0000","last_active":"2024-07-26T08:36:12.734806+0000","last_peered":"2024-07-26T08:36:12.734806+0000","last_clean":"2024-07-26T08:36:12.734806+0000","last_became_active":"2024-07-26T08:33:03.142107+0000","last_became_peered":"2024-07-26T08:33:03.142107+0000","last_unstale":"2024-07-26T08:36:12.734806+0000","last_undegraded":"2024-07-26T08:36:12.734806+0000","last_fullsized":"2024-07-26T08:36:12.734806+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:30.293831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:48.580530+0000","last_change":"2024-07-26T08:33:17.899781+0000","last_active":"2024-07-26T08:35:48.580530+0000","last_peered":"2024-07-26T08:35:48.580530+0000","last_clean":"2024-07-26T08:35:48.580530+0000","last_became_active":"2024-07-26T08:33:17.899378+0000","last_became_peered":"2024-07-26T08:33:17.899378+0000","last_unstale":"2024-07-26T08:35:48.580530+0000","last_undegraded":"2024-07-26T08:35:48.580530+0000","last_fullsized":"2024-07-26T08:35:48.580530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:35.519181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742218+0000","last_change":"2024-07-26T08:33:03.144747+0000","last_active":"2024-07-26T08:36:12.742218+0000","last_peered":"2024-07-26T08:36:12.742218+0000","last_clean":"2024-07-26T08:36:12.742218+0000","last_became_active":"2024-07-26T08:33:03.144258+0000","last_became_peered":"2024-07-26T08:33:03.144258+0000","last_unstale":"2024-07-26T08:36:12.742218+0000","last_undegraded":"2024-07-26T08:36:12.742218+0000","last_fullsized":"2024-07-26T08:36:12.742218+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:30:48.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:15.594066+0000","last_change":"2024-07-26T08:32:58.094646+0000","last_active":"2024-07-26T08:35:15.594066+0000","last_peered":"2024-07-26T08:35:15.594066+0000","last_clean":"2024-07-26T08:35:15.594066+0000","last_became_active":"2024-07-26T08:32:58.093923+0000","last_became_peered":"2024-07-26T08:32:58.093923+0000","last_unstale":"2024-07-26T08:35:15.594066+0000","last_undegraded":"2024-07-26T08:35:15.594066+0000","last_fullsized":"2024-07-26T08:35:15.594066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:25.129777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578665+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:35:33.578665+0000","last_peered":"2024-07-26T08:35:33.578665+0000","last_clean":"2024-07-26T08:35:33.578665+0000","last_became_active":"2024-07-26T08:33:17.894243+0000","last_became_peered":"2024-07-26T08:33:17.894243+0000","last_unstale":"2024-07-26T08:35:33.578665+0000","last_undegraded":"2024-07-26T08:35:33.578665+0000","last_fullsized":"2024-07-26T08:35:33.578665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:05.673091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744388+0000","last_change":"2024-07-26T08:33:03.160897+0000","last_active":"2024-07-26T08:36:12.744388+0000","last_peered":"2024-07-26T08:36:12.744388+0000","last_clean":"2024-07-26T08:36:12.744388+0000","last_became_active":"2024-07-26T08:33:03.160514+0000","last_became_peered":"2024-07-26T08:33:03.160514+0000","last_unstale":"2024-07-26T08:36:12.744388+0000","last_undegraded":"2024-07-26T08:36:12.744388+0000","last_fullsized":"2024-07-26T08:36:12.744388+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:12:59.135145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320730+0000","last_change":"2024-07-26T08:33:13.832138+0000","last_active":"2024-07-26T08:35:26.320730+0000","last_peered":"2024-07-26T08:35:26.320730+0000","last_clean":"2024-07-26T08:35:26.320730+0000","last_became_active":"2024-07-26T08:33:13.831765+0000","last_became_peered":"2024-07-26T08:33:13.831765+0000","last_unstale":"2024-07-26T08:35:26.320730+0000","last_undegraded":"2024-07-26T08:35:26.320730+0000","last_fullsized":"2024-07-26T08:35:26.320730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:50:22.410721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,4],"acting":[11,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578615+0000","last_change":"2024-07-26T08:34:14.417150+0000","last_active":"2024-07-26T08:35:33.578615+0000","last_peered":"2024-07-26T08:35:33.578615+0000","last_clean":"2024-07-26T08:35:33.578615+0000","last_became_active":"2024-07-26T08:34:14.416508+0000","last_became_peered":"2024-07-26T08:34:14.416508+0000","last_unstale":"2024-07-26T08:35:33.578615+0000","last_undegraded":"2024-07-26T08:35:33.578615+0000","last_fullsized":"2024-07-26T08:35:33.578615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:26.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740319+0000","last_change":"2024-07-26T08:33:03.153273+0000","last_active":"2024-07-26T08:36:12.740319+0000","last_peered":"2024-07-26T08:36:12.740319+0000","last_clean":"2024-07-26T08:36:12.740319+0000","last_became_active":"2024-07-26T08:33:03.152779+0000","last_became_peered":"2024-07-26T08:33:03.152779+0000","last_unstale":"2024-07-26T08:36:12.740319+0000","last_undegraded":"2024-07-26T08:36:12.740319+0000","last_fullsized":"2024-07-26T08:36:12.740319+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:43:33.014571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.272221+0000","last_change":"2024-07-26T08:32:58.098499+0000","last_active":"2024-07-26T08:35:07.272221+0000","last_peered":"2024-07-26T08:35:07.272221+0000","last_clean":"2024-07-26T08:35:07.272221+0000","last_became_active":"2024-07-26T08:32:58.098036+0000","last_became_peered":"2024-07-26T08:32:58.098036+0000","last_unstale":"2024-07-26T08:35:07.272221+0000","last_undegraded":"2024-07-26T08:35:07.272221+0000","last_fullsized":"2024-07-26T08:35:07.272221+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:32:58.249409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:36.345867+0000","last_change":"2024-07-26T08:34:14.594699+0000","last_active":"2024-07-26T08:35:36.345867+0000","last_peered":"2024-07-26T08:35:36.345867+0000","last_clean":"2024-07-26T08:35:36.345867+0000","last_became_active":"2024-07-26T08:34:14.593561+0000","last_became_peered":"2024-07-26T08:34:14.593561+0000","last_unstale":"2024-07-26T08:35:36.345867+0000","last_undegraded":"2024-07-26T08:35:36.345867+0000","last_fullsized":"2024-07-26T08:35:36.345867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:30:30.746688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.712601+0000","last_change":"2024-07-26T08:33:03.135573+0000","last_active":"2024-07-26T08:36:12.712601+0000","last_peered":"2024-07-26T08:36:12.712601+0000","last_clean":"2024-07-26T08:36:12.712601+0000","last_became_active":"2024-07-26T08:33:03.135185+0000","last_became_peered":"2024-07-26T08:33:03.135185+0000","last_unstale":"2024-07-26T08:36:12.712601+0000","last_undegraded":"2024-07-26T08:36:12.712601+0000","last_fullsized":"2024-07-26T08:36:12.712601+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:54:05.161887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:07.548672+0000","last_change":"2024-07-26T08:32:58.225269+0000","last_active":"2024-07-26T08:35:07.548672+0000","last_peered":"2024-07-26T08:35:07.548672+0000","last_clean":"2024-07-26T08:35:07.548672+0000","last_became_active":"2024-07-26T08:32:58.224881+0000","last_became_peered":"2024-07-26T08:32:58.224881+0000","last_unstale":"2024-07-26T08:35:07.548672+0000","last_undegraded":"2024-07-26T08:35:07.548672+0000","last_fullsized":"2024-07-26T08:35:07.548672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:08:21.517912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:33.578686+0000","last_change":"2024-07-26T08:33:17.895424+0000","last_active":"2024-07-26T08:35:33.578686+0000","last_peered":"2024-07-26T08:35:33.578686+0000","last_clean":"2024-07-26T08:35:33.578686+0000","last_became_active":"2024-07-26T08:33:17.894734+0000","last_became_peered":"2024-07-26T08:33:17.894734+0000","last_unstale":"2024-07-26T08:35:33.578686+0000","last_undegraded":"2024-07-26T08:35:33.578686+0000","last_fullsized":"2024-07-26T08:35:33.578686+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:23:06.150974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.747319+0000","last_change":"2024-07-26T08:33:03.162936+0000","last_active":"2024-07-26T08:36:12.747319+0000","last_peered":"2024-07-26T08:36:12.747319+0000","last_clean":"2024-07-26T08:36:12.747319+0000","last_became_active":"2024-07-26T08:33:03.162543+0000","last_became_peered":"2024-07-26T08:33:03.162543+0000","last_unstale":"2024-07-26T08:36:12.747319+0000","last_undegraded":"2024-07-26T08:36:12.747319+0000","last_fullsized":"2024-07-26T08:36:12.747319+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:29.186844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:08.575512+0000","last_change":"2024-07-26T08:32:58.309672+0000","last_active":"2024-07-26T08:35:08.575512+0000","last_peered":"2024-07-26T08:35:08.575512+0000","last_clean":"2024-07-26T08:35:08.575512+0000","last_became_active":"2024-07-26T08:32:58.309207+0000","last_became_peered":"2024-07-26T08:32:58.309207+0000","last_unstale":"2024-07-26T08:35:08.575512+0000","last_undegraded":"2024-07-26T08:35:08.575512+0000","last_fullsized":"2024-07-26T08:35:08.575512+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:37.190777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:25.159325+0000","last_change":"2024-07-26T08:33:17.869896+0000","last_active":"2024-07-26T08:35:25.159325+0000","last_peered":"2024-07-26T08:35:25.159325+0000","last_clean":"2024-07-26T08:35:25.159325+0000","last_became_active":"2024-07-26T08:33:17.869447+0000","last_became_peered":"2024-07-26T08:33:17.869447+0000","last_unstale":"2024-07-26T08:35:25.159325+0000","last_undegraded":"2024-07-26T08:35:25.159325+0000","last_fullsized":"2024-07-26T08:35:25.159325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:51.572604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728535+0000","last_change":"2024-07-26T08:33:13.825662+0000","last_active":"2024-07-26T08:36:12.728535+0000","last_peered":"2024-07-26T08:36:12.728535+0000","last_clean":"2024-07-26T08:36:12.728535+0000","last_became_active":"2024-07-26T08:33:13.821997+0000","last_became_peered":"2024-07-26T08:33:13.821997+0000","last_unstale":"2024-07-26T08:36:12.728535+0000","last_undegraded":"2024-07-26T08:36:12.728535+0000","last_fullsized":"2024-07-26T08:36:12.728535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:41:21.488133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"91'18","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:37:21.364570+0000","last_change":"2024-07-26T08:32:58.098391+0000","last_active":"2024-07-26T08:37:21.364570+0000","last_peered":"2024-07-26T08:37:21.364570+0000","last_clean":"2024-07-26T08:37:21.364570+0000","last_became_active":"2024-07-26T08:32:58.097957+0000","last_became_peered":"2024-07-26T08:32:58.097957+0000","last_unstale":"2024-07-26T08:37:21.364570+0000","last_undegraded":"2024-07-26T08:37:21.364570+0000","last_fullsized":"2024-07-26T08:37:21.364570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:34:04.898903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:17.274168+0000","last_change":"2024-07-26T08:33:17.895094+0000","last_active":"2024-07-26T08:35:17.274168+0000","last_peered":"2024-07-26T08:35:17.274168+0000","last_clean":"2024-07-26T08:35:17.274168+0000","last_became_active":"2024-07-26T08:33:17.894496+0000","last_became_peered":"2024-07-26T08:33:17.894496+0000","last_unstale":"2024-07-26T08:35:17.274168+0000","last_undegraded":"2024-07-26T08:35:17.274168+0000","last_fullsized":"2024-07-26T08:35:17.274168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:45:36.577293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734147+0000","last_change":"2024-07-26T08:33:03.148918+0000","last_active":"2024-07-26T08:36:12.734147+0000","last_peered":"2024-07-26T08:36:12.734147+0000","last_clean":"2024-07-26T08:36:12.734147+0000","last_became_active":"2024-07-26T08:33:03.148454+0000","last_became_peered":"2024-07-26T08:33:03.148454+0000","last_unstale":"2024-07-26T08:36:12.734147+0000","last_undegraded":"2024-07-26T08:36:12.734147+0000","last_fullsized":"2024-07-26T08:36:12.734147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:29.089233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:05.592648+0000","last_change":"2024-07-26T08:32:58.228202+0000","last_active":"2024-07-26T08:35:05.592648+0000","last_peered":"2024-07-26T08:35:05.592648+0000","last_clean":"2024-07-26T08:35:05.592648+0000","last_became_active":"2024-07-26T08:32:58.227754+0000","last_became_peered":"2024-07-26T08:32:58.227754+0000","last_unstale":"2024-07-26T08:35:05.592648+0000","last_undegraded":"2024-07-26T08:35:05.592648+0000","last_fullsized":"2024-07-26T08:35:05.592648+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:18:09.952547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320776+0000","last_change":"2024-07-26T08:33:17.884482+0000","last_active":"2024-07-26T08:35:26.320776+0000","last_peered":"2024-07-26T08:35:26.320776+0000","last_clean":"2024-07-26T08:35:26.320776+0000","last_became_active":"2024-07-26T08:33:17.883990+0000","last_became_peered":"2024-07-26T08:33:17.883990+0000","last_unstale":"2024-07-26T08:35:26.320776+0000","last_undegraded":"2024-07-26T08:35:26.320776+0000","last_fullsized":"2024-07-26T08:35:26.320776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:10.316974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744052+0000","last_change":"2024-07-26T08:33:03.153810+0000","last_active":"2024-07-26T08:36:12.744052+0000","last_peered":"2024-07-26T08:36:12.744052+0000","last_clean":"2024-07-26T08:36:12.744052+0000","last_became_active":"2024-07-26T08:33:03.153407+0000","last_became_peered":"2024-07-26T08:33:03.153407+0000","last_unstale":"2024-07-26T08:36:12.744052+0000","last_undegraded":"2024-07-26T08:36:12.744052+0000","last_fullsized":"2024-07-26T08:36:12.744052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:00:00.266252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:06.339791+0000","last_change":"2024-07-26T08:32:58.231259+0000","last_active":"2024-07-26T08:35:06.339791+0000","last_peered":"2024-07-26T08:35:06.339791+0000","last_clean":"2024-07-26T08:35:06.339791+0000","last_became_active":"2024-07-26T08:32:58.229184+0000","last_became_peered":"2024-07-26T08:32:58.229184+0000","last_unstale":"2024-07-26T08:35:06.339791+0000","last_undegraded":"2024-07-26T08:35:06.339791+0000","last_fullsized":"2024-07-26T08:35:06.339791+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:03:44.080581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:25.595196+0000","last_change":"2024-07-26T08:33:17.893610+0000","last_active":"2024-07-26T08:35:25.595196+0000","last_peered":"2024-07-26T08:35:25.595196+0000","last_clean":"2024-07-26T08:35:25.595196+0000","last_became_active":"2024-07-26T08:33:17.893265+0000","last_became_peered":"2024-07-26T08:33:17.893265+0000","last_unstale":"2024-07-26T08:35:25.595196+0000","last_undegraded":"2024-07-26T08:35:25.595196+0000","last_fullsized":"2024-07-26T08:35:25.595196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:13:03.532660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740048+0000","last_change":"2024-07-26T08:33:03.159635+0000","last_active":"2024-07-26T08:36:12.740048+0000","last_peered":"2024-07-26T08:36:12.740048+0000","last_clean":"2024-07-26T08:36:12.740048+0000","last_became_active":"2024-07-26T08:33:03.159212+0000","last_became_peered":"2024-07-26T08:33:03.159212+0000","last_unstale":"2024-07-26T08:36:12.740048+0000","last_undegraded":"2024-07-26T08:36:12.740048+0000","last_fullsized":"2024-07-26T08:36:12.740048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:52:08.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:35.696701+0000","last_change":"2024-07-26T08:32:58.096452+0000","last_active":"2024-07-26T08:38:35.696701+0000","last_peered":"2024-07-26T08:38:35.696701+0000","last_clean":"2024-07-26T08:38:35.696701+0000","last_became_active":"2024-07-26T08:32:58.095588+0000","last_became_peered":"2024-07-26T08:32:58.095588+0000","last_unstale":"2024-07-26T08:38:35.696701+0000","last_undegraded":"2024-07-26T08:38:35.696701+0000","last_fullsized":"2024-07-26T08:38:35.696701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:15.265059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":246,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.566680+0000","last_change":"2024-07-26T08:33:17.898367+0000","last_active":"2024-07-26T08:35:26.566680+0000","last_peered":"2024-07-26T08:35:26.566680+0000","last_clean":"2024-07-26T08:35:26.566680+0000","last_became_active":"2024-07-26T08:33:17.896669+0000","last_became_peered":"2024-07-26T08:33:17.896669+0000","last_unstale":"2024-07-26T08:35:26.566680+0000","last_undegraded":"2024-07-26T08:35:26.566680+0000","last_fullsized":"2024-07-26T08:35:26.566680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:21.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729599+0000","last_change":"2024-07-26T08:33:13.822598+0000","last_active":"2024-07-26T08:36:12.729599+0000","last_peered":"2024-07-26T08:36:12.729599+0000","last_clean":"2024-07-26T08:36:12.729599+0000","last_became_active":"2024-07-26T08:33:13.821252+0000","last_became_peered":"2024-07-26T08:33:13.821252+0000","last_unstale":"2024-07-26T08:36:12.729599+0000","last_undegraded":"2024-07-26T08:36:12.729599+0000","last_fullsized":"2024-07-26T08:36:12.729599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:14:38.377751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.779037+0000","last_change":"2024-07-26T08:32:58.232125+0000","last_active":"2024-07-26T08:38:34.779037+0000","last_peered":"2024-07-26T08:38:34.779037+0000","last_clean":"2024-07-26T08:38:34.779037+0000","last_became_active":"2024-07-26T08:32:58.230603+0000","last_became_peered":"2024-07-26T08:32:58.230603+0000","last_unstale":"2024-07-26T08:38:34.779037+0000","last_undegraded":"2024-07-26T08:38:34.779037+0000","last_fullsized":"2024-07-26T08:38:34.779037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:45:12.273634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":245,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.567573+0000","last_change":"2024-07-26T08:33:17.901735+0000","last_active":"2024-07-26T08:35:31.567573+0000","last_peered":"2024-07-26T08:35:31.567573+0000","last_clean":"2024-07-26T08:35:31.567573+0000","last_became_active":"2024-07-26T08:33:17.898614+0000","last_became_peered":"2024-07-26T08:33:17.898614+0000","last_unstale":"2024-07-26T08:35:31.567573+0000","last_undegraded":"2024-07-26T08:35:31.567573+0000","last_fullsized":"2024-07-26T08:35:31.567573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:53:43.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733996+0000","last_change":"2024-07-26T08:33:03.148745+0000","last_active":"2024-07-26T08:36:12.733996+0000","last_peered":"2024-07-26T08:36:12.733996+0000","last_clean":"2024-07-26T08:36:12.733996+0000","last_became_active":"2024-07-26T08:33:03.146510+0000","last_became_peered":"2024-07-26T08:33:03.146510+0000","last_unstale":"2024-07-26T08:36:12.733996+0000","last_undegraded":"2024-07-26T08:36:12.733996+0000","last_fullsized":"2024-07-26T08:36:12.733996+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:59:57.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:57.422284+0000","last_change":"2024-07-26T08:32:58.093350+0000","last_active":"2024-07-26T08:34:57.422284+0000","last_peered":"2024-07-26T08:34:57.422284+0000","last_clean":"2024-07-26T08:34:57.422284+0000","last_became_active":"2024-07-26T08:32:58.092966+0000","last_became_peered":"2024-07-26T08:32:58.092966+0000","last_unstale":"2024-07-26T08:34:57.422284+0000","last_undegraded":"2024-07-26T08:34:57.422284+0000","last_fullsized":"2024-07-26T08:34:57.422284+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:11.103968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406163+0000","last_change":"2024-07-26T08:33:17.894704+0000","last_active":"2024-07-26T08:35:31.406163+0000","last_peered":"2024-07-26T08:35:31.406163+0000","last_clean":"2024-07-26T08:35:31.406163+0000","last_became_active":"2024-07-26T08:33:17.894272+0000","last_became_peered":"2024-07-26T08:33:17.894272+0000","last_unstale":"2024-07-26T08:35:31.406163+0000","last_undegraded":"2024-07-26T08:35:31.406163+0000","last_fullsized":"2024-07-26T08:35:31.406163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:07.687908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745276+0000","last_change":"2024-07-26T08:33:03.156155+0000","last_active":"2024-07-26T08:36:12.745276+0000","last_peered":"2024-07-26T08:36:12.745276+0000","last_clean":"2024-07-26T08:36:12.745276+0000","last_became_active":"2024-07-26T08:33:03.151322+0000","last_became_peered":"2024-07-26T08:33:03.151322+0000","last_unstale":"2024-07-26T08:36:12.745276+0000","last_undegraded":"2024-07-26T08:36:12.745276+0000","last_fullsized":"2024-07-26T08:36:12.745276+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:30:54.142723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:03.574928+0000","last_change":"2024-07-26T08:32:58.093120+0000","last_active":"2024-07-26T08:35:03.574928+0000","last_peered":"2024-07-26T08:35:03.574928+0000","last_clean":"2024-07-26T08:35:03.574928+0000","last_became_active":"2024-07-26T08:32:58.092615+0000","last_became_peered":"2024-07-26T08:32:58.092615+0000","last_unstale":"2024-07-26T08:35:03.574928+0000","last_undegraded":"2024-07-26T08:35:03.574928+0000","last_fullsized":"2024-07-26T08:35:03.574928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:14.227438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:07.430772+0000","last_change":"2024-07-26T08:31:28.902534+0000","last_active":"2024-07-26T08:36:07.430772+0000","last_peered":"2024-07-26T08:36:07.430772+0000","last_clean":"2024-07-26T08:36:07.430772+0000","last_became_active":"2024-07-26T08:31:28.897816+0000","last_became_peered":"2024-07-26T08:31:28.897816+0000","last_unstale":"2024-07-26T08:36:07.430772+0000","last_undegraded":"2024-07-26T08:36:07.430772+0000","last_fullsized":"2024-07-26T08:36:07.430772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:27:41.550504+0000","last_clean_scrub_stamp":"2024-07-26T08:27:41.550504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:43:06.749608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:22.425289+0000","last_change":"2024-07-26T08:33:17.878029+0000","last_active":"2024-07-26T08:35:22.425289+0000","last_peered":"2024-07-26T08:35:22.425289+0000","last_clean":"2024-07-26T08:35:22.425289+0000","last_became_active":"2024-07-26T08:33:17.877458+0000","last_became_peered":"2024-07-26T08:33:17.877458+0000","last_unstale":"2024-07-26T08:35:22.425289+0000","last_undegraded":"2024-07-26T08:35:22.425289+0000","last_fullsized":"2024-07-26T08:35:22.425289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:15:56.922832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742063+0000","last_change":"2024-07-26T08:33:03.142618+0000","last_active":"2024-07-26T08:36:12.742063+0000","last_peered":"2024-07-26T08:36:12.742063+0000","last_clean":"2024-07-26T08:36:12.742063+0000","last_became_active":"2024-07-26T08:33:03.142241+0000","last_became_peered":"2024-07-26T08:33:03.142241+0000","last_unstale":"2024-07-26T08:36:12.742063+0000","last_undegraded":"2024-07-26T08:36:12.742063+0000","last_fullsized":"2024-07-26T08:36:12.742063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:35:37.249043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:38.504277+0000","last_change":"2024-07-26T08:32:58.309710+0000","last_active":"2024-07-26T08:38:38.504277+0000","last_peered":"2024-07-26T08:38:38.504277+0000","last_clean":"2024-07-26T08:38:38.504277+0000","last_became_active":"2024-07-26T08:32:58.309153+0000","last_became_peered":"2024-07-26T08:32:58.309153+0000","last_unstale":"2024-07-26T08:38:38.504277+0000","last_undegraded":"2024-07-26T08:38:38.504277+0000","last_fullsized":"2024-07-26T08:38:38.504277+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:25:39.766705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:27.551514+0000","last_change":"2024-07-26T08:33:17.889453+0000","last_active":"2024-07-26T08:35:27.551514+0000","last_peered":"2024-07-26T08:35:27.551514+0000","last_clean":"2024-07-26T08:35:27.551514+0000","last_became_active":"2024-07-26T08:33:17.888939+0000","last_became_peered":"2024-07-26T08:33:17.888939+0000","last_unstale":"2024-07-26T08:35:27.551514+0000","last_undegraded":"2024-07-26T08:35:27.551514+0000","last_fullsized":"2024-07-26T08:35:27.551514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:26:23.570278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.744223+0000","last_change":"2024-07-26T08:33:03.147921+0000","last_active":"2024-07-26T08:36:12.744223+0000","last_peered":"2024-07-26T08:36:12.744223+0000","last_clean":"2024-07-26T08:36:12.744223+0000","last_became_active":"2024-07-26T08:33:03.147454+0000","last_became_peered":"2024-07-26T08:33:03.147454+0000","last_unstale":"2024-07-26T08:36:12.744223+0000","last_undegraded":"2024-07-26T08:36:12.744223+0000","last_fullsized":"2024-07-26T08:36:12.744223+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:37:50.489310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:16.404127+0000","last_change":"2024-07-26T08:32:58.098289+0000","last_active":"2024-07-26T08:35:16.404127+0000","last_peered":"2024-07-26T08:35:16.404127+0000","last_clean":"2024-07-26T08:35:16.404127+0000","last_became_active":"2024-07-26T08:32:58.097703+0000","last_became_peered":"2024-07-26T08:32:58.097703+0000","last_unstale":"2024-07-26T08:35:16.404127+0000","last_undegraded":"2024-07-26T08:35:16.404127+0000","last_fullsized":"2024-07-26T08:35:16.404127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:24:52.099608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:32.864361+0000","last_change":"2024-07-26T08:33:17.889744+0000","last_active":"2024-07-26T08:35:32.864361+0000","last_peered":"2024-07-26T08:35:32.864361+0000","last_clean":"2024-07-26T08:35:32.864361+0000","last_became_active":"2024-07-26T08:33:17.887160+0000","last_became_peered":"2024-07-26T08:33:17.887160+0000","last_unstale":"2024-07-26T08:35:32.864361+0000","last_undegraded":"2024-07-26T08:35:32.864361+0000","last_fullsized":"2024-07-26T08:35:32.864361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:26:17.415301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727635+0000","last_change":"2024-07-26T08:33:03.136112+0000","last_active":"2024-07-26T08:36:12.727635+0000","last_peered":"2024-07-26T08:36:12.727635+0000","last_clean":"2024-07-26T08:36:12.727635+0000","last_became_active":"2024-07-26T08:33:03.135728+0000","last_became_peered":"2024-07-26T08:33:03.135728+0000","last_unstale":"2024-07-26T08:36:12.727635+0000","last_undegraded":"2024-07-26T08:36:12.727635+0000","last_fullsized":"2024-07-26T08:36:12.727635+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:16.848682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:16.565096+0000","last_change":"2024-07-26T08:33:13.838170+0000","last_active":"2024-07-26T08:35:16.565096+0000","last_peered":"2024-07-26T08:35:16.565096+0000","last_clean":"2024-07-26T08:35:16.565096+0000","last_became_active":"2024-07-26T08:33:13.832974+0000","last_became_peered":"2024-07-26T08:33:13.832974+0000","last_unstale":"2024-07-26T08:35:16.565096+0000","last_undegraded":"2024-07-26T08:35:16.565096+0000","last_fullsized":"2024-07-26T08:35:16.565096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:59:22.111969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406097+0000","last_change":"2024-07-26T08:33:17.871498+0000","last_active":"2024-07-26T08:35:31.406097+0000","last_peered":"2024-07-26T08:35:31.406097+0000","last_clean":"2024-07-26T08:35:31.406097+0000","last_became_active":"2024-07-26T08:33:17.870854+0000","last_became_peered":"2024-07-26T08:33:17.870854+0000","last_unstale":"2024-07-26T08:35:31.406097+0000","last_undegraded":"2024-07-26T08:35:31.406097+0000","last_fullsized":"2024-07-26T08:35:31.406097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:39.795241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.722492+0000","last_change":"2024-07-26T08:33:03.138327+0000","last_active":"2024-07-26T08:36:12.722492+0000","last_peered":"2024-07-26T08:36:12.722492+0000","last_clean":"2024-07-26T08:36:12.722492+0000","last_became_active":"2024-07-26T08:33:03.137631+0000","last_became_peered":"2024-07-26T08:33:03.137631+0000","last_unstale":"2024-07-26T08:36:12.722492+0000","last_undegraded":"2024-07-26T08:36:12.722492+0000","last_fullsized":"2024-07-26T08:36:12.722492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:15.021998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:15.594107+0000","last_change":"2024-07-26T08:32:58.094374+0000","last_active":"2024-07-26T08:35:15.594107+0000","last_peered":"2024-07-26T08:35:15.594107+0000","last_clean":"2024-07-26T08:35:15.594107+0000","last_became_active":"2024-07-26T08:32:58.093942+0000","last_became_peered":"2024-07-26T08:32:58.093942+0000","last_unstale":"2024-07-26T08:35:15.594107+0000","last_undegraded":"2024-07-26T08:35:15.594107+0000","last_fullsized":"2024-07-26T08:35:15.594107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:57:37.080433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:31.406127+0000","last_change":"2024-07-26T08:33:17.885627+0000","last_active":"2024-07-26T08:35:31.406127+0000","last_peered":"2024-07-26T08:35:31.406127+0000","last_clean":"2024-07-26T08:35:31.406127+0000","last_became_active":"2024-07-26T08:33:17.885184+0000","last_became_peered":"2024-07-26T08:33:17.885184+0000","last_unstale":"2024-07-26T08:35:31.406127+0000","last_undegraded":"2024-07-26T08:35:31.406127+0000","last_fullsized":"2024-07-26T08:35:31.406127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:29:49.729773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733854+0000","last_change":"2024-07-26T08:33:13.821677+0000","last_active":"2024-07-26T08:36:12.733854+0000","last_peered":"2024-07-26T08:36:12.733854+0000","last_clean":"2024-07-26T08:36:12.733854+0000","last_became_active":"2024-07-26T08:33:13.821260+0000","last_became_peered":"2024-07-26T08:33:13.821260+0000","last_unstale":"2024-07-26T08:36:12.733854+0000","last_undegraded":"2024-07-26T08:36:12.733854+0000","last_fullsized":"2024-07-26T08:36:12.733854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:54.920933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:34:32.108992+0000","last_change":"2024-07-26T08:34:13.978988+0000","last_active":"2024-07-26T08:34:32.108992+0000","last_peered":"2024-07-26T08:34:32.108992+0000","last_clean":"2024-07-26T08:34:32.108992+0000","last_became_active":"2024-07-26T08:34:13.971046+0000","last_became_peered":"2024-07-26T08:34:13.971046+0000","last_unstale":"2024-07-26T08:34:32.108992+0000","last_undegraded":"2024-07-26T08:34:32.108992+0000","last_fullsized":"2024-07-26T08:34:32.108992+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:57:31.599516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:46.570284+0000","last_change":"2024-07-26T08:33:17.899455+0000","last_active":"2024-07-26T08:35:46.570284+0000","last_peered":"2024-07-26T08:35:46.570284+0000","last_clean":"2024-07-26T08:35:46.570284+0000","last_became_active":"2024-07-26T08:33:17.895516+0000","last_became_peered":"2024-07-26T08:33:17.895516+0000","last_unstale":"2024-07-26T08:35:46.570284+0000","last_undegraded":"2024-07-26T08:35:46.570284+0000","last_fullsized":"2024-07-26T08:35:46.570284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:48:38.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.731664+0000","last_change":"2024-07-26T08:33:03.135949+0000","last_active":"2024-07-26T08:36:12.731664+0000","last_peered":"2024-07-26T08:36:12.731664+0000","last_clean":"2024-07-26T08:36:12.731664+0000","last_became_active":"2024-07-26T08:33:03.135558+0000","last_became_peered":"2024-07-26T08:33:03.135558+0000","last_unstale":"2024-07-26T08:36:12.731664+0000","last_undegraded":"2024-07-26T08:36:12.731664+0000","last_fullsized":"2024-07-26T08:36:12.731664+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:48:48.067244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'8","reported_seq":306,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:36.784851+0000","last_change":"2024-07-26T08:32:58.092950+0000","last_active":"2024-07-26T08:38:36.784851+0000","last_peered":"2024-07-26T08:38:36.784851+0000","last_clean":"2024-07-26T08:38:36.784851+0000","last_became_active":"2024-07-26T08:32:58.092450+0000","last_became_peered":"2024-07-26T08:32:58.092450+0000","last_unstale":"2024-07-26T08:38:36.784851+0000","last_undegraded":"2024-07-26T08:38:36.784851+0000","last_fullsized":"2024-07-26T08:38:36.784851+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:10.687760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1427,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:46.348069+0000","last_change":"2024-07-26T08:33:17.891927+0000","last_active":"2024-07-26T08:35:46.348069+0000","last_peered":"2024-07-26T08:35:46.348069+0000","last_clean":"2024-07-26T08:35:46.348069+0000","last_became_active":"2024-07-26T08:33:17.891351+0000","last_became_peered":"2024-07-26T08:33:17.891351+0000","last_unstale":"2024-07-26T08:35:46.348069+0000","last_undegraded":"2024-07-26T08:35:46.348069+0000","last_fullsized":"2024-07-26T08:35:46.348069+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:45:33.019340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.734066+0000","last_change":"2024-07-26T08:33:03.145129+0000","last_active":"2024-07-26T08:36:12.734066+0000","last_peered":"2024-07-26T08:36:12.734066+0000","last_clean":"2024-07-26T08:36:12.734066+0000","last_became_active":"2024-07-26T08:33:03.144518+0000","last_became_peered":"2024-07-26T08:33:03.144518+0000","last_unstale":"2024-07-26T08:36:12.734066+0000","last_undegraded":"2024-07-26T08:36:12.734066+0000","last_fullsized":"2024-07-26T08:36:12.734066+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:16:36.420811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:21.404762+0000","last_change":"2024-07-26T08:32:58.097663+0000","last_active":"2024-07-26T08:35:21.404762+0000","last_peered":"2024-07-26T08:35:21.404762+0000","last_clean":"2024-07-26T08:35:21.404762+0000","last_became_active":"2024-07-26T08:32:58.097231+0000","last_became_peered":"2024-07-26T08:32:58.097231+0000","last_unstale":"2024-07-26T08:35:21.404762+0000","last_undegraded":"2024-07-26T08:35:21.404762+0000","last_fullsized":"2024-07-26T08:35:21.404762+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:36.343200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.322567+0000","last_change":"2024-07-26T08:33:17.889438+0000","last_active":"2024-07-26T08:35:41.322567+0000","last_peered":"2024-07-26T08:35:41.322567+0000","last_clean":"2024-07-26T08:35:41.322567+0000","last_became_active":"2024-07-26T08:33:17.889052+0000","last_became_peered":"2024-07-26T08:33:17.889052+0000","last_unstale":"2024-07-26T08:35:41.322567+0000","last_undegraded":"2024-07-26T08:35:41.322567+0000","last_fullsized":"2024-07-26T08:35:41.322567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:39:41.269747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741015+0000","last_change":"2024-07-26T08:33:03.158618+0000","last_active":"2024-07-26T08:36:12.741015+0000","last_peered":"2024-07-26T08:36:12.741015+0000","last_clean":"2024-07-26T08:36:12.741015+0000","last_became_active":"2024-07-26T08:33:03.158239+0000","last_became_peered":"2024-07-26T08:33:03.158239+0000","last_unstale":"2024-07-26T08:36:12.741015+0000","last_undegraded":"2024-07-26T08:36:12.741015+0000","last_fullsized":"2024-07-26T08:36:12.741015+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:01:26.464113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:20.158778+0000","last_change":"2024-07-26T08:32:58.098913+0000","last_active":"2024-07-26T08:35:20.158778+0000","last_peered":"2024-07-26T08:35:20.158778+0000","last_clean":"2024-07-26T08:35:20.158778+0000","last_became_active":"2024-07-26T08:32:58.098327+0000","last_became_peered":"2024-07-26T08:32:58.098327+0000","last_unstale":"2024-07-26T08:35:20.158778+0000","last_undegraded":"2024-07-26T08:35:20.158778+0000","last_fullsized":"2024-07-26T08:35:20.158778+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:23:52.234409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:45.923456+0000","last_change":"2024-07-26T08:33:17.884320+0000","last_active":"2024-07-26T08:35:45.923456+0000","last_peered":"2024-07-26T08:35:45.923456+0000","last_clean":"2024-07-26T08:35:45.923456+0000","last_became_active":"2024-07-26T08:33:17.883864+0000","last_became_peered":"2024-07-26T08:33:17.883864+0000","last_unstale":"2024-07-26T08:35:45.923456+0000","last_undegraded":"2024-07-26T08:35:45.923456+0000","last_fullsized":"2024-07-26T08:35:45.923456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:24.880834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.717243+0000","last_change":"2024-07-26T08:33:03.135444+0000","last_active":"2024-07-26T08:36:12.717243+0000","last_peered":"2024-07-26T08:36:12.717243+0000","last_clean":"2024-07-26T08:36:12.717243+0000","last_became_active":"2024-07-26T08:33:03.135034+0000","last_became_peered":"2024-07-26T08:33:03.135034+0000","last_unstale":"2024-07-26T08:36:12.717243+0000","last_undegraded":"2024-07-26T08:36:12.717243+0000","last_fullsized":"2024-07-26T08:36:12.717243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:01.367656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"91'44","reported_seq":595,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:37.502224+0000","last_change":"2024-07-26T08:32:58.093186+0000","last_active":"2024-07-26T08:38:37.502224+0000","last_peered":"2024-07-26T08:38:37.502224+0000","last_clean":"2024-07-26T08:38:37.502224+0000","last_became_active":"2024-07-26T08:32:58.092659+0000","last_became_peered":"2024-07-26T08:32:58.092659+0000","last_unstale":"2024-07-26T08:38:37.502224+0000","last_undegraded":"2024-07-26T08:38:37.502224+0000","last_fullsized":"2024-07-26T08:38:37.502224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:52.200077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72963,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":504,"num_read_kb":324,"num_write":46,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.346929+0000","last_change":"2024-07-26T08:33:17.876484+0000","last_active":"2024-07-26T08:35:41.346929+0000","last_peered":"2024-07-26T08:35:41.346929+0000","last_clean":"2024-07-26T08:35:41.346929+0000","last_became_active":"2024-07-26T08:33:17.875963+0000","last_became_peered":"2024-07-26T08:33:17.875963+0000","last_unstale":"2024-07-26T08:35:41.346929+0000","last_undegraded":"2024-07-26T08:35:41.346929+0000","last_fullsized":"2024-07-26T08:35:41.346929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:46:48.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738676+0000","last_change":"2024-07-26T08:33:03.134335+0000","last_active":"2024-07-26T08:36:12.738676+0000","last_peered":"2024-07-26T08:36:12.738676+0000","last_clean":"2024-07-26T08:36:12.738676+0000","last_became_active":"2024-07-26T08:33:03.132922+0000","last_became_peered":"2024-07-26T08:33:03.132922+0000","last_unstale":"2024-07-26T08:36:12.738676+0000","last_undegraded":"2024-07-26T08:36:12.738676+0000","last_fullsized":"2024-07-26T08:36:12.738676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:37:00.982127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.405488+0000","last_change":"2024-07-26T08:32:58.228213+0000","last_active":"2024-07-26T08:35:26.405488+0000","last_peered":"2024-07-26T08:35:26.405488+0000","last_clean":"2024-07-26T08:35:26.405488+0000","last_became_active":"2024-07-26T08:32:58.227826+0000","last_became_peered":"2024-07-26T08:32:58.227826+0000","last_unstale":"2024-07-26T08:35:26.405488+0000","last_undegraded":"2024-07-26T08:35:26.405488+0000","last_fullsized":"2024-07-26T08:35:26.405488+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:17:21.340298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:38.579362+0000","last_change":"2024-07-26T08:34:13.969364+0000","last_active":"2024-07-26T08:35:38.579362+0000","last_peered":"2024-07-26T08:35:38.579362+0000","last_clean":"2024-07-26T08:35:38.579362+0000","last_became_active":"2024-07-26T08:34:13.968819+0000","last_became_peered":"2024-07-26T08:34:13.968819+0000","last_unstale":"2024-07-26T08:35:38.579362+0000","last_undegraded":"2024-07-26T08:35:38.579362+0000","last_fullsized":"2024-07-26T08:35:38.579362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:47:22.667372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740282+0000","last_change":"2024-07-26T08:33:13.826617+0000","last_active":"2024-07-26T08:36:12.740282+0000","last_peered":"2024-07-26T08:36:12.740282+0000","last_clean":"2024-07-26T08:36:12.740282+0000","last_became_active":"2024-07-26T08:33:13.824753+0000","last_became_peered":"2024-07-26T08:33:13.824753+0000","last_unstale":"2024-07-26T08:36:12.740282+0000","last_undegraded":"2024-07-26T08:36:12.740282+0000","last_fullsized":"2024-07-26T08:36:12.740282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:15:35.528861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:22.425306+0000","last_change":"2024-07-26T08:32:58.096953+0000","last_active":"2024-07-26T08:35:22.425306+0000","last_peered":"2024-07-26T08:35:22.425306+0000","last_clean":"2024-07-26T08:35:22.425306+0000","last_became_active":"2024-07-26T08:32:58.096534+0000","last_became_peered":"2024-07-26T08:32:58.096534+0000","last_unstale":"2024-07-26T08:35:22.425306+0000","last_undegraded":"2024-07-26T08:35:22.425306+0000","last_fullsized":"2024-07-26T08:35:22.425306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:55:54.237780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:41.346745+0000","last_change":"2024-07-26T08:33:17.891663+0000","last_active":"2024-07-26T08:35:41.346745+0000","last_peered":"2024-07-26T08:35:41.346745+0000","last_clean":"2024-07-26T08:35:41.346745+0000","last_became_active":"2024-07-26T08:33:17.890421+0000","last_became_peered":"2024-07-26T08:33:17.890421+0000","last_unstale":"2024-07-26T08:35:41.346745+0000","last_undegraded":"2024-07-26T08:35:41.346745+0000","last_fullsized":"2024-07-26T08:35:41.346745+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:31:44.907007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741135+0000","last_change":"2024-07-26T08:33:03.143008+0000","last_active":"2024-07-26T08:36:12.741135+0000","last_peered":"2024-07-26T08:36:12.741135+0000","last_clean":"2024-07-26T08:36:12.741135+0000","last_became_active":"2024-07-26T08:33:03.142660+0000","last_became_peered":"2024-07-26T08:33:03.142660+0000","last_unstale":"2024-07-26T08:36:12.741135+0000","last_undegraded":"2024-07-26T08:36:12.741135+0000","last_fullsized":"2024-07-26T08:36:12.741135+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:12:40.474858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":544,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:37.501908+0000","last_change":"2024-07-26T08:32:58.091177+0000","last_active":"2024-07-26T08:38:37.501908+0000","last_peered":"2024-07-26T08:38:37.501908+0000","last_clean":"2024-07-26T08:38:37.501908+0000","last_became_active":"2024-07-26T08:32:58.090718+0000","last_became_peered":"2024-07-26T08:32:58.090718+0000","last_unstale":"2024-07-26T08:38:37.501908+0000","last_undegraded":"2024-07-26T08:38:37.501908+0000","last_fullsized":"2024-07-26T08:38:37.501908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:09:10.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":492,"num_read_kb":492,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:37.278601+0000","last_change":"2024-07-26T08:33:17.874548+0000","last_active":"2024-07-26T08:35:37.278601+0000","last_peered":"2024-07-26T08:35:37.278601+0000","last_clean":"2024-07-26T08:35:37.278601+0000","last_became_active":"2024-07-26T08:33:17.874066+0000","last_became_peered":"2024-07-26T08:33:17.874066+0000","last_unstale":"2024-07-26T08:35:37.278601+0000","last_undegraded":"2024-07-26T08:35:37.278601+0000","last_fullsized":"2024-07-26T08:35:37.278601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:03:55.321589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:17.861977+0000","last_change":"2024-07-26T08:33:17.899994+0000","last_active":"2024-07-26T08:35:17.861977+0000","last_peered":"2024-07-26T08:35:17.861977+0000","last_clean":"2024-07-26T08:35:17.861977+0000","last_became_active":"2024-07-26T08:33:17.899481+0000","last_became_peered":"2024-07-26T08:33:17.899481+0000","last_unstale":"2024-07-26T08:35:17.861977+0000","last_undegraded":"2024-07-26T08:35:17.861977+0000","last_fullsized":"2024-07-26T08:35:17.861977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:29:30.152541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746413+0000","last_change":"2024-07-26T08:33:03.153063+0000","last_active":"2024-07-26T08:36:12.746413+0000","last_peered":"2024-07-26T08:36:12.746413+0000","last_clean":"2024-07-26T08:36:12.746413+0000","last_became_active":"2024-07-26T08:33:03.152247+0000","last_became_peered":"2024-07-26T08:33:03.152247+0000","last_unstale":"2024-07-26T08:36:12.746413+0000","last_undegraded":"2024-07-26T08:36:12.746413+0000","last_fullsized":"2024-07-26T08:36:12.746413+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:21:00.584893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.405429+0000","last_change":"2024-07-26T08:32:58.095996+0000","last_active":"2024-07-26T08:35:26.405429+0000","last_peered":"2024-07-26T08:35:26.405429+0000","last_clean":"2024-07-26T08:35:26.405429+0000","last_became_active":"2024-07-26T08:32:58.095602+0000","last_became_peered":"2024-07-26T08:32:58.095602+0000","last_unstale":"2024-07-26T08:35:26.405429+0000","last_undegraded":"2024-07-26T08:35:26.405429+0000","last_fullsized":"2024-07-26T08:35:26.405429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:11.604555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.320795+0000","last_change":"2024-07-26T08:33:17.890457+0000","last_active":"2024-07-26T08:35:26.320795+0000","last_peered":"2024-07-26T08:35:26.320795+0000","last_clean":"2024-07-26T08:35:26.320795+0000","last_became_active":"2024-07-26T08:33:17.890027+0000","last_became_peered":"2024-07-26T08:33:17.890027+0000","last_unstale":"2024-07-26T08:35:26.320795+0000","last_undegraded":"2024-07-26T08:35:26.320795+0000","last_fullsized":"2024-07-26T08:35:26.320795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:55:27.859574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.741858+0000","last_change":"2024-07-26T08:33:03.153233+0000","last_active":"2024-07-26T08:36:12.741858+0000","last_peered":"2024-07-26T08:36:12.741858+0000","last_clean":"2024-07-26T08:36:12.741858+0000","last_became_active":"2024-07-26T08:33:03.152479+0000","last_became_peered":"2024-07-26T08:33:03.152479+0000","last_unstale":"2024-07-26T08:36:12.741858+0000","last_undegraded":"2024-07-26T08:36:12.741858+0000","last_fullsized":"2024-07-26T08:36:12.741858+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:03.667108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:26.566623+0000","last_change":"2024-07-26T08:32:58.095714+0000","last_active":"2024-07-26T08:35:26.566623+0000","last_peered":"2024-07-26T08:35:26.566623+0000","last_clean":"2024-07-26T08:35:26.566623+0000","last_became_active":"2024-07-26T08:32:58.095178+0000","last_became_peered":"2024-07-26T08:32:58.095178+0000","last_unstale":"2024-07-26T08:35:26.566623+0000","last_undegraded":"2024-07-26T08:35:26.566623+0000","last_fullsized":"2024-07-26T08:35:26.566623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:12.365342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:27.846852+0000","last_change":"2024-07-26T08:32:58.092887+0000","last_active":"2024-07-26T08:35:27.846852+0000","last_peered":"2024-07-26T08:35:27.846852+0000","last_clean":"2024-07-26T08:35:27.846852+0000","last_became_active":"2024-07-26T08:32:58.092466+0000","last_became_peered":"2024-07-26T08:32:58.092466+0000","last_unstale":"2024-07-26T08:35:27.846852+0000","last_undegraded":"2024-07-26T08:35:27.846852+0000","last_fullsized":"2024-07-26T08:35:27.846852+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:00:52.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740942+0000","last_change":"2024-07-26T08:33:13.834671+0000","last_active":"2024-07-26T08:36:12.740942+0000","last_peered":"2024-07-26T08:36:12.740942+0000","last_clean":"2024-07-26T08:36:12.740942+0000","last_became_active":"2024-07-26T08:33:13.833987+0000","last_became_peered":"2024-07-26T08:33:13.833987+0000","last_unstale":"2024-07-26T08:36:12.740942+0000","last_undegraded":"2024-07-26T08:36:12.740942+0000","last_fullsized":"2024-07-26T08:36:12.740942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:27:51.809203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:50.162384+0000","last_change":"2024-07-26T08:33:17.886754+0000","last_active":"2024-07-26T08:35:50.162384+0000","last_peered":"2024-07-26T08:35:50.162384+0000","last_clean":"2024-07-26T08:35:50.162384+0000","last_became_active":"2024-07-26T08:33:17.886330+0000","last_became_peered":"2024-07-26T08:33:17.886330+0000","last_unstale":"2024-07-26T08:35:50.162384+0000","last_undegraded":"2024-07-26T08:35:50.162384+0000","last_fullsized":"2024-07-26T08:35:50.162384+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:52:56.190551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:38:34.497991+0000","last_change":"2024-07-26T08:32:58.093940+0000","last_active":"2024-07-26T08:38:34.497991+0000","last_peered":"2024-07-26T08:38:34.497991+0000","last_clean":"2024-07-26T08:38:34.497991+0000","last_became_active":"2024-07-26T08:32:58.093538+0000","last_became_peered":"2024-07-26T08:32:58.093538+0000","last_unstale":"2024-07-26T08:38:34.497991+0000","last_undegraded":"2024-07-26T08:38:34.497991+0000","last_fullsized":"2024-07-26T08:38:34.497991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:32:57.041456+0000","last_clean_scrub_stamp":"2024-07-26T08:32:57.041456+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:44:47.106923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":246,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.727350+0000","last_change":"2024-07-26T08:33:03.128555+0000","last_active":"2024-07-26T08:36:12.727350+0000","last_peered":"2024-07-26T08:36:12.727350+0000","last_clean":"2024-07-26T08:36:12.727350+0000","last_became_active":"2024-07-26T08:33:03.128132+0000","last_became_peered":"2024-07-26T08:33:03.128132+0000","last_unstale":"2024-07-26T08:36:12.727350+0000","last_undegraded":"2024-07-26T08:36:12.727350+0000","last_fullsized":"2024-07-26T08:36:12.727350+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:14:43.638339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:35:50.162409+0000","last_change":"2024-07-26T08:33:17.871061+0000","last_active":"2024-07-26T08:35:50.162409+0000","last_peered":"2024-07-26T08:35:50.162409+0000","last_clean":"2024-07-26T08:35:50.162409+0000","last_became_active":"2024-07-26T08:33:17.870596+0000","last_became_peered":"2024-07-26T08:33:17.870596+0000","last_unstale":"2024-07-26T08:35:50.162409+0000","last_undegraded":"2024-07-26T08:35:50.162409+0000","last_fullsized":"2024-07-26T08:35:50.162409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:16.837402+0000","last_clean_scrub_stamp":"2024-07-26T08:33:16.837402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:29:42.262889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723745+0000","last_change":"2024-07-26T08:33:03.135269+0000","last_active":"2024-07-26T08:36:12.723745+0000","last_peered":"2024-07-26T08:36:12.723745+0000","last_clean":"2024-07-26T08:36:12.723745+0000","last_became_active":"2024-07-26T08:33:03.134879+0000","last_became_peered":"2024-07-26T08:33:03.134879+0000","last_unstale":"2024-07-26T08:36:12.723745+0000","last_undegraded":"2024-07-26T08:36:12.723745+0000","last_fullsized":"2024-07-26T08:36:12.723745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:58:57.065224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.722200+0000","last_change":"2024-07-26T08:33:03.131241+0000","last_active":"2024-07-26T08:36:12.722200+0000","last_peered":"2024-07-26T08:36:12.722200+0000","last_clean":"2024-07-26T08:36:12.722200+0000","last_became_active":"2024-07-26T08:33:03.130847+0000","last_became_peered":"2024-07-26T08:33:03.130847+0000","last_unstale":"2024-07-26T08:36:12.722200+0000","last_undegraded":"2024-07-26T08:36:12.722200+0000","last_fullsized":"2024-07-26T08:36:12.722200+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:13:39.154681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.737483+0000","last_change":"2024-07-26T08:33:03.159497+0000","last_active":"2024-07-26T08:36:12.737483+0000","last_peered":"2024-07-26T08:36:12.737483+0000","last_clean":"2024-07-26T08:36:12.737483+0000","last_became_active":"2024-07-26T08:33:03.158958+0000","last_became_peered":"2024-07-26T08:33:03.158958+0000","last_unstale":"2024-07-26T08:36:12.737483+0000","last_undegraded":"2024-07-26T08:36:12.737483+0000","last_fullsized":"2024-07-26T08:36:12.737483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:44.991185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729462+0000","last_change":"2024-07-26T08:33:13.828001+0000","last_active":"2024-07-26T08:36:12.729462+0000","last_peered":"2024-07-26T08:36:12.729462+0000","last_clean":"2024-07-26T08:36:12.729462+0000","last_became_active":"2024-07-26T08:33:13.824323+0000","last_became_peered":"2024-07-26T08:33:13.824323+0000","last_unstale":"2024-07-26T08:36:12.729462+0000","last_undegraded":"2024-07-26T08:36:12.729462+0000","last_fullsized":"2024-07-26T08:36:12.729462+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:46:17.224617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742274+0000","last_change":"2024-07-26T08:33:03.160430+0000","last_active":"2024-07-26T08:36:12.742274+0000","last_peered":"2024-07-26T08:36:12.742274+0000","last_clean":"2024-07-26T08:36:12.742274+0000","last_became_active":"2024-07-26T08:33:03.160065+0000","last_became_peered":"2024-07-26T08:33:03.160065+0000","last_unstale":"2024-07-26T08:36:12.742274+0000","last_undegraded":"2024-07-26T08:36:12.742274+0000","last_fullsized":"2024-07-26T08:36:12.742274+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:39:59.122274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.718573+0000","last_change":"2024-07-26T08:33:03.140442+0000","last_active":"2024-07-26T08:36:12.718573+0000","last_peered":"2024-07-26T08:36:12.718573+0000","last_clean":"2024-07-26T08:36:12.718573+0000","last_became_active":"2024-07-26T08:33:03.138109+0000","last_became_peered":"2024-07-26T08:33:03.138109+0000","last_unstale":"2024-07-26T08:36:12.718573+0000","last_undegraded":"2024-07-26T08:36:12.718573+0000","last_fullsized":"2024-07-26T08:36:12.718573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:13:13.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.732735+0000","last_change":"2024-07-26T08:33:03.143095+0000","last_active":"2024-07-26T08:36:12.732735+0000","last_peered":"2024-07-26T08:36:12.732735+0000","last_clean":"2024-07-26T08:36:12.732735+0000","last_became_active":"2024-07-26T08:33:03.142646+0000","last_became_peered":"2024-07-26T08:33:03.142646+0000","last_unstale":"2024-07-26T08:36:12.732735+0000","last_undegraded":"2024-07-26T08:36:12.732735+0000","last_fullsized":"2024-07-26T08:36:12.732735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:14:15.945753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.745906+0000","last_change":"2024-07-26T08:33:03.146010+0000","last_active":"2024-07-26T08:36:12.745906+0000","last_peered":"2024-07-26T08:36:12.745906+0000","last_clean":"2024-07-26T08:36:12.745906+0000","last_became_active":"2024-07-26T08:33:03.145634+0000","last_became_peered":"2024-07-26T08:33:03.145634+0000","last_unstale":"2024-07-26T08:36:12.745906+0000","last_undegraded":"2024-07-26T08:36:12.745906+0000","last_fullsized":"2024-07-26T08:36:12.745906+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:57:12.412563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746340+0000","last_change":"2024-07-26T08:33:03.156455+0000","last_active":"2024-07-26T08:36:12.746340+0000","last_peered":"2024-07-26T08:36:12.746340+0000","last_clean":"2024-07-26T08:36:12.746340+0000","last_became_active":"2024-07-26T08:33:03.156000+0000","last_became_peered":"2024-07-26T08:33:03.156000+0000","last_unstale":"2024-07-26T08:36:12.746340+0000","last_undegraded":"2024-07-26T08:36:12.746340+0000","last_fullsized":"2024-07-26T08:36:12.746340+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:44.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742178+0000","last_change":"2024-07-26T08:33:03.153424+0000","last_active":"2024-07-26T08:36:12.742178+0000","last_peered":"2024-07-26T08:36:12.742178+0000","last_clean":"2024-07-26T08:36:12.742178+0000","last_became_active":"2024-07-26T08:33:03.150777+0000","last_became_peered":"2024-07-26T08:33:03.150777+0000","last_unstale":"2024-07-26T08:36:12.742178+0000","last_undegraded":"2024-07-26T08:36:12.742178+0000","last_fullsized":"2024-07-26T08:36:12.742178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:31.610328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746534+0000","last_change":"2024-07-26T08:33:03.154539+0000","last_active":"2024-07-26T08:36:12.746534+0000","last_peered":"2024-07-26T08:36:12.746534+0000","last_clean":"2024-07-26T08:36:12.746534+0000","last_became_active":"2024-07-26T08:33:03.153478+0000","last_became_peered":"2024-07-26T08:33:03.153478+0000","last_unstale":"2024-07-26T08:36:12.746534+0000","last_undegraded":"2024-07-26T08:36:12.746534+0000","last_fullsized":"2024-07-26T08:36:12.746534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:04:55.250480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742408+0000","last_change":"2024-07-26T08:33:03.166523+0000","last_active":"2024-07-26T08:36:12.742408+0000","last_peered":"2024-07-26T08:36:12.742408+0000","last_clean":"2024-07-26T08:36:12.742408+0000","last_became_active":"2024-07-26T08:33:03.166055+0000","last_became_peered":"2024-07-26T08:33:03.166055+0000","last_unstale":"2024-07-26T08:36:12.742408+0000","last_undegraded":"2024-07-26T08:36:12.742408+0000","last_fullsized":"2024-07-26T08:36:12.742408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:00:00.748019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.723201+0000","last_change":"2024-07-26T08:33:03.137123+0000","last_active":"2024-07-26T08:36:12.723201+0000","last_peered":"2024-07-26T08:36:12.723201+0000","last_clean":"2024-07-26T08:36:12.723201+0000","last_became_active":"2024-07-26T08:33:03.136644+0000","last_became_peered":"2024-07-26T08:33:03.136644+0000","last_unstale":"2024-07-26T08:36:12.723201+0000","last_undegraded":"2024-07-26T08:36:12.723201+0000","last_fullsized":"2024-07-26T08:36:12.723201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:14.689892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736113+0000","last_change":"2024-07-26T08:33:03.150119+0000","last_active":"2024-07-26T08:36:12.736113+0000","last_peered":"2024-07-26T08:36:12.736113+0000","last_clean":"2024-07-26T08:36:12.736113+0000","last_became_active":"2024-07-26T08:33:03.149717+0000","last_became_peered":"2024-07-26T08:33:03.149717+0000","last_unstale":"2024-07-26T08:36:12.736113+0000","last_undegraded":"2024-07-26T08:36:12.736113+0000","last_fullsized":"2024-07-26T08:36:12.736113+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:52:05.641059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.733731+0000","last_change":"2024-07-26T08:33:03.135685+0000","last_active":"2024-07-26T08:36:12.733731+0000","last_peered":"2024-07-26T08:36:12.733731+0000","last_clean":"2024-07-26T08:36:12.733731+0000","last_became_active":"2024-07-26T08:33:03.135216+0000","last_became_peered":"2024-07-26T08:33:03.135216+0000","last_unstale":"2024-07-26T08:36:12.733731+0000","last_undegraded":"2024-07-26T08:36:12.733731+0000","last_fullsized":"2024-07-26T08:36:12.733731+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:33:17.857327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740824+0000","last_change":"2024-07-26T08:33:03.159320+0000","last_active":"2024-07-26T08:36:12.740824+0000","last_peered":"2024-07-26T08:36:12.740824+0000","last_clean":"2024-07-26T08:36:12.740824+0000","last_became_active":"2024-07-26T08:33:03.158914+0000","last_became_peered":"2024-07-26T08:33:03.158914+0000","last_unstale":"2024-07-26T08:36:12.740824+0000","last_undegraded":"2024-07-26T08:36:12.740824+0000","last_fullsized":"2024-07-26T08:36:12.740824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:22:47.803920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743134+0000","last_change":"2024-07-26T08:33:03.155383+0000","last_active":"2024-07-26T08:36:12.743134+0000","last_peered":"2024-07-26T08:36:12.743134+0000","last_clean":"2024-07-26T08:36:12.743134+0000","last_became_active":"2024-07-26T08:33:03.154958+0000","last_became_peered":"2024-07-26T08:33:03.154958+0000","last_unstale":"2024-07-26T08:36:12.743134+0000","last_undegraded":"2024-07-26T08:36:12.743134+0000","last_fullsized":"2024-07-26T08:36:12.743134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:55.042899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739206+0000","last_change":"2024-07-26T08:33:03.164427+0000","last_active":"2024-07-26T08:36:12.739206+0000","last_peered":"2024-07-26T08:36:12.739206+0000","last_clean":"2024-07-26T08:36:12.739206+0000","last_became_active":"2024-07-26T08:33:03.161114+0000","last_became_peered":"2024-07-26T08:33:03.161114+0000","last_unstale":"2024-07-26T08:36:12.739206+0000","last_undegraded":"2024-07-26T08:36:12.739206+0000","last_fullsized":"2024-07-26T08:36:12.739206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:24.587146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.738943+0000","last_change":"2024-07-26T08:33:03.155785+0000","last_active":"2024-07-26T08:36:12.738943+0000","last_peered":"2024-07-26T08:36:12.738943+0000","last_clean":"2024-07-26T08:36:12.738943+0000","last_became_active":"2024-07-26T08:33:03.155427+0000","last_became_peered":"2024-07-26T08:33:03.155427+0000","last_unstale":"2024-07-26T08:36:12.738943+0000","last_undegraded":"2024-07-26T08:36:12.738943+0000","last_fullsized":"2024-07-26T08:36:12.738943+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:22:17.126349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.729478+0000","last_change":"2024-07-26T08:33:03.141721+0000","last_active":"2024-07-26T08:36:12.729478+0000","last_peered":"2024-07-26T08:36:12.729478+0000","last_clean":"2024-07-26T08:36:12.729478+0000","last_became_active":"2024-07-26T08:33:03.141358+0000","last_became_peered":"2024-07-26T08:33:03.141358+0000","last_unstale":"2024-07-26T08:36:12.729478+0000","last_undegraded":"2024-07-26T08:36:12.729478+0000","last_fullsized":"2024-07-26T08:36:12.729478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:23.275415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719269+0000","last_change":"2024-07-26T08:33:03.137414+0000","last_active":"2024-07-26T08:36:12.719269+0000","last_peered":"2024-07-26T08:36:12.719269+0000","last_clean":"2024-07-26T08:36:12.719269+0000","last_became_active":"2024-07-26T08:33:03.136615+0000","last_became_peered":"2024-07-26T08:33:03.136615+0000","last_unstale":"2024-07-26T08:36:12.719269+0000","last_undegraded":"2024-07-26T08:36:12.719269+0000","last_fullsized":"2024-07-26T08:36:12.719269+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:30.957804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.742197+0000","last_change":"2024-07-26T08:33:03.142634+0000","last_active":"2024-07-26T08:36:12.742197+0000","last_peered":"2024-07-26T08:36:12.742197+0000","last_clean":"2024-07-26T08:36:12.742197+0000","last_became_active":"2024-07-26T08:33:03.142263+0000","last_became_peered":"2024-07-26T08:33:03.142263+0000","last_unstale":"2024-07-26T08:36:12.742197+0000","last_undegraded":"2024-07-26T08:36:12.742197+0000","last_fullsized":"2024-07-26T08:36:12.742197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:17:01.139298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.721942+0000","last_change":"2024-07-26T08:33:03.139559+0000","last_active":"2024-07-26T08:36:12.721942+0000","last_peered":"2024-07-26T08:36:12.721942+0000","last_clean":"2024-07-26T08:36:12.721942+0000","last_became_active":"2024-07-26T08:33:03.138464+0000","last_became_peered":"2024-07-26T08:33:03.138464+0000","last_unstale":"2024-07-26T08:36:12.721942+0000","last_undegraded":"2024-07-26T08:36:12.721942+0000","last_fullsized":"2024-07-26T08:36:12.721942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:57.201026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740367+0000","last_change":"2024-07-26T08:33:03.136371+0000","last_active":"2024-07-26T08:36:12.740367+0000","last_peered":"2024-07-26T08:36:12.740367+0000","last_clean":"2024-07-26T08:36:12.740367+0000","last_became_active":"2024-07-26T08:33:03.135993+0000","last_became_peered":"2024-07-26T08:33:03.135993+0000","last_unstale":"2024-07-26T08:36:12.740367+0000","last_undegraded":"2024-07-26T08:36:12.740367+0000","last_fullsized":"2024-07-26T08:36:12.740367+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:04:59.307777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.746129+0000","last_change":"2024-07-26T08:33:03.152042+0000","last_active":"2024-07-26T08:36:12.746129+0000","last_peered":"2024-07-26T08:36:12.746129+0000","last_clean":"2024-07-26T08:36:12.746129+0000","last_became_active":"2024-07-26T08:33:03.151598+0000","last_became_peered":"2024-07-26T08:33:03.151598+0000","last_unstale":"2024-07-26T08:36:12.746129+0000","last_undegraded":"2024-07-26T08:36:12.746129+0000","last_fullsized":"2024-07-26T08:36:12.746129+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:48:45.472212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739206+0000","last_change":"2024-07-26T08:33:03.141884+0000","last_active":"2024-07-26T08:36:12.739206+0000","last_peered":"2024-07-26T08:36:12.739206+0000","last_clean":"2024-07-26T08:36:12.739206+0000","last_became_active":"2024-07-26T08:33:03.141504+0000","last_became_peered":"2024-07-26T08:33:03.141504+0000","last_unstale":"2024-07-26T08:36:12.739206+0000","last_undegraded":"2024-07-26T08:36:12.739206+0000","last_fullsized":"2024-07-26T08:36:12.739206+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:40:33.581709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.728644+0000","last_change":"2024-07-26T08:33:13.816949+0000","last_active":"2024-07-26T08:36:12.728644+0000","last_peered":"2024-07-26T08:36:12.728644+0000","last_clean":"2024-07-26T08:36:12.728644+0000","last_became_active":"2024-07-26T08:33:13.816478+0000","last_became_peered":"2024-07-26T08:33:13.816478+0000","last_unstale":"2024-07-26T08:36:12.728644+0000","last_undegraded":"2024-07-26T08:36:12.728644+0000","last_fullsized":"2024-07-26T08:36:12.728644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:39:52.972007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.743895+0000","last_change":"2024-07-26T08:33:03.154319+0000","last_active":"2024-07-26T08:36:12.743895+0000","last_peered":"2024-07-26T08:36:12.743895+0000","last_clean":"2024-07-26T08:36:12.743895+0000","last_became_active":"2024-07-26T08:33:03.153922+0000","last_became_peered":"2024-07-26T08:33:03.153922+0000","last_unstale":"2024-07-26T08:36:12.743895+0000","last_undegraded":"2024-07-26T08:36:12.743895+0000","last_fullsized":"2024-07-26T08:36:12.743895+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:25:42.263383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.719735+0000","last_change":"2024-07-26T08:33:03.133876+0000","last_active":"2024-07-26T08:36:12.719735+0000","last_peered":"2024-07-26T08:36:12.719735+0000","last_clean":"2024-07-26T08:36:12.719735+0000","last_became_active":"2024-07-26T08:33:03.133408+0000","last_became_peered":"2024-07-26T08:33:03.133408+0000","last_unstale":"2024-07-26T08:36:12.719735+0000","last_undegraded":"2024-07-26T08:36:12.719735+0000","last_fullsized":"2024-07-26T08:36:12.719735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:21.649832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.736112+0000","last_change":"2024-07-26T08:33:03.162410+0000","last_active":"2024-07-26T08:36:12.736112+0000","last_peered":"2024-07-26T08:36:12.736112+0000","last_clean":"2024-07-26T08:36:12.736112+0000","last_became_active":"2024-07-26T08:33:03.161187+0000","last_became_peered":"2024-07-26T08:33:03.161187+0000","last_unstale":"2024-07-26T08:36:12.736112+0000","last_undegraded":"2024-07-26T08:36:12.736112+0000","last_fullsized":"2024-07-26T08:36:12.736112+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:49:42.870159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.739102+0000","last_change":"2024-07-26T08:33:03.160326+0000","last_active":"2024-07-26T08:36:12.739102+0000","last_peered":"2024-07-26T08:36:12.739102+0000","last_clean":"2024-07-26T08:36:12.739102+0000","last_became_active":"2024-07-26T08:33:03.159831+0000","last_became_peered":"2024-07-26T08:33:03.159831+0000","last_unstale":"2024-07-26T08:36:12.739102+0000","last_undegraded":"2024-07-26T08:36:12.739102+0000","last_fullsized":"2024-07-26T08:36:12.739102+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:39:39.797090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-07-26T08:36:12.740316+0000","last_change":"2024-07-26T08:33:03.153773+0000","last_active":"2024-07-26T08:36:12.740316+0000","last_peered":"2024-07-26T08:36:12.740316+0000","last_clean":"2024-07-26T08:36:12.740316+0000","last_became_active":"2024-07-26T08:33:03.152875+0000","last_became_peered":"2024-07-26T08:33:03.152875+0000","last_unstale":"2024-07-26T08:36:12.740316+0000","last_undegraded":"2024-07-26T08:36:12.740316+0000","last_fullsized":"2024-07-26T08:36:12.740316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T08:33:02.089692+0000","last_clean_scrub_stamp":"2024-07-26T08:33:02.089692+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:12.167399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80756,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2502,"num_read_kb":1339,"num_write":269,"num_write_kb":167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":242268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282352,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645397,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540314,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841625,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1175552,"data_stored":1204384,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005150,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":539600,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":55,"seq":236223201382,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":539866,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397611,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1136923,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561134,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":535104,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757365,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1131666,"data_compressed":68781,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920890,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":540017,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117119,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":534812,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378760,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":609849,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542283,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":609097,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705807,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":537092,"data_compressed":32878,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34279,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35903,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":75467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T08:38:41.390 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:38:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:41 smithi044 ceph-mon[32404]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:41 smithi171 ceph-mon[37789]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:41.886 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:41 smithi116 ceph-mon[37097]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:42 smithi044 ceph-mon[32404]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:42 smithi044 ceph-mon[32404]: pgmap v483: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:38:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:42 smithi044 ceph-mon[32404]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:42 smithi171 ceph-mon[37789]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:42 smithi171 ceph-mon[37789]: pgmap v483: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:38:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:42 smithi171 ceph-mon[37789]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:42 smithi116 ceph-mon[37097]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:42 smithi116 ceph-mon[37097]: pgmap v483: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:38:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:42 smithi116 ceph-mon[37097]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T08:38:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 -- 172.21.15.44:0/3110549928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 msgr2=0x7f5ba010b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 --2- 172.21.15.44:0/3110549928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba010b530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009900 tx=0x7f5b8c02a400 comp rx=0 tx=0).stop 2024-07-26T08:38:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 -- 172.21.15.44:0/3110549928 shutdown_connections 2024-07-26T08:38:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 --2- 172.21.15.44:0/3110549928 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba010ba70 0x7f5ba01122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 --2- 172.21.15.44:0/3110549928 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba010b530 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 --2- 172.21.15.44:0/3110549928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba01057e0 0x7f5ba0105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 -- 172.21.15.44:0/3110549928 >> 172.21.15.44:0/3110549928 conn(0x7f5ba01011f0 msgr2=0x7f5ba0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:44.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 -- 172.21.15.44:0/3110549928 shutdown_connections 2024-07-26T08:38:44.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.274+0000 7f5ba7200640 1 -- 172.21.15.44:0/3110549928 wait complete. 2024-07-26T08:38:44.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.275+0000 7f5ba7200640 1 Processor -- start 2024-07-26T08:38:44.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.275+0000 7f5ba7200640 1 -- start start 2024-07-26T08:38:44.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba7200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba01057e0 0x7f5ba019c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:44.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba7200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:44.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba010ba70 0x7f5ba01a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba7200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba0115700 con 0x7f5ba010b120 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba0115580 con 0x7f5ba010ba70 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba7200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba0115880 con 0x7f5ba01057e0 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5b9f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5ba5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba010ba70 0x7f5ba01a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5b9f400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41424/0 (socket says 172.21.15.44:41424) 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5b9f400640 1 -- 172.21.15.44:0/2108127391 learned_addr learned my addr 172.21.15.44:0/2108127391 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.276+0000 7f5b9fe00640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba01057e0 0x7f5ba019c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba5200640 1 -- 172.21.15.44:0/2108127391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba01057e0 msgr2=0x7f5ba019c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba5200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba01057e0 0x7f5ba019c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba5200640 1 -- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 msgr2=0x7f5ba019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba5200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba5200640 1 -- 172.21.15.44:0/2108127391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba01a2160 con 0x7f5ba010ba70 2024-07-26T08:38:44.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba5200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba010ba70 0x7f5ba01a1a30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009fd0 tx=0x7f5b9800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:44.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9800ce20 con 0x7f5ba010ba70 2024-07-26T08:38:44.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b98005020 con 0x7f5ba010ba70 2024-07-26T08:38:44.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba01a2450 con 0x7f5ba010ba70 2024-07-26T08:38:44.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98010690 con 0x7f5ba010ba70 2024-07-26T08:38:44.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.277+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba01a8bb0 con 0x7f5ba010ba70 2024-07-26T08:38:44.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.279+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba0105bf0 con 0x7f5ba010ba70 2024-07-26T08:38:44.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.281+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f5b980054d0 con 0x7f5ba010ba70 2024-07-26T08:38:44.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.283+0000 7f5b9cc00640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5b840b3c20 0x7f5b840b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:44.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.283+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f5b980e4720 con 0x7f5ba010ba70 2024-07-26T08:38:44.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.283+0000 7f5b9fe00640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5b840b3c20 0x7f5b840b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:44.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.284+0000 7f5b9fe00640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5b840b3c20 0x7f5b840b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009bd0 tx=0x7f5b90005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:44.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.284+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5b980ab860 con 0x7f5ba010ba70 2024-07-26T08:38:44.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.525+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ba019d320 con 0x7f5ba010ba70 2024-07-26T08:38:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.526+0000 7f5b9cc00640 1 -- 172.21.15.44:0/2108127391 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7f5b980ab160 con 0x7f5ba010ba70 2024-07-26T08:38:44.528 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:44.528 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":30,"btime":"2024-07-26T08:38:32:520674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:38:32.022353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24701,24686,24679,24668,24661]},"id":1}]} 2024-07-26T08:38:44.528 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 30 2024-07-26T08:38:44.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5b840b3c20 msgr2=0x7f5b840b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5b840b3c20 0x7f5b840b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009bd0 tx=0x7f5b90005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba010ba70 msgr2=0x7f5ba01a1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba010ba70 0x7f5ba01a1a30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009fd0 tx=0x7f5b9800ef10 comp rx=0 tx=0).stop 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 shutdown_connections 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f5b840b3c20 0x7f5b840b6130 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ba010ba70 0x7f5ba01a1a30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5ba010b120 0x7f5ba019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 --2- 172.21.15.44:0/2108127391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba01057e0 0x7f5ba019c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 >> 172.21.15.44:0/2108127391 conn(0x7f5ba01011f0 msgr2=0x7f5ba01098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 shutdown_connections 2024-07-26T08:38:44.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:44.529+0000 7f5ba7200640 1 -- 172.21.15.44:0/2108127391 wait complete. 2024-07-26T08:38:44.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:44 smithi044 ceph-mon[32404]: pgmap v484: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T08:38:44.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:44 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2108127391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:38:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:44 smithi116 ceph-mon[37097]: pgmap v484: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T08:38:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:44 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2108127391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:38:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:44 smithi171 ceph-mon[37789]: pgmap v484: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T08:38:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:44 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2108127391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:38:45.390 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T08:38:45.885 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:46 smithi044 ceph-mon[32404]: pgmap v485: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T08:38:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:46 smithi116 ceph-mon[37097]: pgmap v485: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T08:38:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:46 smithi171 ceph-mon[37789]: pgmap v485: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T08:38:47.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:47 smithi044 ceph-mon[32404]: pgmap v486: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T08:38:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:47 smithi116 ceph-mon[37097]: pgmap v486: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T08:38:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:47 smithi171 ceph-mon[37789]: pgmap v486: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 -- 172.21.15.44:0/3935290119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 msgr2=0x7fd4f00751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3935290119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f00751e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009900 tx=0x7fd4e402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 -- 172.21.15.44:0/3935290119 shutdown_connections 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3935290119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4f0075800 0x7fd4f0075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3935290119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4f0113c90 0x7fd4f0116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3935290119 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f00751e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 -- 172.21.15.44:0/3935290119 >> 172.21.15.44:0/3935290119 conn(0x7fd4f00fee10 msgr2=0x7fd4f0101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 -- 172.21.15.44:0/3935290119 shutdown_connections 2024-07-26T08:38:48.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.274+0000 7fd4f7200640 1 -- 172.21.15.44:0/3935290119 wait complete. 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.275+0000 7fd4f7200640 1 Processor -- start 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 -- start start 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4f0075800 0x7fd4f01a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f01a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4f0113c90 0x7fd4f01a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f01198d0 con 0x7fd4f0076db0 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f0119750 con 0x7fd4f0113c90 2024-07-26T08:38:48.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f7200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f0119a50 con 0x7fd4f0075800 2024-07-26T08:38:48.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f01a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:48.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.276+0000 7fd4f5400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f01a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41442/0 (socket says 172.21.15.44:41442) 2024-07-26T08:38:48.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 -- 172.21.15.44:0/3740642569 learned_addr learned my addr 172.21.15.44:0/3740642569 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:48.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5e00640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4f0075800 0x7fd4f01a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:48.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f6800640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4f0113c90 0x7fd4f01a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 -- 172.21.15.44:0/3740642569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4f0075800 msgr2=0x7fd4f01a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4f0075800 0x7fd4f01a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 -- 172.21.15.44:0/3740642569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4f0113c90 msgr2=0x7fd4f01a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4f0113c90 0x7fd4f01a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 -- 172.21.15.44:0/3740642569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f01a62b0 con 0x7fd4f0076db0 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.277+0000 7fd4f5400640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f01a08e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00e9c0 tx=0x7fd4dc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.278+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00ccc0 con 0x7fd4f0076db0 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.278+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4dc0054d0 con 0x7fd4f0076db0 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.278+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc010430 con 0x7fd4f0076db0 2024-07-26T08:38:48.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.278+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f01a65a0 con 0x7fd4f0076db0 2024-07-26T08:38:48.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.278+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f01acd00 con 0x7fd4f0076db0 2024-07-26T08:38:48.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.279+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f0075c10 con 0x7fd4f0076db0 2024-07-26T08:38:48.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.282+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7fd4dc004f20 con 0x7fd4f0076db0 2024-07-26T08:38:48.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.284+0000 7fd4eaa00640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd4d80b3c20 0x7fd4d80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:48.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.284+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fd4dc0e4700 con 0x7fd4f0076db0 2024-07-26T08:38:48.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.284+0000 7fd4f5e00640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd4d80b3c20 0x7fd4d80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:48.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.285+0000 7fd4f5e00640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd4d80b3c20 0x7fd4d80b6130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4031440 tx=0x7fd4e4009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:48.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.288+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd4dc0ab840 con 0x7fd4f0076db0 2024-07-26T08:38:48.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.491+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4f0112a20 con 0x7fd4f0076db0 2024-07-26T08:38:48.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.495+0000 7fd4eaa00640 1 -- 172.21.15.44:0/3740642569 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25172 (secure 0 0 0) 0x7fd4dc0ab140 con 0x7fd4f0076db0 2024-07-26T08:38:48.498 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:38:48.498 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":91,"fsid":"10471e4e-4b28-11ef-bcb7-c7b262605968","created":"2024-07-26T08:22:42.161051+0000","modified":"2024-07-26T08:34:13.946257+0000","last_up_change":"2024-07-26T08:31:50.733881+0000","last_in_change":"2024-07-26T08:31:31.738472+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T08:27:40.734083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T08:32:56.237662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T08:33:01.125230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T08:33:15.867271+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"96ac5961-fb53-49d7-925f-8be1ece073f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6803","nonce":3315928929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6805","nonce":3315928929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6809","nonce":3315928929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3315928929},{"type":"v1","addr":"172.21.15.44:6807","nonce":3315928929}]},"public_addr":"172.21.15.44:6803/3315928929","cluster_addr":"172.21.15.44:6805/3315928929","heartbeat_back_addr":"172.21.15.44:6809/3315928929","heartbeat_front_addr":"172.21.15.44:6807/3315928929","state":["exists","up"]},{"osd":1,"uuid":"853c8479-ad0a-4af4-996a-646245d17441","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6801","nonce":2431548674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6803","nonce":2431548674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6807","nonce":2431548674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2431548674},{"type":"v1","addr":"172.21.15.116:6805","nonce":2431548674}]},"public_addr":"172.21.15.116:6801/2431548674","cluster_addr":"172.21.15.116:6803/2431548674","heartbeat_back_addr":"172.21.15.116:6807/2431548674","heartbeat_front_addr":"172.21.15.116:6805/2431548674","state":["exists","up"]},{"osd":2,"uuid":"284a11b8-2736-4d1c-90b8-59bdb63c3311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6801","nonce":2604165495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6803","nonce":2604165495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6807","nonce":2604165495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2604165495},{"type":"v1","addr":"172.21.15.171:6805","nonce":2604165495}]},"public_addr":"172.21.15.171:6801/2604165495","cluster_addr":"172.21.15.171:6803/2604165495","heartbeat_back_addr":"172.21.15.171:6807/2604165495","heartbeat_front_addr":"172.21.15.171:6805/2604165495","state":["exists","up"]},{"osd":3,"uuid":"895978fc-b2e6-4251-bb79-9bcb77079e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6811","nonce":2033336910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6813","nonce":2033336910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6817","nonce":2033336910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":2033336910},{"type":"v1","addr":"172.21.15.44:6815","nonce":2033336910}]},"public_addr":"172.21.15.44:6811/2033336910","cluster_addr":"172.21.15.44:6813/2033336910","heartbeat_back_addr":"172.21.15.44:6817/2033336910","heartbeat_front_addr":"172.21.15.44:6815/2033336910","state":["exists","up"]},{"osd":4,"uuid":"1e29d022-e84f-43f7-badb-18797f4a3f67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6809","nonce":3661934334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6811","nonce":3661934334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6815","nonce":3661934334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3661934334},{"type":"v1","addr":"172.21.15.116:6813","nonce":3661934334}]},"public_addr":"172.21.15.116:6809/3661934334","cluster_addr":"172.21.15.116:6811/3661934334","heartbeat_back_addr":"172.21.15.116:6815/3661934334","heartbeat_front_addr":"172.21.15.116:6813/3661934334","state":["exists","up"]},{"osd":5,"uuid":"71c826e9-696b-47de-ae4c-023b02ceb6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6809","nonce":3188487524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6811","nonce":3188487524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6815","nonce":3188487524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3188487524},{"type":"v1","addr":"172.21.15.171:6813","nonce":3188487524}]},"public_addr":"172.21.15.171:6809/3188487524","cluster_addr":"172.21.15.171:6811/3188487524","heartbeat_back_addr":"172.21.15.171:6815/3188487524","heartbeat_front_addr":"172.21.15.171:6813/3188487524","state":["exists","up"]},{"osd":6,"uuid":"275225ee-11a7-4988-851b-7be0b8d4d155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6819","nonce":51483018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6821","nonce":51483018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6825","nonce":51483018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":51483018},{"type":"v1","addr":"172.21.15.44:6823","nonce":51483018}]},"public_addr":"172.21.15.44:6819/51483018","cluster_addr":"172.21.15.44:6821/51483018","heartbeat_back_addr":"172.21.15.44:6825/51483018","heartbeat_front_addr":"172.21.15.44:6823/51483018","state":["exists","up"]},{"osd":7,"uuid":"cbc14308-bca8-4549-8079-6483bda3f101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6817","nonce":776412453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6819","nonce":776412453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6823","nonce":776412453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":776412453},{"type":"v1","addr":"172.21.15.116:6821","nonce":776412453}]},"public_addr":"172.21.15.116:6817/776412453","cluster_addr":"172.21.15.116:6819/776412453","heartbeat_back_addr":"172.21.15.116:6823/776412453","heartbeat_front_addr":"172.21.15.116:6821/776412453","state":["exists","up"]},{"osd":8,"uuid":"20dacf62-1c49-4dd9-8d1f-467506b1d1bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6817","nonce":2299573831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6819","nonce":2299573831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6823","nonce":2299573831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":2299573831},{"type":"v1","addr":"172.21.15.171:6821","nonce":2299573831}]},"public_addr":"172.21.15.171:6817/2299573831","cluster_addr":"172.21.15.171:6819/2299573831","heartbeat_back_addr":"172.21.15.171:6823/2299573831","heartbeat_front_addr":"172.21.15.171:6821/2299573831","state":["exists","up"]},{"osd":9,"uuid":"8fa16cab-5114-438f-bb1e-e636366e290d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6827","nonce":2869135265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6829","nonce":2869135265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6833","nonce":2869135265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":2869135265},{"type":"v1","addr":"172.21.15.44:6831","nonce":2869135265}]},"public_addr":"172.21.15.44:6827/2869135265","cluster_addr":"172.21.15.44:6829/2869135265","heartbeat_back_addr":"172.21.15.44:6833/2869135265","heartbeat_front_addr":"172.21.15.44:6831/2869135265","state":["exists","up"]},{"osd":10,"uuid":"8cec6245-933e-4d2f-8d4f-5107276fe094","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6825","nonce":1630121926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6827","nonce":1630121926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6831","nonce":1630121926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1630121926},{"type":"v1","addr":"172.21.15.116:6829","nonce":1630121926}]},"public_addr":"172.21.15.116:6825/1630121926","cluster_addr":"172.21.15.116:6827/1630121926","heartbeat_back_addr":"172.21.15.116:6831/1630121926","heartbeat_front_addr":"172.21.15.116:6829/1630121926","state":["exists","up"]},{"osd":11,"uuid":"f406895e-aab0-4a79-bd14-20a6f0626b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6825","nonce":4225876845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6827","nonce":4225876845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6831","nonce":4225876845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":4225876845},{"type":"v1","addr":"172.21.15.171:6829","nonce":4225876845}]},"public_addr":"172.21.15.171:6825/4225876845","cluster_addr":"172.21.15.171:6827/4225876845","heartbeat_back_addr":"172.21.15.171:6831/4225876845","heartbeat_front_addr":"172.21.15.171:6829/4225876845","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:26:50.768114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:14.820290+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:27:37.528725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:10.679979+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:36.103215+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:28:59.492476+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:33.761588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:29:59.467021+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:23.888498+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:30:58.787175+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:25.234141+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T08:31:49.121917+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11},{"from":8,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2545437880":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1988892875":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1840681309":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6801/3951671566":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/475987513":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/1872480755":"2024-07-27T08:23:20.144063+0000","172.21.15.44:6800/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2363513350":"2024-07-27T08:23:50.924919+0000","172.21.15.44:6801/1673664031":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/2833279364":"2024-07-27T08:23:50.924919+0000","172.21.15.44:0/3302011051":"2024-07-27T08:23:20.144063+0000","172.21.15.44:0/487473":"2024-07-27T08:25:12.575949+0000","172.21.15.44:0/3491826252":"2024-07-27T08:25:12.575949+0000","172.21.15.44:6800/1840681309":"2024-07-27T08:25:12.575949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T08:38:48.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.499+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd4d80b3c20 msgr2=0x7fd4d80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:48.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.499+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd4d80b3c20 0x7fd4d80b6130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4031440 tx=0x7fd4e4009560 comp rx=0 tx=0).stop 2024-07-26T08:38:48.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.499+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 msgr2=0x7fd4f01a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:48.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.499+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f01a08e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00e9c0 tx=0x7fd4dc00edc0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 shutdown_connections 2024-07-26T08:38:48.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fd4d80b3c20 0x7fd4d80b6130 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4f0113c90 0x7fd4f01a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd4f0076db0 0x7fd4f01a08e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 --2- 172.21.15.44:0/3740642569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd4f0075800 0x7fd4f01a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:48.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 >> 172.21.15.44:0/3740642569 conn(0x7fd4f00fee10 msgr2=0x7fd4f010fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:48.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 shutdown_connections 2024-07-26T08:38:48.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:48.500+0000 7fd4f7200640 1 -- 172.21.15.44:0/3740642569 wait complete. 2024-07-26T08:38:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:48 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/3740642569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:48 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/3740642569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:48 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/3740642569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T08:38:49.493 DEBUG:tasks.fwd_scrub:Yielding 2024-07-26T08:38:49.493 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T08:38:49.503 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T08:38:49.503 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T08:38:49.503 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-07-26T08:38:49.504 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-07-26T08:38:49.504 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T08:38:49.504 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T08:38:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:49 smithi044 ceph-mon[32404]: pgmap v487: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 -- 172.21.15.44:0/3114534792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 msgr2=0x7f45f810fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 --2- 172.21.15.44:0/3114534792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 0x7f45f810fad0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009ca0 tx=0x7f45f002a8c0 comp rx=0 tx=0).stop 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 -- 172.21.15.44:0/3114534792 shutdown_connections 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 --2- 172.21.15.44:0/3114534792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 0x7f45f810fad0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 --2- 172.21.15.44:0/3114534792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 0x7f45f810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 --2- 172.21.15.44:0/3114534792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f81045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 -- 172.21.15.44:0/3114534792 >> 172.21.15.44:0/3114534792 conn(0x7f45f80fdc40 msgr2=0x7f45f8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 -- 172.21.15.44:0/3114534792 shutdown_connections 2024-07-26T08:38:49.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.781+0000 7f45fee00640 1 -- 172.21.15.44:0/3114534792 wait complete. 2024-07-26T08:38:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.783+0000 7f45fee00640 1 Processor -- start 2024-07-26T08:38:49.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.783+0000 7f45fee00640 1 -- start start 2024-07-26T08:38:49.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fee00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f819be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:49.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 0x7f45f819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:49.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fee00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 0x7f45f81a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:49.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fee00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f45f81132c0 con 0x7f45f81041d0 2024-07-26T08:38:49.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45f8113140 con 0x7f45f8104b20 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fee00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45f8113440 con 0x7f45f810d5f0 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fda00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f819be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fda00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f819be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41446/0 (socket says 172.21.15.44:41446) 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fda00640 1 -- 172.21.15.44:0/1567630485 learned_addr learned my addr 172.21.15.44:0/1567630485 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fd000640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 0x7f45f819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.784+0000 7f45fe400640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 0x7f45f81a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fd000640 1 -- 172.21.15.44:0/1567630485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 msgr2=0x7f45f81a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fd000640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 0x7f45f81a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fd000640 1 -- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 msgr2=0x7f45f819be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fd000640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f819be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:49.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fd000640 1 -- 172.21.15.44:0/1567630485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f81a1d50 con 0x7f45f8104b20 2024-07-26T08:38:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fda00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f819be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fd000640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 0x7f45f819c3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45f806a030 tx=0x7f45e400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:49.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e400cc40 con 0x7f45f8104b20 2024-07-26T08:38:49.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f81a2040 con 0x7f45f8104b20 2024-07-26T08:38:49.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45e400cde0 con 0x7f45f8104b20 2024-07-26T08:38:49.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.785+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4004a00 con 0x7f45f8104b20 2024-07-26T08:38:49.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.786+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f81a87d0 con 0x7f45f8104b20 2024-07-26T08:38:49.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.787+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f81045e0 con 0x7f45f8104b20 2024-07-26T08:38:49.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.789+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f45e4004ba0 con 0x7f45f8104b20 2024-07-26T08:38:49.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.791+0000 7f45f6000640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f45e00b3c20 0x7f45e00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:49.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.791+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f45e40e40d0 con 0x7f45f8104b20 2024-07-26T08:38:49.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.791+0000 7f45fda00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f45e00b3c20 0x7f45e00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:49.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.792+0000 7f45fda00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f45e00b3c20 0x7f45e00b6130 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f45ec011550 tx=0x7f45ec005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:49.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:49.795+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f45e40ab210 con 0x7f45f8104b20 2024-07-26T08:38:49.997 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:49 smithi116 ceph-mon[37097]: pgmap v487: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T08:38:50.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.048+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f45f81024f0 con 0x7f45e00b3c20 2024-07-26T08:38:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:49 smithi171 ceph-mon[37789]: pgmap v487: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T08:38:50.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.360+0000 7f45f6000640 1 -- 172.21.15.44:0/1567630485 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f45f81024f0 con 0x7f45e00b3c20 2024-07-26T08:38:50.362 INFO:teuthology.orchestra.run.smithi044.stdout:Schedule set for path / 2024-07-26T08:38:50.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f45e00b3c20 msgr2=0x7f45e00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:50.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f45e00b3c20 0x7f45e00b6130 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f45ec011550 tx=0x7f45ec005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 msgr2=0x7f45f819c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 0x7f45f819c3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f45f806a030 tx=0x7f45e400ee50 comp rx=0 tx=0).stop 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 shutdown_connections 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f45e00b3c20 0x7f45e00b6130 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45f810d5f0 0x7f45f81a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45f8104b20 0x7f45f819c3c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 --2- 172.21.15.44:0/1567630485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45f81041d0 0x7f45f819be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 >> 172.21.15.44:0/1567630485 conn(0x7f45f80fdc40 msgr2=0x7f45f8101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 shutdown_connections 2024-07-26T08:38:50.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.364+0000 7f45fee00640 1 -- 172.21.15.44:0/1567630485 wait complete. 2024-07-26T08:38:50.392 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 -- 172.21.15.44:0/3476838591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c104170 msgr2=0x7eff3c104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 --2- 172.21.15.44:0/3476838591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c104170 0x7eff3c104580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009900 tx=0x7eff2c02a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 -- 172.21.15.44:0/3476838591 shutdown_connections 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 --2- 172.21.15.44:0/3476838591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 0x7eff3c10d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 --2- 172.21.15.44:0/3476838591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c10d5d0 0x7eff3c10fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 --2- 172.21.15.44:0/3476838591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c104170 0x7eff3c104580 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 -- 172.21.15.44:0/3476838591 >> 172.21.15.44:0/3476838591 conn(0x7eff3c0fdc40 msgr2=0x7eff3c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 -- 172.21.15.44:0/3476838591 shutdown_connections 2024-07-26T08:38:50.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.650+0000 7eff3be00640 1 -- 172.21.15.44:0/3476838591 wait complete. 2024-07-26T08:38:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 Processor -- start 2024-07-26T08:38:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 -- start start 2024-07-26T08:38:50.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 0x7eff3c19bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 0x7eff3c19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c10d5d0 0x7eff3c1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c1132b0 con 0x7eff3c104170 2024-07-26T08:38:50.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c113130 con 0x7eff3c10d5d0 2024-07-26T08:38:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3be00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c113430 con 0x7eff3c104ac0 2024-07-26T08:38:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.652+0000 7eff3aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 0x7eff3c19bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:50.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 0x7eff3c19bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41470/0 (socket says 172.21.15.44:41470) 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 -- 172.21.15.44:0/2708985584 learned_addr learned my addr 172.21.15.44:0/2708985584 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff33e00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 0x7eff3c19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3b400640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c10d5d0 0x7eff3c1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 -- 172.21.15.44:0/2708985584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 msgr2=0x7eff3c19c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 0x7eff3c19c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 -- 172.21.15.44:0/2708985584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c10d5d0 msgr2=0x7eff3c1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c10d5d0 0x7eff3c1a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 -- 172.21.15.44:0/2708985584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c1a1e00 con 0x7eff3c104170 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff33e00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 0x7eff3c19c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3b400640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c10d5d0 0x7eff3c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.653+0000 7eff3aa00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 0x7eff3c19bed0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7eff2c005ab0 tx=0x7eff2c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.654+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c03f070 con 0x7eff3c104170 2024-07-26T08:38:50.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.654+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff2c031b60 con 0x7eff3c104170 2024-07-26T08:38:50.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.654+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c035030 con 0x7eff3c104170 2024-07-26T08:38:50.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.654+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3c1a2090 con 0x7eff3c104170 2024-07-26T08:38:50.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.654+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3c1a8880 con 0x7eff3c104170 2024-07-26T08:38:50.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.655+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3c104580 con 0x7eff3c104170 2024-07-26T08:38:50.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.658+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7eff2c042070 con 0x7eff3c104170 2024-07-26T08:38:50.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.660+0000 7eff33400640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7eff1c0b3c20 0x7eff1c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.661+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7eff2c1050d0 con 0x7eff3c104170 2024-07-26T08:38:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.661+0000 7eff33e00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7eff1c0b3c20 0x7eff1c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:50.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.661+0000 7eff33e00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7eff1c0b3c20 0x7eff1c0b6130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff3c10fb90 tx=0x7eff24005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:50.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.663+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7eff2c0cc210 con 0x7eff3c104170 2024-07-26T08:38:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:50 smithi044 ceph-mon[32404]: from='client.25469 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:50.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:50.909+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7eff3c100f00 con 0x7eff1c0b3c20 2024-07-26T08:38:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:50 smithi116 ceph-mon[37097]: from='client.25469 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:50 smithi171 ceph-mon[37789]: from='client.25469 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:51.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.086+0000 7eff33400640 1 -- 172.21.15.44:0/2708985584 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7eff3c100f00 con 0x7eff1c0b3c20 2024-07-26T08:38:51.088 INFO:teuthology.orchestra.run.smithi044.stdout:Retention added to path / 2024-07-26T08:38:51.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7eff1c0b3c20 msgr2=0x7eff1c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7eff1c0b3c20 0x7eff1c0b6130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff3c10fb90 tx=0x7eff24005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 msgr2=0x7eff3c19bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 0x7eff3c19bed0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7eff2c005ab0 tx=0x7eff2c003c10 comp rx=0 tx=0).stop 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 shutdown_connections 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7eff1c0b3c20 0x7eff1c0b6130 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff3c104ac0 0x7eff3c19c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c10d5d0 0x7eff3c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 --2- 172.21.15.44:0/2708985584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7eff3c104170 0x7eff3c19bed0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.090+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 >> 172.21.15.44:0/2708985584 conn(0x7eff3c0fdc40 msgr2=0x7eff3c1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.091+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 shutdown_connections 2024-07-26T08:38:51.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.091+0000 7eff3be00640 1 -- 172.21.15.44:0/2708985584 wait complete. 2024-07-26T08:38:51.122 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-26T08:38:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.392+0000 7fceeea00640 1 -- 172.21.15.44:0/2812541192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee80751e0 msgr2=0x7fcee80755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.392+0000 7fceeea00640 1 --2- 172.21.15.44:0/2812541192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee80751e0 0x7fcee80755f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fced8009900 tx=0x7fced802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.392+0000 7fceeea00640 1 -- 172.21.15.44:0/2812541192 shutdown_connections 2024-07-26T08:38:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.393+0000 7fceeea00640 1 --2- 172.21.15.44:0/2812541192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcee8075b30 0x7fcee810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.393+0000 7fceeea00640 1 --2- 172.21.15.44:0/2812541192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee810fa80 0x7fcee8111ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.393+0000 7fceeea00640 1 --2- 172.21.15.44:0/2812541192 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee80751e0 0x7fcee80755f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.393+0000 7fceeea00640 1 -- 172.21.15.44:0/2812541192 >> 172.21.15.44:0/2812541192 conn(0x7fcee80fef30 msgr2=0x7fcee8101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.393+0000 7fceeea00640 1 -- 172.21.15.44:0/2812541192 shutdown_connections 2024-07-26T08:38:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.393+0000 7fceeea00640 1 -- 172.21.15.44:0/2812541192 wait complete. 2024-07-26T08:38:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.394+0000 7fceeea00640 1 Processor -- start 2024-07-26T08:38:51.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.394+0000 7fceeea00640 1 -- start start 2024-07-26T08:38:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 0x7fcee819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeea00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee8075b30 0x7fcee819c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcee810fa80 0x7fcee81a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeea00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcee8115670 con 0x7fcee8075b30 2024-07-26T08:38:51.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcee81154f0 con 0x7fcee80751e0 2024-07-26T08:38:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeea00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcee81157f0 con 0x7fcee810fa80 2024-07-26T08:38:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 0x7fcee819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 0x7fcee819c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54934/0 (socket says 172.21.15.44:54934) 2024-07-26T08:38:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 -- 172.21.15.44:0/550409229 learned_addr learned my addr 172.21.15.44:0/550409229 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcedfe00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee8075b30 0x7fcee819c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fceeca00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcee810fa80 0x7fcee81a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 -- 172.21.15.44:0/550409229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcee810fa80 msgr2=0x7fcee81a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcee810fa80 0x7fcee81a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 -- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee8075b30 msgr2=0x7fcee819c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee8075b30 0x7fcee819c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.395+0000 7fcee7e00640 1 -- 172.21.15.44:0/550409229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee81a1fa0 con 0x7fcee80751e0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.396+0000 7fcee7e00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 0x7fcee819c070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fced8005940 tx=0x7fced80040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.396+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced803f030 con 0x7fcee80751e0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.396+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee81a2230 con 0x7fcee80751e0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.396+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fced80044c0 con 0x7fcee80751e0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.396+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8035070 con 0x7fcee80751e0 2024-07-26T08:38:51.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.396+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee81a8a20 con 0x7fcee80751e0 2024-07-26T08:38:51.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.397+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee80755f0 con 0x7fcee80751e0 2024-07-26T08:38:51.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.400+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7fced8042030 con 0x7fcee80751e0 2024-07-26T08:38:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.402+0000 7fcee5600640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcecc0b3c20 0x7fcecc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.402+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fced8104770 con 0x7fcee80751e0 2024-07-26T08:38:51.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.402+0000 7fcedfe00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcecc0b3c20 0x7fcecc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.403+0000 7fcedfe00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcecc0b3c20 0x7fcecc0b6130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcee8111db0 tx=0x7fced0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:51.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.403+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fced80cb260 con 0x7fcee80751e0 2024-07-26T08:38:51.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.659+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fcee8109d30 con 0x7fcecc0b3c20 2024-07-26T08:38:51.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.674+0000 7fcee5600640 1 -- 172.21.15.44:0/550409229 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fcee8109d30 con 0x7fcecc0b3c20 2024-07-26T08:38:51.676 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-07-26T00:00:00", "created": "2024-07-26T08:38:50", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-07-26T08:38:51.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.677+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcecc0b3c20 msgr2=0x7fcecc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.677+0000 7fceeea00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcecc0b3c20 0x7fcecc0b6130 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcee8111db0 tx=0x7fced0005f90 comp rx=0 tx=0).stop 2024-07-26T08:38:51.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.677+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 msgr2=0x7fcee819c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.677+0000 7fceeea00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 0x7fcee819c070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fced8005940 tx=0x7fced80040c0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 shutdown_connections 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcecc0b3c20 0x7fcecc0b6130 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcee810fa80 0x7fcee81a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcee8075b30 0x7fcee819c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 --2- 172.21.15.44:0/550409229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcee80751e0 0x7fcee819c070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 >> 172.21.15.44:0/550409229 conn(0x7fcee80fef30 msgr2=0x7fcee8109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 shutdown_connections 2024-07-26T08:38:51.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.678+0000 7fceeea00640 1 -- 172.21.15.44:0/550409229 wait complete. 2024-07-26T08:38:51.714 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-07-26T08:38:51.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:51 smithi044 ceph-mon[32404]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:51.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:51 smithi044 ceph-mon[32404]: pgmap v488: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 -- 172.21.15.44:0/3173472843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 msgr2=0x7f8e6c112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3173472843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c112000 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009be0 tx=0x7f8e6402a6c0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 -- 172.21.15.44:0/3173472843 shutdown_connections 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3173472843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c112000 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3173472843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c10b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3173472843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 0x7f8e6c105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 -- 172.21.15.44:0/3173472843 >> 172.21.15.44:0/3173472843 conn(0x7f8e6c100f90 msgr2=0x7f8e6c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:51.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.973+0000 7f8e70c00640 1 -- 172.21.15.44:0/3173472843 shutdown_connections 2024-07-26T08:38:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.974+0000 7f8e70c00640 1 -- 172.21.15.44:0/3173472843 wait complete. 2024-07-26T08:38:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.975+0000 7f8e70c00640 1 Processor -- start 2024-07-26T08:38:51.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.975+0000 7f8e70c00640 1 -- start start 2024-07-26T08:38:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e70c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 0x7f8e6c19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e70c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e70c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c1a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e70c00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c115480 con 0x7f8e6c10ae80 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e70c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c115300 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e70c00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c115600 con 0x7f8e6c105550 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e6b400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 0x7f8e6c19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e6aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e6b400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 0x7f8e6c19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35068/0 (socket says 172.21.15.44:35068) 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e6aa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c19c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47562/0 (socket says 172.21.15.44:47562) 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e6b400640 1 -- 172.21.15.44:0/3603742276 learned_addr learned my addr 172.21.15.44:0/3603742276 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:51.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.976+0000 7f8e6be00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c1a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6be00640 1 -- 172.21.15.44:0/3603742276 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 msgr2=0x7f8e6c19be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6be00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 0x7f8e6c19be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6be00640 1 -- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 msgr2=0x7f8e6c19c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6be00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c19c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6be00640 1 -- 172.21.15.44:0/3603742276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6c1a1d90 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6aa00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e6be00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c1a1690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009b30 tx=0x7f8e64003b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64041020 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e640318f0 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6c1a2020 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.977+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64036040 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.978+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e6c1a8810 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.979+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e6c1067c0 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.981+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f8e64044020 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.982+0000 7f8e63e00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8e500b3d30 0x7f8e500b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:51.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.983+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8e64105290 con 0x7f8e6c10b7d0 2024-07-26T08:38:51.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.983+0000 7f8e6b400640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8e500b3d30 0x7f8e500b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:51.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.983+0000 7f8e6b400640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8e500b3d30 0x7f8e500b6240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c005c80 tx=0x7f8e5c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:51.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:51.984+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8e6404b020 con 0x7f8e6c10b7d0 2024-07-26T08:38:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:51 smithi116 ceph-mon[37097]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:51 smithi116 ceph-mon[37097]: pgmap v488: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:51 smithi171 ceph-mon[37789]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:51 smithi171 ceph-mon[37789]: pgmap v488: 129 pgs: 129 active+clean; 656 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:38:52.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.225+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8e6c063230 con 0x7f8e500b3d30 2024-07-26T08:38:52.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.240+0000 7f8e63e00640 1 -- 172.21.15.44:0/3603742276 <== mgr.14211 v2:172.21.15.44:6800/2852968588 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8e6c063230 con 0x7f8e500b3d30 2024-07-26T08:38:52.241 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1m 6m3h 2024-07-26T08:38:52.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8e500b3d30 msgr2=0x7f8e500b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8e500b3d30 0x7f8e500b6240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c005c80 tx=0x7f8e5c009260 comp rx=0 tx=0).stop 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 msgr2=0x7f8e6c1a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c1a1690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009b30 tx=0x7f8e64003b80 comp rx=0 tx=0).stop 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 shutdown_connections 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f8e500b3d30 0x7f8e500b6240 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e6c10b7d0 0x7f8e6c1a1690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8e6c10ae80 0x7f8e6c19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 --2- 172.21.15.44:0/3603742276 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e6c105550 0x7f8e6c19be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 >> 172.21.15.44:0/3603742276 conn(0x7f8e6c100f90 msgr2=0x7f8e6c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 shutdown_connections 2024-07-26T08:38:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.243+0000 7f8e70c00640 1 -- 172.21.15.44:0/3603742276 wait complete. 2024-07-26T08:38:52.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 -- 172.21.15.44:0/2002291905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a4104ce0 msgr2=0x7f45a410d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 --2- 172.21.15.44:0/2002291905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a4104ce0 0x7f45a410d320 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4590009900 tx=0x7f459002a410 comp rx=0 tx=0).stop 2024-07-26T08:38:52.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 -- 172.21.15.44:0/2002291905 shutdown_connections 2024-07-26T08:38:52.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 --2- 172.21.15.44:0/2002291905 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a410d860 0x7f45a410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 --2- 172.21.15.44:0/2002291905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a4104ce0 0x7f45a410d320 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 --2- 172.21.15.44:0/2002291905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a41046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.269+0000 7f45ac000640 1 -- 172.21.15.44:0/2002291905 >> 172.21.15.44:0/2002291905 conn(0x7f45a4079160 msgr2=0x7f45a4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.270+0000 7f45ac000640 1 -- 172.21.15.44:0/2002291905 shutdown_connections 2024-07-26T08:38:52.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.270+0000 7f45ac000640 1 -- 172.21.15.44:0/2002291905 wait complete. 2024-07-26T08:38:52.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 Processor -- start 2024-07-26T08:38:52.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 -- start start 2024-07-26T08:38:52.273 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-07-26T08:38:52.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a4104ce0 0x7f45a419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a410d860 0x7f45a41a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4113460 con 0x7f45a41042b0 2024-07-26T08:38:52.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41132e0 con 0x7f45a410d860 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.271+0000 7f45ac000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41135e0 con 0x7f45a4104ce0 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a8c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a4104ce0 0x7f45a419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a419c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47588/0 (socket says 172.21.15.44:47588) 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 -- 172.21.15.44:0/1539304699 learned_addr learned my addr 172.21.15.44:0/1539304699 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45aa000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a410d860 0x7f45a41a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 -- 172.21.15.44:0/1539304699 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a4104ce0 msgr2=0x7f45a419c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a4104ce0 0x7f45a419c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 -- 172.21.15.44:0/1539304699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a410d860 msgr2=0x7f45a41a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a410d860 0x7f45a41a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.272+0000 7f45a9600640 1 -- 172.21.15.44:0/1539304699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a41a2000 con 0x7f45a41042b0 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.273+0000 7f45a9600640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a419c090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f459800e9c0 tx=0x7f459800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:52.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.273+0000 7f45a2000640 1 -- 172.21.15.44:0/1539304699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459800ccc0 con 0x7f45a41042b0 2024-07-26T08:38:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.273+0000 7f45a2000640 1 -- 172.21.15.44:0/1539304699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45980035f0 con 0x7f45a41042b0 2024-07-26T08:38:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.273+0000 7f45a2000640 1 -- 172.21.15.44:0/1539304699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459800ce60 con 0x7f45a41042b0 2024-07-26T08:38:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.273+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a41a22f0 con 0x7f45a41042b0 2024-07-26T08:38:52.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.273+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a41a8a50 con 0x7f45a41042b0 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.278+0000 7f45b2c33740 1 Processor -- start 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.278+0000 7f45b2c33740 1 -- start start 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.278+0000 7f45a2000640 1 -- 172.21.15.44:0/1539304699 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f45980037a0 con 0x7f45a41042b0 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.278+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45b2c33740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5645ea82bbe0 con 0x5645ea4e7490 2024-07-26T08:38:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45b2c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5645ea81b0e0 con 0x5645ea4ef670 2024-07-26T08:38:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45b2c33740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5645ea81f120 con 0x5645ea510350 2024-07-26T08:38:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45aa000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54674/0 (socket says 172.21.15.44:54674) 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 -- 172.21.15.44:0/3580415991 learned_addr learned my addr 172.21.15.44:0/3580415991 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 -- 172.21.15.44:0/3580415991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 msgr2=0x5645ea8b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 -- 172.21.15.44:0/3580415991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 msgr2=0x5645ea8b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.279+0000 7f45a8c00640 1 -- 172.21.15.44:0/3580415991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ea57b930 con 0x5645ea4ef670 2024-07-26T08:38:52.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.280+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8b5800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4590009900 tx=0x7f4590032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73686beb4b6a7333 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:52.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.280+0000 7f4596600640 1 -- 172.21.15.44:0/3580415991 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590040070 con 0x5645ea4ef670 2024-07-26T08:38:52.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.280+0000 7f4596600640 1 -- 172.21.15.44:0/3580415991 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4590004030 con 0x5645ea4ef670 2024-07-26T08:38:52.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.280+0000 7f4596600640 1 -- 172.21.15.44:0/3580415991 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590035040 con 0x5645ea4ef670 2024-07-26T08:38:52.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.281+0000 7f45a2000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f458c0b3cb0 0x7f458c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.281+0000 7f45a2000640 1 -- 172.21.15.44:0/1539304699 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f45980e4620 con 0x7f45a41042b0 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.281+0000 7f45aa000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f458c0b3cb0 0x7f458c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 -- 172.21.15.44:0/3580415991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 msgr2=0x5645ea8b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8b5800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4590009900 tx=0x7f4590032b20 comp rx=0 tx=0).stop 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 -- 172.21.15.44:0/3580415991 shutdown_connections 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8b5800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3580415991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 -- 172.21.15.44:0/3580415991 >> 172.21.15.44:0/3580415991 conn(0x5645ea504740 msgr2=0x5645ea8b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 -- 172.21.15.44:0/3580415991 shutdown_connections 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45aa000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f458c0b3cb0 0x7f458c0b61c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f459c005fd0 tx=0x7f459c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:52.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.282+0000 7f45b2c33740 1 -- 172.21.15.44:0/3580415991 wait complete. 2024-07-26T08:38:52.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.283+0000 7f45b2c33740 1 -- start start 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.283+0000 7f45b2c33740 1 Processor -- start 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8c0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea8c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5645ea67e890 con 0x5645ea4e7490 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5645ea7b7320 con 0x5645ea510350 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5645ea8c3500 con 0x5645ea4ef670 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45b2c33740 10 client.0 _ll_register_callbacks cb 0x5645ea7e3000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45a8c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea8c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8c0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45a8c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea8c09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35096/0 (socket says 172.21.15.44:35096) 2024-07-26T08:38:52.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45a8c00640 1 -- 172.21.15.44:0/3837871831 learned_addr learned my addr 172.21.15.44:0/3837871831 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:52.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.284+0000 7f45aa000640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a8c00640 1 -- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 msgr2=0x5645ea8cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a8c00640 1 -- 172.21.15.44:0/3837871831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 msgr2=0x5645ea8c0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8c0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a8c00640 1 -- 172.21.15.44:0/3837871831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ea8c11f0 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a9600640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8c0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea8c09c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f459002a930 tx=0x7f459002abb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba51e7a3c543f64 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.285+0000 7f458aa00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.286+0000 7f458aa00640 1 -- 172.21.15.44:0/3837871831 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590035070 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.286+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4590004030 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.286+0000 7f458aa00640 1 -- 172.21.15.44:0/3837871831 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4590005480 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.286+0000 7f458aa00640 1 -- 172.21.15.44:0/3837871831 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4590058050 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.286+0000 7f45b2c33740 10 client.25501 fetch_fsmap learned FSMap version 30 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.286+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5645ea8cf3f0 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f458aa00640 1 -- 172.21.15.44:0/3837871831 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4590004030 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f458aa00640 1 -- 172.21.15.44:0/3837871831 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4590004be0 con 0x5645ea4ef670 2024-07-26T08:38:52.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45b2c33740 10 client.25501 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:38:52.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45b2c33740 10 client.25501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:38:52.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45b2c33740 20 client.25501 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:52.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4cecd0 0x5645ea8d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45b2c33740 4 client.25501 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:52.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645ea82db30 con 0x5645ea4cecd0 2024-07-26T08:38:52.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.287+0000 7f45a9600640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4cecd0 0x5645ea8d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.288+0000 7f45a9600640 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4cecd0 0x5645ea8d2750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=406bc1eb4362979b server_cookie=6215a8ea6fb43e7e in_seq=0 out_seq=0 2024-07-26T08:38:52.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.288+0000 7f458aa00640 10 client.25501 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f458aa00640 1 -- 172.21.15.44:0/3837871831 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5645ea82db30 con 0x5645ea4cecd0 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f458aa00640 10 client.25501 handle_command_reply: tid=0 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 client.25501 shutdown 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 msgr2=0x5645ea8c09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea8c09c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f459002a930 tx=0x7f459002abb0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 shutdown_connections 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4cecd0 0x5645ea8d2750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea8c09c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3837871831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8c0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f458aa00640 0 client.25501 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 >> 172.21.15.44:0/3837871831 conn(0x5645ea504740 msgr2=0x5645ea8b7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.289+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 shutdown_connections 2024-07-26T08:38:52.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.290+0000 7f45b2c33740 1 -- 172.21.15.44:0/3837871831 wait complete. 2024-07-26T08:38:52.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.290+0000 7f45b2c33740 20 client.25501 trim_cache size 0 max 16384 2024-07-26T08:38:52.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.309+0000 7f45b2c33740 1 Processor -- start 2024-07-26T08:38:52.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.309+0000 7f45b2c33740 1 -- start start 2024-07-26T08:38:52.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8bbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8c2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45b2c33740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41b2a90 con 0x5645ea4e7490 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45b2c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41b45d0 con 0x5645ea4ef670 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45b2c33740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41b3160 con 0x5645ea510350 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8bbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8bbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47622/0 (socket says 172.21.15.44:47622) 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45a9600640 1 -- 172.21.15.44:0/2446372884 learned_addr learned my addr 172.21.15.44:0/2446372884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:52.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45a8c00640 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.310+0000 7f45aa000640 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a8c00640 1 -- 172.21.15.44:0/2446372884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 msgr2=0x5645ea8b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a8c00640 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a8c00640 1 -- 172.21.15.44:0/2446372884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 msgr2=0x5645ea8bbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a8c00640 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8bbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a8c00640 1 -- 172.21.15.44:0/2446372884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ea8c1740 con 0x5645ea4ef670 2024-07-26T08:38:52.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a9600640 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8bbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:38:52.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.311+0000 7f45a8c00640 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8c2360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4590004fb0 tx=0x7f459002a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ae94bd3693eebf5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:52.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.312+0000 7f4595c00640 1 -- 172.21.15.44:0/2446372884 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590060020 con 0x5645ea4ef670 2024-07-26T08:38:52.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.312+0000 7f4595c00640 1 -- 172.21.15.44:0/2446372884 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45900038c0 con 0x5645ea4ef670 2024-07-26T08:38:52.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.312+0000 7f4595c00640 1 -- 172.21.15.44:0/2446372884 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590035040 con 0x5645ea4ef670 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 -- 172.21.15.44:0/2446372884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 msgr2=0x5645ea8c2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8c2360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4590004fb0 tx=0x7f459002a410 comp rx=0 tx=0).stop 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 -- 172.21.15.44:0/2446372884 shutdown_connections 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea510350 0x5645ea8b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea4ef670 0x5645ea8c2360 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 --2- 172.21.15.44:0/2446372884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea8bbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 -- 172.21.15.44:0/2446372884 >> 172.21.15.44:0/2446372884 conn(0x5645ea504740 msgr2=0x5645ea8b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 -- 172.21.15.44:0/2446372884 shutdown_connections 2024-07-26T08:38:52.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.313+0000 7f45b2c33740 1 -- 172.21.15.44:0/2446372884 wait complete. 2024-07-26T08:38:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.314+0000 7f45b2c33740 1 -- start start 2024-07-26T08:38:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.314+0000 7f45b2c33740 1 Processor -- start 2024-07-26T08:38:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea932f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea933520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8d2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41b3f00 con 0x5645ea4e7490 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41b3830 con 0x5645ea510350 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41b0a70 con 0x5645ea4ef670 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45b2c33740 10 client.0 _ll_register_callbacks cb 0x7f45980ea400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea932f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45a8c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea933520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45a9600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea932f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47632/0 (socket says 172.21.15.44:47632) 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45a9600640 1 -- 172.21.15.44:0/3630064945 learned_addr learned my addr 172.21.15.44:0/3630064945 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:52.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.315+0000 7f45aa000640 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8d2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45a8c00640 1 -- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 msgr2=0x5645ea8d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8d2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45a8c00640 1 -- 172.21.15.44:0/3630064945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 msgr2=0x5645ea932f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea932f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45a8c00640 1 -- 172.21.15.44:0/3630064945 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ea8d34c0 con 0x5645ea4ef670 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45a8c00640 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea933520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4590005080 tx=0x7f4590000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac518c858a989a2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f458aa00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f458aa00640 1 -- 172.21.15.44:0/3630064945 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590060030 con 0x5645ea4ef670 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45900038c0 con 0x5645ea4ef670 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.316+0000 7f458aa00640 1 -- 172.21.15.44:0/3630064945 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f45900314a0 con 0x5645ea4ef670 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.317+0000 7f458aa00640 1 -- 172.21.15.44:0/3630064945 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4590073050 con 0x5645ea4ef670 2024-07-26T08:38:52.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.317+0000 7f45b2c33740 10 client.25507 fetch_fsmap learned FSMap version 30 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.317+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5645ea93a6c0 con 0x5645ea4ef670 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.317+0000 7f458aa00640 1 -- 172.21.15.44:0/3630064945 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45900038c0 con 0x5645ea4ef670 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f458aa00640 1 -- 172.21.15.44:0/3630064945 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4590004730 con 0x5645ea4ef670 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45b2c33740 10 client.25507 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45b2c33740 10 client.25507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45b2c33740 20 client.25507 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4a4080 0x5645ea93cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:52.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45b2c33740 4 client.25507 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5645ea8295d0 con 0x5645ea4a4080 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.318+0000 7f45a9600640 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4a4080 0x5645ea93cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.319+0000 7f45a9600640 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4a4080 0x5645ea93cc80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db327c3fb2301bd5 server_cookie=2d079493e5396045 in_seq=0 out_seq=0 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.319+0000 7f458aa00640 10 client.25507 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.320+0000 7f458aa00640 1 -- 172.21.15.44:0/3630064945 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5645ea8295d0 con 0x5645ea4a4080 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.320+0000 7f458aa00640 10 client.25507 handle_command_reply: tid=0 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.320+0000 7f45b2c33740 1 client.25507 shutdown 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.320+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 msgr2=0x5645ea933520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.320+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea933520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4590005080 tx=0x7f4590000bc0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.320+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 shutdown_connections 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5645ea4a4080 0x5645ea93cc80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5645ea4ef670 0x5645ea933520 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645ea510350 0x5645ea8d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 --2- 172.21.15.44:0/3630064945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5645ea4e7490 0x5645ea932f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f458aa00640 0 client.25507 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 >> 172.21.15.44:0/3630064945 conn(0x5645ea504740 msgr2=0x5645ea8b6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 shutdown_connections 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 1 -- 172.21.15.44:0/3630064945 wait complete. 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.321+0000 7f45b2c33740 20 client.25507 trim_cache size 0 max 16384 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "b8d04a66-208a-4e89-9fcd-6892679c2e07", 2024-07-26T08:38:52.327 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-07-26T08:38:52.328 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:52.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f458c0b3cb0 msgr2=0x7f458c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f458c0b3cb0 0x7f458c0b61c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f459c005fd0 tx=0x7f459c009500 comp rx=0 tx=0).stop 2024-07-26T08:38:52.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 msgr2=0x7f45a419c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a419c090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f459800e9c0 tx=0x7f459800edc0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 shutdown_connections 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f458c0b3cb0 0x7f458c0b61c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a4104ce0 0x7f45a419c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a410d860 0x7f45a41a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 --2- 172.21.15.44:0/1539304699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f45a41042b0 0x7f45a419c090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 >> 172.21.15.44:0/1539304699 conn(0x7f45a4079160 msgr2=0x7f45a40fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 shutdown_connections 2024-07-26T08:38:52.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:52.322+0000 7f45ac000640 1 -- 172.21.15.44:0/1539304699 wait complete. 2024-07-26T08:38:52.347 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-07-26T08:38:52.380 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T08:38:52.417 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:38:52.418 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T08:38:52.418 INFO:tasks.workunit:timeout=3h 2024-07-26T08:38:52.418 INFO:tasks.workunit:cleanup=True 2024-07-26T08:38:52.419 DEBUG:teuthology.orchestra.run.smithi044:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:52.464 INFO:teuthology.orchestra.run.smithi044.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-07-26 08:34:53.456786510 +0000 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-07-26 08:34:53.456786510 +0000 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-07-26 08:38:30.869688338 +0000 2024-07-26T08:38:52.465 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-07-26 08:34:53.456786510 +0000 2024-07-26T08:38:52.465 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-26T08:38:52.466 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-26T08:38:52.611 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T08:38:52.675 INFO:tasks.workunit.client.0.smithi044.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T08:38:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:52 smithi044 ceph-mon[32404]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:52 smithi044 ceph-mon[32404]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T08:38:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:52 smithi044 ceph-mon[32404]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:52 smithi116 ceph-mon[37097]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:52 smithi116 ceph-mon[37097]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T08:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:52 smithi116 ceph-mon[37097]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:52 smithi171 ceph-mon[37789]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:52 smithi171 ceph-mon[37789]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T08:38:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:52 smithi171 ceph-mon[37789]: from='client.25493 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T08:38:53.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T08:38:53.717 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:38:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:53 smithi044 ceph-mon[32404]: scrub queued for path: / 2024-07-26T08:38:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:53 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:38:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:53 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:38:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:53 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:38:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:53 smithi044 ceph-mon[32404]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 55 op/s 2024-07-26T08:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:53 smithi116 ceph-mon[37097]: scrub queued for path: / 2024-07-26T08:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:53 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:53 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:53 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:53 smithi116 ceph-mon[37097]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 55 op/s 2024-07-26T08:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:53 smithi171 ceph-mon[37789]: scrub queued for path: / 2024-07-26T08:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:53 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:53 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:53 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:53 smithi171 ceph-mon[37789]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 55 op/s 2024-07-26T08:38:56.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 -- 172.21.15.44:0/2772583793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 msgr2=0x7ffbcc10b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 --2- 172.21.15.44:0/2772583793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 0x7ffbcc10b340 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009900 tx=0x7ffbb402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 -- 172.21.15.44:0/2772583793 shutdown_connections 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 --2- 172.21.15.44:0/2772583793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbcc10b880 0x7ffbcc1120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 --2- 172.21.15.44:0/2772583793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 0x7ffbcc10b340 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 --2- 172.21.15.44:0/2772583793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 -- 172.21.15.44:0/2772583793 >> 172.21.15.44:0/2772583793 conn(0x7ffbcc101060 msgr2=0x7ffbcc1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:56.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 -- 172.21.15.44:0/2772583793 shutdown_connections 2024-07-26T08:38:56.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.012+0000 7ffbd1400640 1 -- 172.21.15.44:0/2772583793 wait complete. 2024-07-26T08:38:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.014+0000 7ffbd1400640 1 Processor -- start 2024-07-26T08:38:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.014+0000 7ffbd1400640 1 -- start start 2024-07-26T08:38:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd1400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc19bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 0x7ffbcc19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd1400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbcc10b880 0x7ffbcc1a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd1400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc115560 con 0x7ffbcc10b880 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1153e0 con 0x7ffbcc10af30 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd1400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1156e0 con 0x7ffbcc1055f0 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc19bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc19bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:35128/0 (socket says 172.21.15.44:35128) 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 -- 172.21.15.44:0/1479316409 learned_addr learned my addr 172.21.15.44:0/1479316409 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcb400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 0x7ffbcc19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbcc10b880 0x7ffbcc1a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 -- 172.21.15.44:0/1479316409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 msgr2=0x7ffbcc19c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 0x7ffbcc19c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 -- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbcc10b880 msgr2=0x7ffbcc1a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbcc10b880 0x7ffbcc1a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.015+0000 7ffbcbe00640 1 -- 172.21.15.44:0/1479316409 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbcc1a1ec0 con 0x7ffbcc1055f0 2024-07-26T08:38:56.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.016+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc19bf50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00e9f0 tx=0x7ffbbc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:56.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.016+0000 7ffbc8c00640 1 -- 172.21.15.44:0/1479316409 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00cd80 con 0x7ffbcc1055f0 2024-07-26T08:38:56.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.016+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbcc1a21b0 con 0x7ffbcc1055f0 2024-07-26T08:38:56.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.016+0000 7ffbc8c00640 1 -- 172.21.15.44:0/1479316409 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbbc0054d0 con 0x7ffbcc1055f0 2024-07-26T08:38:56.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.016+0000 7ffbc8c00640 1 -- 172.21.15.44:0/1479316409 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc010690 con 0x7ffbcc1055f0 2024-07-26T08:38:56.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.016+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbcc1a8910 con 0x7ffbcc1055f0 2024-07-26T08:38:56.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.020+0000 7ffbd95da740 1 Processor -- start 2024-07-26T08:38:56.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.020+0000 7ffbd95da740 1 -- start start 2024-07-26T08:38:56.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.020+0000 7ffbc8c00640 1 -- 172.21.15.44:0/1479316409 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7ffbbc004e90 con 0x7ffbcc1055f0 2024-07-26T08:38:56.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.020+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x560398269000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.020+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x560398269540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.020+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x560398269a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd95da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5603981dfbc0 con 0x560397ea3390 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd95da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5603981cf0c0 con 0x560397ec4070 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd95da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5603981d3100 con 0x560397e9b1b0 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x560398269000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbcb400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x560398269a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x560398269000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54718/0 (socket says 172.21.15.44:54718) 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2602789624 learned_addr learned my addr 172.21.15.44:0/2602789624 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x560398269540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2602789624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 msgr2=0x560398269540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x560398269540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2602789624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 msgr2=0x560398269a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x560398269a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2602789624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560397f2f650 con 0x560397ec4070 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.021+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x560398269540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.022+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x560398269000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4010370 tx=0x7ffbc4010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c339bff8fc07d2d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.022+0000 7ffbc1200640 1 -- 172.21.15.44:0/2602789624 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4016070 con 0x560397ec4070 2024-07-26T08:38:56.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.022+0000 7ffbc1200640 1 -- 172.21.15.44:0/2602789624 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbc4004d20 con 0x560397ec4070 2024-07-26T08:38:56.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.022+0000 7ffbc1200640 1 -- 172.21.15.44:0/2602789624 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4005420 con 0x560397ec4070 2024-07-26T08:38:56.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.022+0000 7ffbc8c00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbac0b3cb0 0x7ffbac0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbc8c00640 1 -- 172.21.15.44:0/1479316409 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ffbbc0e46d0 con 0x7ffbcc1055f0 2024-07-26T08:38:56.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbac0b3cb0 0x7ffbac0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 -- 172.21.15.44:0/2602789624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 msgr2=0x560398269000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x560398269000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4010370 tx=0x7ffbc4010830 comp rx=0 tx=0).stop 2024-07-26T08:38:56.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 -- 172.21.15.44:0/2602789624 shutdown_connections 2024-07-26T08:38:56.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x560398269540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x560398269a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2602789624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x560398269000 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 -- 172.21.15.44:0/2602789624 >> 172.21.15.44:0/2602789624 conn(0x560397eb8460 msgr2=0x560398266ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:56.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.023+0000 7ffbd95da740 1 -- 172.21.15.44:0/2602789624 shutdown_connections 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.024+0000 7ffbd95da740 1 -- 172.21.15.44:0/2602789624 wait complete. 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.024+0000 7ffbcbe00640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbac0b3cb0 0x7ffbac0b61c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00c6a0 tx=0x7ffbbc004a90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.024+0000 7ffbd95da740 1 -- start start 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.024+0000 7ffbd95da740 1 Processor -- start 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397e9b1b0 0x560398274510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ea3390 0x560398274a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ec4070 0x560398274f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560398032700 con 0x560397ec4070 2024-07-26T08:38:56.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56039816b690 con 0x560397e9b1b0 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5603982777e0 con 0x560397ea3390 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd95da740 10 client.0 _ll_register_callbacks cb 0x56039817b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397e9b1b0 0x560398274510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbcb400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ea3390 0x560398274a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397e9b1b0 0x560398274510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:54728/0 (socket says 172.21.15.44:54728) 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd0a00640 1 -- 172.21.15.44:0/341819558 learned_addr learned my addr 172.21.15.44:0/341819558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.025+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ec4070 0x560398274f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.034 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbcb400640 1 -- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397e9b1b0 msgr2=0x560398274510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbcb400640 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397e9b1b0 0x560398274510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbcb400640 1 -- 172.21.15.44:0/341819558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ec4070 msgr2=0x560398274f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbcb400640 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ec4070 0x560398274f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbcb400640 1 -- 172.21.15.44:0/341819558 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560398182830 con 0x560397ea3390 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbcb400640 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ea3390 0x560398274a50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009230 tx=0x7ffbb4004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7bd250786a2e80a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbb9600640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.026+0000 7ffbb9600640 1 -- 172.21.15.44:0/341819558 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4035040 con 0x560397ea3390 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.027+0000 7ffbb9600640 1 -- 172.21.15.44:0/341819558 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ffbb4005510 con 0x560397ea3390 2024-07-26T08:38:56.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.027+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc4004d20 con 0x560397ea3390 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.027+0000 7ffbb9600640 1 -- 172.21.15.44:0/341819558 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffbb404f050 con 0x560397ea3390 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.027+0000 7ffbd95da740 10 client.25519 fetch_fsmap learned FSMap version 30 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.027+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5603982833a0 con 0x560397ea3390 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.027+0000 7ffbb9600640 1 -- 172.21.15.44:0/341819558 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbc4004d20 con 0x560397ea3390 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbb9600640 1 -- 172.21.15.44:0/341819558 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ffbb4003900 con 0x560397ea3390 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd95da740 10 client.25519 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:38:56.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd95da740 10 client.25519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd95da740 20 client.25519 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd95da740 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e829f0 0x5603982868c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd95da740 4 client.25519 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603981e1b10 con 0x560397e829f0 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.028+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e829f0 0x5603982868c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.029+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e829f0 0x5603982868c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bae0b5c901ffc48 server_cookie=bbca5b5c55418c0e in_seq=0 out_seq=0 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.029+0000 7ffbb9600640 10 client.25519 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.029+0000 7ffbb9600640 1 -- 172.21.15.44:0/341819558 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5603981e1b10 con 0x560397e829f0 2024-07-26T08:38:56.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.029+0000 7ffbb9600640 10 client.25519 handle_command_reply: tid=0 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.029+0000 7ffbd95da740 1 client.25519 shutdown 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ea3390 msgr2=0x560398274a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ea3390 0x560398274a50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009230 tx=0x7ffbb4004430 comp rx=0 tx=0).stop 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 shutdown_connections 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e829f0 0x5603982868c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ea3390 0x560398274a50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ec4070 0x560398274f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 --2- 172.21.15.44:0/341819558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397e9b1b0 0x560398274510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbb9600640 0 client.25519 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:56.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 >> 172.21.15.44:0/341819558 conn(0x560397eb8460 msgr2=0x56039826c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:56.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 shutdown_connections 2024-07-26T08:38:56.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 1 -- 172.21.15.44:0/341819558 wait complete. 2024-07-26T08:38:56.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.030+0000 7ffbd95da740 20 client.25519 trim_cache size 0 max 16384 2024-07-26T08:38:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.049+0000 7ffbd95da740 1 Processor -- start 2024-07-26T08:38:56.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.049+0000 7ffbd95da740 1 -- start start 2024-07-26T08:38:56.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x5603982763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x56039826a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x56039828c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd95da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b2890 con 0x560397ea3390 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd95da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b43d0 con 0x560397ec4070 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd95da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b2f60 con 0x560397e9b1b0 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbcb400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x56039826a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbcb400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x56039826a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47680/0 (socket says 172.21.15.44:47680) 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbcb400640 1 -- 172.21.15.44:0/3696940171 learned_addr learned my addr 172.21.15.44:0/3696940171 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x56039828c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.050+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x5603982763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbcb400640 1 -- 172.21.15.44:0/3696940171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 msgr2=0x5603982763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbcb400640 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x5603982763f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbcb400640 1 -- 172.21.15.44:0/3696940171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 msgr2=0x56039828c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbcb400640 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x56039828c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbcb400640 1 -- 172.21.15.44:0/3696940171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603982757d0 con 0x560397ea3390 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbcb400640 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x56039826a740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402a9e0 tx=0x7ffbb4009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eac203e5243ab75 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbbbe00640 1 -- 172.21.15.44:0/3696940171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4035030 con 0x560397ea3390 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbbbe00640 1 -- 172.21.15.44:0/3696940171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb4041070 con 0x560397ea3390 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.051+0000 7ffbbbe00640 1 -- 172.21.15.44:0/3696940171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40317c0 con 0x560397ea3390 2024-07-26T08:38:56.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 -- 172.21.15.44:0/3696940171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 msgr2=0x56039826a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x56039826a740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402a9e0 tx=0x7ffbb4009830 comp rx=0 tx=0).stop 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 -- 172.21.15.44:0/3696940171 shutdown_connections 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ec4070 0x56039828c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397ea3390 0x56039826a740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 --2- 172.21.15.44:0/3696940171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397e9b1b0 0x5603982763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 -- 172.21.15.44:0/3696940171 >> 172.21.15.44:0/3696940171 conn(0x560397eb8460 msgr2=0x56039826a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.052+0000 7ffbd95da740 1 -- 172.21.15.44:0/3696940171 shutdown_connections 2024-07-26T08:38:56.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.053+0000 7ffbd95da740 1 -- 172.21.15.44:0/3696940171 wait complete. 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.053+0000 7ffbd95da740 1 -- start start 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.053+0000 7ffbd95da740 1 Processor -- start 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 0x56039827b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ea3390 0x56039827b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ec4070 0x5603982ee060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b3d00 con 0x560397e9b1b0 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b3630 con 0x560397ea3390 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b0870 con 0x560397ec4070 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd95da740 10 client.0 _ll_register_callbacks cb 0x7ffbb4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd0a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 0x56039827b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd0a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 0x56039827b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47686/0 (socket says 172.21.15.44:47686) 2024-07-26T08:38:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2981857558 learned_addr learned my addr 172.21.15.44:0/2981857558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.054+0000 7ffbcb400640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ea3390 0x56039827b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ec4070 0x5603982ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2981857558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ec4070 msgr2=0x5603982ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ec4070 0x5603982ee060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ea3390 msgr2=0x56039827b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ea3390 0x56039827b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 -- 172.21.15.44:0/2981857558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603982ee5f0 con 0x560397e9b1b0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbd0a00640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 0x56039827b160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4010d30 tx=0x7ffbb4005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e02d61a0074fee server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbb9600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.055+0000 7ffbb9600640 1 -- 172.21.15.44:0/2981857558 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc403c020 con 0x560397e9b1b0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbb9600640 1 -- 172.21.15.44:0/2981857558 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7ffbc4013b20 con 0x560397e9b1b0 2024-07-26T08:38:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb4041070 con 0x560397e9b1b0 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbb9600640 1 -- 172.21.15.44:0/2981857558 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffbc4016040 con 0x560397e9b1b0 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbd95da740 10 client.15402 fetch_fsmap learned FSMap version 30 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5603982ef880 con 0x560397e9b1b0 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbb9600640 1 -- 172.21.15.44:0/2981857558 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb4041070 con 0x560397e9b1b0 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbb9600640 1 -- 172.21.15.44:0/2981857558 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ffbc4028040 con 0x560397e9b1b0 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbd95da740 10 client.15402 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbd95da740 10 client.15402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.056+0000 7ffbd95da740 20 client.15402 populate_metadata read hostname 'smithi044' 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.057+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e57da0 0x5603982f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.057+0000 7ffbd95da740 4 client.15402 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.057+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5603981dd5b0 con 0x560397e57da0 2024-07-26T08:38:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.057+0000 7ffbcb400640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e57da0 0x5603982f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:38:56.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.057+0000 7ffbcb400640 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e57da0 0x5603982f1d70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7e993272fd489c2 server_cookie=b3cd0835647a8596 in_seq=0 out_seq=0 2024-07-26T08:38:56.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.057+0000 7ffbb9600640 10 client.15402 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:56.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:55 smithi044 ceph-mon[32404]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T08:38:56.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbb9600640 1 -- 172.21.15.44:0/2981857558 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5603981dd5b0 con 0x560397e57da0 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbb9600640 10 client.15402 handle_command_reply: tid=0 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 client.15402 shutdown 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 msgr2=0x56039827b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 0x56039827b160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4010d30 tx=0x7ffbb4005ec0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 shutdown_connections 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x560397e57da0 0x5603982f1d70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbb9600640 0 client.15402 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560397ec4070 0x5603982ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560397ea3390 0x56039827b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 --2- 172.21.15.44:0/2981857558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560397e9b1b0 0x56039827b160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 >> 172.21.15.44:0/2981857558 conn(0x560397eb8460 msgr2=0x560398276130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 shutdown_connections 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.058+0000 7ffbd95da740 1 -- 172.21.15.44:0/2981857558 wait complete. 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd95da740 20 client.15402 trim_cache size 0 max 16384 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stdout: "b8d04a66-208a-4e89-9fcd-6892679c2e07": { 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-07-26T08:38:56.063 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "b8d04a66-208a-4e89-9fcd-6892679c2e07", 2024-07-26T08:38:56.064 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-07-26T08:38:56.064 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-07-26T08:38:56.064 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-07-26T08:38:56.064 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:38:56.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbac0b3cb0 msgr2=0x7ffbac0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd1400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbac0b3cb0 0x7ffbac0b61c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00c6a0 tx=0x7ffbbc004a90 comp rx=0 tx=0).stop 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 msgr2=0x7ffbcc19bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd1400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc19bf50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00e9f0 tx=0x7ffbbc00edf0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 shutdown_connections 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.059+0000 7ffbd1400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7ffbac0b3cb0 0x7ffbac0b61c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.060+0000 7ffbd1400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbcc10b880 0x7ffbcc1a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.060+0000 7ffbd1400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbcc10af30 0x7ffbcc19c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.060+0000 7ffbd1400640 1 --2- 172.21.15.44:0/1479316409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffbcc1055f0 0x7ffbcc19bf50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.060+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 >> 172.21.15.44:0/1479316409 conn(0x7ffbcc101060 msgr2=0x7ffbcc103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.060+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 shutdown_connections 2024-07-26T08:38:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:38:56.060+0000 7ffbd1400640 1 -- 172.21.15.44:0/1479316409 wait complete. 2024-07-26T08:38:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:55 smithi116 ceph-mon[37097]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T08:38:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:55 smithi171 ceph-mon[37789]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 175 KiB/s wr, 53 op/s 2024-07-26T08:38:56.859 INFO:tasks.cephfs.filesystem:scrub status for tag:b8d04a66-208a-4e89-9fcd-6892679c2e07 - {'path': '/', 'tag': 'b8d04a66-208a-4e89-9fcd-6892679c2e07', 'options': 'recursive,force'} 2024-07-26T08:38:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:57 smithi044 ceph-mon[32404]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-07-26T08:38:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:57 smithi116 ceph-mon[37097]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-07-26T08:38:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:57 smithi171 ceph-mon[37789]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-07-26T08:39:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:38:59 smithi044 ceph-mon[32404]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-07-26T08:39:00.267 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:39:00 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:39:00.114+0000 7f4542a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:39:00.267 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:39:00 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:39:00.115+0000 7f4542a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:39:00.267 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:39:00 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:39:00.115+0000 7f4542a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:39:00.267 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:39:00 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:39:00.115+0000 7f4542a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:39:00.267 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Jul 26 08:39:00 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mgr-x[32642]: 2024-07-26T08:39:00.115+0000 7f4542a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T08:39:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:38:59 smithi116 ceph-mon[37097]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-07-26T08:39:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:38:59 smithi171 ceph-mon[37789]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-07-26T08:39:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:02 smithi044 ceph-mon[32404]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T08:39:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:02 smithi116 ceph-mon[37097]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T08:39:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:02 smithi171 ceph-mon[37789]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T08:39:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:03 smithi171 ceph-mon[37789]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T08:39:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:03 smithi044 ceph-mon[32404]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T08:39:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:03 smithi116 ceph-mon[37097]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T08:39:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:04 smithi171 ceph-mon[37789]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 197 KiB/s wr, 74 op/s 2024-07-26T08:39:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:04 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:39:04.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:04 smithi116 ceph-mon[37097]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 197 KiB/s wr, 74 op/s 2024-07-26T08:39:04.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:04 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:39:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:04 smithi044 ceph-mon[32404]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 197 KiB/s wr, 74 op/s 2024-07-26T08:39:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:04 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:39:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:06 smithi171 ceph-mon[37789]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-07-26T08:39:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:06 smithi044 ceph-mon[32404]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-07-26T08:39:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:06 smithi116 ceph-mon[37097]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-07-26T08:39:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:39:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:39:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:39:08.334 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:08 smithi044 ceph-mon[32404]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-07-26T08:39:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:08 smithi171 ceph-mon[37789]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-07-26T08:39:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:08 smithi116 ceph-mon[37097]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-07-26T08:39:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:10 smithi171 ceph-mon[37789]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-07-26T08:39:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:10 smithi044 ceph-mon[32404]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-07-26T08:39:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:10 smithi116 ceph-mon[37097]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-07-26T08:39:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:12 smithi171 ceph-mon[37789]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-07-26T08:39:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:12 smithi044 ceph-mon[32404]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-07-26T08:39:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:12 smithi116 ceph-mon[37097]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-07-26T08:39:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:14 smithi171 ceph-mon[37789]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-07-26T08:39:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:14 smithi044 ceph-mon[32404]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-07-26T08:39:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:14 smithi116 ceph-mon[37097]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-07-26T08:39:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:16 smithi171 ceph-mon[37789]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 9 op/s 2024-07-26T08:39:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:16 smithi044 ceph-mon[32404]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 9 op/s 2024-07-26T08:39:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:16 smithi116 ceph-mon[37097]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 9 op/s 2024-07-26T08:39:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:18 smithi171 ceph-mon[37789]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-07-26T08:39:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:18 smithi044 ceph-mon[32404]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-07-26T08:39:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:18 smithi116 ceph-mon[37097]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-07-26T08:39:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:20 smithi171 ceph-mon[37789]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-07-26T08:39:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:20 smithi044 ceph-mon[32404]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-07-26T08:39:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:20 smithi116 ceph-mon[37097]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-07-26T08:39:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:22 smithi171 ceph-mon[37789]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:39:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:22 smithi044 ceph-mon[32404]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:39:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:22 smithi116 ceph-mon[37097]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T08:39:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:24 smithi171 ceph-mon[37789]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:39:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:24 smithi044 ceph-mon[32404]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:39:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:24 smithi116 ceph-mon[37097]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:39:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:26 smithi171 ceph-mon[37789]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:26 smithi044 ceph-mon[32404]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:26 smithi116 ceph-mon[37097]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:26.860 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T08:39:27.384 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:28 smithi044 ceph-mon[32404]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:39:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:28 smithi116 ceph-mon[37097]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:39:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:28 smithi171 ceph-mon[37789]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:39:29.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 -- 172.21.15.44:0/360949189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840410b880 msgr2=0x7f84041120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 --2- 172.21.15.44:0/360949189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840410b880 0x7f84041120b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8400009be0 tx=0x7f840002a6c0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 -- 172.21.15.44:0/360949189 shutdown_connections 2024-07-26T08:39:29.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 --2- 172.21.15.44:0/360949189 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f840410b880 0x7f84041120b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 --2- 172.21.15.44:0/360949189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840410af30 0x7f840410b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 --2- 172.21.15.44:0/360949189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f84041055f0 0x7f8404105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.698+0000 7f840b200640 1 -- 172.21.15.44:0/360949189 >> 172.21.15.44:0/360949189 conn(0x7f8404101060 msgr2=0x7f84041034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:29.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.699+0000 7f840b200640 1 -- 172.21.15.44:0/360949189 shutdown_connections 2024-07-26T08:39:29.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.699+0000 7f840b200640 1 -- 172.21.15.44:0/360949189 wait complete. 2024-07-26T08:39:29.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.701+0000 7f840b200640 1 Processor -- start 2024-07-26T08:39:29.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.701+0000 7f840b200640 1 -- start start 2024-07-26T08:39:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f840b200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 0x7f840419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f840b200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840410af30 0x7f840419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f840b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840410b880 0x7f84041a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f840b200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8404115530 con 0x7f84041055f0 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f840b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84041153b0 con 0x7f840410b880 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f840b200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f84041156b0 con 0x7f840410af30 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f8409e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 0x7f840419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f8409e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 0x7f840419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53188/0 (socket says 172.21.15.44:53188) 2024-07-26T08:39:29.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f8409e00640 1 -- 172.21.15.44:0/1442908086 learned_addr learned my addr 172.21.15.44:0/1442908086 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.702+0000 7f8409400640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840410af30 0x7f840419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.703+0000 7f840a800640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840410b880 0x7f84041a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.703+0000 7f8409e00640 1 -- 172.21.15.44:0/1442908086 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840410af30 msgr2=0x7f840419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.703+0000 7f8409e00640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840410af30 0x7f840419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.703+0000 7f8409e00640 1 -- 172.21.15.44:0/1442908086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840410b880 msgr2=0x7f84041a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.703+0000 7f8409e00640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840410b880 0x7f84041a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.703+0000 7f8409e00640 1 -- 172.21.15.44:0/1442908086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84041a1e10 con 0x7f84041055f0 2024-07-26T08:39:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.704+0000 7f8409e00640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 0x7f840419bea0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f83f800d950 tx=0x7f83f800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.705+0000 7f83fea00640 1 -- 172.21.15.44:0/1442908086 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8017030 con 0x7f84041055f0 2024-07-26T08:39:29.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.705+0000 7f83fea00640 1 -- 172.21.15.44:0/1442908086 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f8004020 con 0x7f84041055f0 2024-07-26T08:39:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.705+0000 7f83fea00640 1 -- 172.21.15.44:0/1442908086 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8005560 con 0x7f84041055f0 2024-07-26T08:39:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.705+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84041a2100 con 0x7f84041055f0 2024-07-26T08:39:29.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.705+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84041a8860 con 0x7f84041055f0 2024-07-26T08:39:29.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.709+0000 7f8412ffd740 1 Processor -- start 2024-07-26T08:39:29.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.710+0000 7f8412ffd740 1 -- start start 2024-07-26T08:39:29.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.710+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fd8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.710+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fd8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f83fea00640 1 -- 172.21.15.44:0/1442908086 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f83f80047a0 con 0x7f84041055f0 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8412ffd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ca98f4ebc0 con 0x55ca98c33070 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8412ffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca98f3e0c0 con 0x55ca98c0a1b0 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8412ffd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ca98f42100 con 0x55ca98c12390 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8409e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fd8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f840a800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fd8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8409e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fd8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39304/0 (socket says 172.21.15.44:39304) 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.711+0000 7f8409e00640 1 -- 172.21.15.44:0/2752561196 learned_addr learned my addr 172.21.15.44:0/2752561196 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.712+0000 7f8409400640 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.712+0000 7f840a800640 1 -- 172.21.15.44:0/2752561196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 msgr2=0x55ca98fd8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.712+0000 7f840a800640 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.712+0000 7f840a800640 1 -- 172.21.15.44:0/2752561196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 msgr2=0x55ca98fd8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.712+0000 7f840a800640 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fd8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.712+0000 7f840a800640 1 -- 172.21.15.44:0/2752561196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca98c9e650 con 0x55ca98c33070 2024-07-26T08:39:29.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.713+0000 7f840a800640 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fd8000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8400009b30 tx=0x7f8400030430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2712ccd5ea41633d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:29.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.713+0000 7f83f3000640 1 -- 172.21.15.44:0/2752561196 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400040030 con 0x55ca98c33070 2024-07-26T08:39:29.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.713+0000 7f83fea00640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83ec0b3e90 0x7f83ec0b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.714+0000 7f83fea00640 1 -- 172.21.15.44:0/1442908086 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f83f80e3cd0 con 0x7f84041055f0 2024-07-26T08:39:29.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.714+0000 7f83f3000640 1 -- 172.21.15.44:0/2752561196 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400003e20 con 0x55ca98c33070 2024-07-26T08:39:29.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.714+0000 7f83f3000640 1 -- 172.21.15.44:0/2752561196 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400036040 con 0x55ca98c33070 2024-07-26T08:39:29.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.714+0000 7f8409e00640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83ec0b3e90 0x7f83ec0b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 -- 172.21.15.44:0/2752561196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 msgr2=0x55ca98fd8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8409e00640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83ec0b3e90 0x7f83ec0b63a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83f80a7760 tx=0x7f83f80e7780 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:29.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fd8000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8400009b30 tx=0x7f8400030430 comp rx=0 tx=0).stop 2024-07-26T08:39:29.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 -- 172.21.15.44:0/2752561196 shutdown_connections 2024-07-26T08:39:29.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fd8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2752561196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fd8000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.715+0000 7f8412ffd740 1 -- 172.21.15.44:0/2752561196 >> 172.21.15.44:0/2752561196 conn(0x55ca98c27460 msgr2=0x55ca98fd5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:29.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.716+0000 7f8412ffd740 1 -- 172.21.15.44:0/2752561196 shutdown_connections 2024-07-26T08:39:29.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.716+0000 7f8412ffd740 1 -- 172.21.15.44:0/2752561196 wait complete. 2024-07-26T08:39:29.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.717+0000 7f8412ffd740 1 -- start start 2024-07-26T08:39:29.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.717+0000 7f8412ffd740 1 Processor -- start 2024-07-26T08:39:29.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fe3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fe4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ca98da1700 con 0x55ca98c33070 2024-07-26T08:39:29.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca98eda690 con 0x55ca98c0a1b0 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ca98fe6780 con 0x55ca98c12390 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8412ffd740 10 client.0 _ll_register_callbacks cb 0x55ca98eea140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8409400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fe3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8409400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fe3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46952/0 (socket says 172.21.15.44:46952) 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f8409400640 1 -- 172.21.15.44:0/1785619933 learned_addr learned my addr 172.21.15.44:0/1785619933 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.718+0000 7f840a800640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fe4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 -- 172.21.15.44:0/1785619933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 msgr2=0x55ca98fe3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fe3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 -- 172.21.15.44:0/1785619933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 msgr2=0x55ca98fe4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fe4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 -- 172.21.15.44:0/1785619933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca98ef1830 con 0x55ca98c0a1b0 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f840a800640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe35a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8400006010 tx=0x7f8400005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad3f5fdd02628bc8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:29.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.719+0000 7f83e7400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:39:29.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.720+0000 7f83e7400640 1 -- 172.21.15.44:0/1785619933 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400036050 con 0x55ca98c0a1b0 2024-07-26T08:39:29.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.720+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8400003e20 con 0x55ca98c0a1b0 2024-07-26T08:39:29.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.720+0000 7f83e7400640 1 -- 172.21.15.44:0/1785619933 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8400030970 con 0x55ca98c0a1b0 2024-07-26T08:39:29.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.720+0000 7f83e7400640 1 -- 172.21.15.44:0/1785619933 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8400058050 con 0x55ca98c0a1b0 2024-07-26T08:39:29.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 10 client.25547 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:29.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca98ff2310 con 0x55ca98c0a1b0 2024-07-26T08:39:29.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f83e7400640 1 -- 172.21.15.44:0/1785619933 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400030b70 con 0x55ca98c0a1b0 2024-07-26T08:39:29.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f83e7400640 1 -- 172.21.15.44:0/1785619933 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8400004430 con 0x55ca98c0a1b0 2024-07-26T08:39:29.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 10 client.25547 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:29.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 10 client.25547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:39:29.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 20 client.25547 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:29.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bf19f0 0x55ca98ff5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 4 client.25547 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:39:29.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.721+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8400003e20 con 0x55ca98bf19f0 2024-07-26T08:39:29.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.722+0000 7f8409400640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bf19f0 0x55ca98ff5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.723+0000 7f8409400640 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bf19f0 0x55ca98ff5670 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bef293ccd3c37150 server_cookie=ba9823554a11c3ce in_seq=0 out_seq=0 2024-07-26T08:39:29.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.723+0000 7f83e7400640 10 client.25547 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:29.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.723+0000 7f83e7400640 1 -- 172.21.15.44:0/1785619933 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f8400003e20 con 0x55ca98bf19f0 2024-07-26T08:39:29.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.723+0000 7f83e7400640 10 client.25547 handle_command_reply: tid=0 2024-07-26T08:39:29.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.723+0000 7f8412ffd740 1 client.25547 shutdown 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 msgr2=0x55ca98fe35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe35a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8400006010 tx=0x7f8400005bd0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 shutdown_connections 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bf19f0 0x55ca98ff5670 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fe3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fe4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1785619933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe35a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 >> 172.21.15.44:0/1785619933 conn(0x55ca98c27460 msgr2=0x55ca98fdb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f83e7400640 0 client.25547 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.724+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 shutdown_connections 2024-07-26T08:39:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.725+0000 7f8412ffd740 1 -- 172.21.15.44:0/1785619933 wait complete. 2024-07-26T08:39:29.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.725+0000 7f8412ffd740 20 client.25547 trim_cache size 0 max 16384 2024-07-26T08:39:29.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.743+0000 7f8412ffd740 1 Processor -- start 2024-07-26T08:39:29.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.743+0000 7f8412ffd740 1 -- start start 2024-07-26T08:39:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8412ffd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84041b2930 con 0x55ca98c33070 2024-07-26T08:39:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8412ffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84041b4470 con 0x55ca98c0a1b0 2024-07-26T08:39:29.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8412ffd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f84041b3000 con 0x55ca98c12390 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8409400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8409400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46968/0 (socket says 172.21.15.44:46968) 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.744+0000 7f8409400640 1 -- 172.21.15.44:0/1032252668 learned_addr learned my addr 172.21.15.44:0/1032252668 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f840a800640 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f840a800640 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f8409400640 1 -- 172.21.15.44:0/1032252668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 msgr2=0x55ca98fe3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f8409400640 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f8409400640 1 -- 172.21.15.44:0/1032252668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 msgr2=0x55ca98fef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f8409400640 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f8409400640 1 -- 172.21.15.44:0/1032252668 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca98fe4860 con 0x55ca98c12390 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.745+0000 7f8409400640 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd4530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f83f4000c00 tx=0x7f83f4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c18bf590913a243 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.746+0000 7f83f2600640 1 -- 172.21.15.44:0/1032252668 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4004310 con 0x55ca98c12390 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.746+0000 7f83f2600640 1 -- 172.21.15.44:0/1032252668 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f40044b0 con 0x55ca98c12390 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.746+0000 7f83f2600640 1 -- 172.21.15.44:0/1032252668 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f400f950 con 0x55ca98c12390 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 -- 172.21.15.44:0/1032252668 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 msgr2=0x55ca98fd4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd4530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f83f4000c00 tx=0x7f83f4009260 comp rx=0 tx=0).stop 2024-07-26T08:39:29.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 -- 172.21.15.44:0/1032252668 shutdown_connections 2024-07-26T08:39:29.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c12390 0x55ca98fd4530 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c33070 0x55ca98fef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 --2- 172.21.15.44:0/1032252668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c0a1b0 0x55ca98fe3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 -- 172.21.15.44:0/1032252668 >> 172.21.15.44:0/1032252668 conn(0x55ca98c27460 msgr2=0x55ca98fee9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.747+0000 7f8412ffd740 1 -- 172.21.15.44:0/1032252668 shutdown_connections 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.748+0000 7f8412ffd740 1 -- 172.21.15.44:0/1032252668 wait complete. 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.748+0000 7f8412ffd740 1 -- start start 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.748+0000 7f8412ffd740 1 Processor -- start 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 0x55ca98ff8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c12390 0x55ca98ff8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c33070 0x55ca99052e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84041b3da0 con 0x55ca98c12390 2024-07-26T08:39:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84041b36d0 con 0x55ca98c33070 2024-07-26T08:39:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f84041b0910 con 0x55ca98c0a1b0 2024-07-26T08:39:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.749+0000 7f8412ffd740 10 client.0 _ll_register_callbacks cb 0x7f8400009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 0x55ca98ff8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 0x55ca98ff8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46972/0 (socket says 172.21.15.44:46972) 2024-07-26T08:39:29.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 -- 172.21.15.44:0/2603497840 learned_addr learned my addr 172.21.15.44:0/2603497840 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:29.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c33070 0x55ca99052e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f8409400640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c12390 0x55ca98ff8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 -- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c33070 msgr2=0x55ca99052e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c33070 0x55ca99052e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 -- 172.21.15.44:0/2603497840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c12390 msgr2=0x55ca98ff8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c12390 0x55ca98ff8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.750+0000 7f840a800640 1 -- 172.21.15.44:0/2603497840 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca990534d0 con 0x55ca98c0a1b0 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.751+0000 7f840a800640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 0x55ca98ff8500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8400031440 tx=0x7f8400047040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8732190db78035e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.751+0000 7f83e7400640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.751+0000 7f83e7400640 1 -- 172.21.15.44:0/2603497840 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400036070 con 0x55ca98c0a1b0 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.751+0000 7f83e7400640 1 -- 172.21.15.44:0/2603497840 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8400030640 con 0x55ca98c0a1b0 2024-07-26T08:39:29.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.751+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83f40044b0 con 0x55ca98c0a1b0 2024-07-26T08:39:29.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.752+0000 7f83e7400640 1 -- 172.21.15.44:0/2603497840 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f840006b050 con 0x55ca98c0a1b0 2024-07-26T08:39:29.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.752+0000 7f8412ffd740 10 client.25549 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:29.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.752+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca9905db10 con 0x55ca98c0a1b0 2024-07-26T08:39:29.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.752+0000 7f83e7400640 1 -- 172.21.15.44:0/2603497840 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f40044b0 con 0x55ca98c0a1b0 2024-07-26T08:39:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f83e7400640 1 -- 172.21.15.44:0/2603497840 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f8400066030 con 0x55ca98c0a1b0 2024-07-26T08:39:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f8412ffd740 10 client.25549 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f8412ffd740 10 client.25549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:39:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f8412ffd740 20 client.25549 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bc6da0 0x55ca99060240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:29.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f8412ffd740 4 client.25549 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T08:39:29.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.753+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca98f50b10 con 0x55ca98bc6da0 2024-07-26T08:39:29.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.754+0000 7f8409400640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bc6da0 0x55ca99060240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.754+0000 7f8409400640 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bc6da0 0x55ca99060240 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f9a79982ee29ed7 server_cookie=84ae74d8a03ae3b9 in_seq=0 out_seq=0 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.754+0000 7f83e7400640 10 client.25549 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f83e7400640 1 -- 172.21.15.44:0/2603497840 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ca98f50b10 con 0x55ca98bc6da0 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f83e7400640 10 client.25549 handle_command_reply: tid=0 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 client.25549 shutdown 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 msgr2=0x55ca98ff8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 0x55ca98ff8500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8400031440 tx=0x7f8400047040 comp rx=0 tx=0).stop 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 shutdown_connections 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55ca98bc6da0 0x55ca99060240 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca98c33070 0x55ca99052e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ca98c12390 0x55ca98ff8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 --2- 172.21.15.44:0/2603497840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ca98c0a1b0 0x55ca98ff8500 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 >> 172.21.15.44:0/2603497840 conn(0x55ca98c27460 msgr2=0x55ca98fef390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.755+0000 7f83e7400640 0 client.25549 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.756+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 shutdown_connections 2024-07-26T08:39:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.756+0000 7f8412ffd740 1 -- 172.21.15.44:0/2603497840 wait complete. 2024-07-26T08:39:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.756+0000 7f8412ffd740 20 client.25549 trim_cache size 0 max 16384 2024-07-26T08:39:29.759 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:39:29.759 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-07-26T08:39:29.759 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-07-26T08:39:29.759 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.757+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83ec0b3e90 msgr2=0x7f83ec0b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.757+0000 7f840b200640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83ec0b3e90 0x7f83ec0b63a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83f80a7760 tx=0x7f83f80e7780 comp rx=0 tx=0).stop 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.757+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 msgr2=0x7f840419bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.757+0000 7f840b200640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 0x7f840419bea0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f83f800d950 tx=0x7f83f800dd50 comp rx=0 tx=0).stop 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 shutdown_connections 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f83ec0b3e90 0x7f83ec0b63a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840410af30 0x7f840419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f840410b880 0x7f84041a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 --2- 172.21.15.44:0/1442908086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84041055f0 0x7f840419bea0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 >> 172.21.15.44:0/1442908086 conn(0x7f8404101060 msgr2=0x7f8404103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:29.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.758+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 shutdown_connections 2024-07-26T08:39:29.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:29.759+0000 7f840b200640 1 -- 172.21.15.44:0/1442908086 wait complete. 2024-07-26T08:39:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:30 smithi044 ceph-mon[32404]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T08:39:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:30 smithi116 ceph-mon[37097]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T08:39:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:30 smithi171 ceph-mon[37789]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T08:39:30.740 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T08:39:30.741 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:39:31.274 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:32.452 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:32 smithi044 ceph-mon[32404]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:39:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:32 smithi116 ceph-mon[37097]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:39:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:32 smithi171 ceph-mon[37789]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:39:33.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.744+0000 7f6625e00640 1 -- 172.21.15.44:0/72435372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662010af30 msgr2=0x7f662010b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:33.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.744+0000 7f6625e00640 1 --2- 172.21.15.44:0/72435372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662010af30 0x7f662010b340 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f660c009860 tx=0x7f660c02a340 comp rx=0 tx=0).stop 2024-07-26T08:39:33.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 -- 172.21.15.44:0/72435372 shutdown_connections 2024-07-26T08:39:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 --2- 172.21.15.44:0/72435372 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010b880 0x7f66201120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 --2- 172.21.15.44:0/72435372 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f662010af30 0x7f662010b340 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 --2- 172.21.15.44:0/72435372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66201055f0 0x7f6620105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 -- 172.21.15.44:0/72435372 >> 172.21.15.44:0/72435372 conn(0x7f6620101060 msgr2=0x7f66201034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 -- 172.21.15.44:0/72435372 shutdown_connections 2024-07-26T08:39:33.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.745+0000 7f6625e00640 1 -- 172.21.15.44:0/72435372 wait complete. 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.746+0000 7f6625e00640 1 Processor -- start 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.746+0000 7f6625e00640 1 -- start start 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66201055f0 0x7f662019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 0x7f662019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f662010b880 0x7f66201a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6620115510 con 0x7f66201055f0 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6620115390 con 0x7f662010b880 2024-07-26T08:39:33.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6620115690 con 0x7f662010af30 2024-07-26T08:39:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f661fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 0x7f662019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f661fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 0x7f662019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:49714/0 (socket says 172.21.15.44:49714) 2024-07-26T08:39:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f661fe00640 1 -- 172.21.15.44:0/187188519 learned_addr learned my addr 172.21.15.44:0/187188519 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.747+0000 7f6625400640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f662010b880 0x7f66201a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:33.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f6624a00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66201055f0 0x7f662019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f661fe00640 1 -- 172.21.15.44:0/187188519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f662010b880 msgr2=0x7f66201a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f661fe00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f662010b880 0x7f66201a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f661fe00640 1 -- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66201055f0 msgr2=0x7f662019bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f661fe00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66201055f0 0x7f662019bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f661fe00640 1 -- 172.21.15.44:0/187188519 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66201a1e90 con 0x7f662010af30 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.748+0000 7f6624a00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66201055f0 0x7f662019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.749+0000 7f661fe00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 0x7f662019c4c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f660c035040 tx=0x7f660c032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.749+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c040020 con 0x7f662010af30 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.749+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f660c030e10 con 0x7f662010af30 2024-07-26T08:39:33.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.749+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c037040 con 0x7f662010af30 2024-07-26T08:39:33.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.749+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66201a2120 con 0x7f662010af30 2024-07-26T08:39:33.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.749+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66201a88e0 con 0x7f662010af30 2024-07-26T08:39:33.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.751+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6620105a00 con 0x7f662010af30 2024-07-26T08:39:33.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.754+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f660c043020 con 0x7f662010af30 2024-07-26T08:39:33.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.758+0000 7f661d600640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66080b3dc0 0x7f66080b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:33.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.758+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f660c1046e0 con 0x7f662010af30 2024-07-26T08:39:33.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.759+0000 7f6624a00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66080b3dc0 0x7f66080b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:33.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.759+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f660c04a020 con 0x7f662010af30 2024-07-26T08:39:33.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:33.760+0000 7f6624a00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66080b3dc0 0x7f66080b62d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6614005fd0 tx=0x7f6614009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:34.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.022+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f662010d660 con 0x7f662010af30 2024-07-26T08:39:34.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.023+0000 7f661d600640 1 -- 172.21.15.44:0/187188519 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7f660c0cb820 con 0x7f662010af30 2024-07-26T08:39:34.025 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:39:34.026 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":30,"btime":"2024-07-26T08:38:32:520674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:38:32.022353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24701,24686,24679,24668,24661]},"id":1}]} 2024-07-26T08:39:34.026 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 30 2024-07-26T08:39:34.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66080b3dc0 msgr2=0x7f66080b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66080b3dc0 0x7f66080b62d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6614005fd0 tx=0x7f6614009260 comp rx=0 tx=0).stop 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 msgr2=0x7f662019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 0x7f662019c4c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f660c035040 tx=0x7f660c032d50 comp rx=0 tx=0).stop 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 shutdown_connections 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f66080b3dc0 0x7f66080b62d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f662010af30 0x7f662019c4c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f662010b880 0x7f66201a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 --2- 172.21.15.44:0/187188519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66201055f0 0x7f662019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.026+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 >> 172.21.15.44:0/187188519 conn(0x7f6620101060 msgr2=0x7f6620103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.027+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 shutdown_connections 2024-07-26T08:39:34.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:34.027+0000 7f6625e00640 1 -- 172.21.15.44:0/187188519 wait complete. 2024-07-26T08:39:34.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:34 smithi044 ceph-mon[32404]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T08:39:34.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:34 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/187188519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:39:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:34 smithi116 ceph-mon[37097]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T08:39:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:34 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/187188519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:39:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:34 smithi171 ceph-mon[37789]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T08:39:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:34 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/187188519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:39:34.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T08:39:35.332 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:36 smithi044 ceph-mon[32404]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:39:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:36 smithi116 ceph-mon[37097]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:39:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:36 smithi171 ceph-mon[37789]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T08:39:37.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 -- 172.21.15.44:0/2313965264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f81033e0 msgr2=0x7f60f81037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 --2- 172.21.15.44:0/2313965264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f81033e0 0x7f60f81037f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009860 tx=0x7f60e802a340 comp rx=0 tx=0).stop 2024-07-26T08:39:37.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 -- 172.21.15.44:0/2313965264 shutdown_connections 2024-07-26T08:39:37.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 --2- 172.21.15.44:0/2313965264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f8109670 0x7f60f810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 --2- 172.21.15.44:0/2313965264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f8108d20 0x7f60f8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 --2- 172.21.15.44:0/2313965264 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f81033e0 0x7f60f81037f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.558+0000 7f60fd400640 1 -- 172.21.15.44:0/2313965264 >> 172.21.15.44:0/2313965264 conn(0x7f60f80fee30 msgr2=0x7f60f8101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:37.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.559+0000 7f60fd400640 1 -- 172.21.15.44:0/2313965264 shutdown_connections 2024-07-26T08:39:37.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.559+0000 7f60fd400640 1 -- 172.21.15.44:0/2313965264 wait complete. 2024-07-26T08:39:37.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.560+0000 7f60fd400640 1 Processor -- start 2024-07-26T08:39:37.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 -- start start 2024-07-26T08:39:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 0x7f60f8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f8108d20 0x7f60f819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f8109670 0x7f60f819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8113320 con 0x7f60f8108d20 2024-07-26T08:39:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81131a0 con 0x7f60f81033e0 2024-07-26T08:39:37.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.561+0000 7f60fd400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81134a0 con 0x7f60f8109670 2024-07-26T08:39:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.562+0000 7f60f7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 0x7f60f8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.562+0000 7f60f7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 0x7f60f8199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:46946/0 (socket says 172.21.15.44:46946) 2024-07-26T08:39:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.562+0000 7f60f7e00640 1 -- 172.21.15.44:0/3405950214 learned_addr learned my addr 172.21.15.44:0/3405950214 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.562+0000 7f60f7400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f8108d20 0x7f60f819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.562+0000 7f60fca00640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f8109670 0x7f60f819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7400640 1 -- 172.21.15.44:0/3405950214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f8109670 msgr2=0x7f60f819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f8109670 0x7f60f819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7400640 1 -- 172.21.15.44:0/3405950214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 msgr2=0x7f60f8199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 0x7f60f8199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7400640 1 -- 172.21.15.44:0/3405950214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f819fc80 con 0x7f60f8108d20 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7e00640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 0x7f60f8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:37.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.563+0000 7f60f7400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f8108d20 0x7f60f819a2b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f60f806b360 tx=0x7f60e400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.564+0000 7f60f4c00640 1 -- 172.21.15.44:0/3405950214 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e401e070 con 0x7f60f8108d20 2024-07-26T08:39:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.564+0000 7f60f4c00640 1 -- 172.21.15.44:0/3405950214 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60e40198c0 con 0x7f60f8108d20 2024-07-26T08:39:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.564+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f819ff70 con 0x7f60f8108d20 2024-07-26T08:39:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.564+0000 7f60f4c00640 1 -- 172.21.15.44:0/3405950214 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e4018730 con 0x7f60f8108d20 2024-07-26T08:39:37.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.564+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f8075020 con 0x7f60f8108d20 2024-07-26T08:39:37.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.570+0000 7f610564f740 1 Processor -- start 2024-07-26T08:39:37.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.570+0000 7f60f4c00640 1 -- 172.21.15.44:0/3405950214 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f60e4021070 con 0x7f60f8108d20 2024-07-26T08:39:37.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.570+0000 7f610564f740 1 -- start start 2024-07-26T08:39:37.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.571+0000 7f610564f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4936010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.571+0000 7f610564f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f4936550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f610564f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4570390 0x5565f4936a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f610564f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5565f48acbd0 con 0x5565f4591070 2024-07-26T08:39:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f610564f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5565f489c0d0 con 0x5565f4570390 2024-07-26T08:39:37.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f610564f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5565f48a0110 con 0x5565f45681b0 2024-07-26T08:39:37.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f60fca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4936010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f60fca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4936010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50038/0 (socket says 172.21.15.44:50038) 2024-07-26T08:39:37.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f60fca00640 1 -- 172.21.15.44:0/3557542626 learned_addr learned my addr 172.21.15.44:0/3557542626 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:37.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.572+0000 7f60f4c00640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f60dc0b3dc0 0x7f60dc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60f4c00640 1 -- 172.21.15.44:0/3405950214 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f60e40e7a50 con 0x7f60f8108d20 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60f7e00640 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f4936550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60f7400640 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4570390 0x5565f4936a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60fca00640 1 -- 172.21.15.44:0/3557542626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 msgr2=0x5565f4936550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60fca00640 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f4936550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60fca00640 1 -- 172.21.15.44:0/3557542626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4570390 msgr2=0x5565f4936a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60fca00640 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4570390 0x5565f4936a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60fca00640 1 -- 172.21.15.44:0/3557542626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f45fc650 con 0x5565f4591070 2024-07-26T08:39:37.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.573+0000 7f60f7400640 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4570390 0x5565f4936a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:39:37.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.574+0000 7f60fca00640 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4936010 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f60f000e580 tx=0x7f60f00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1636ae5c673adb2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:37.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.574+0000 7f60fca00640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f60dc0b3dc0 0x7f60dc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.575+0000 7f60ed200640 1 -- 172.21.15.44:0/3557542626 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f000dad0 con 0x5565f4591070 2024-07-26T08:39:37.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.575+0000 7f60ed200640 1 -- 172.21.15.44:0/3557542626 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f000dc70 con 0x5565f4591070 2024-07-26T08:39:37.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.575+0000 7f60ed200640 1 -- 172.21.15.44:0/3557542626 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f00167b0 con 0x5565f4591070 2024-07-26T08:39:37.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 -- 172.21.15.44:0/3557542626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 msgr2=0x5565f4936010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4936010 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f60f000e580 tx=0x7f60f00095c0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f60fca00640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f60dc0b3dc0 0x7f60dc0b62d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60f001c000 tx=0x7f60f0016a30 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 -- 172.21.15.44:0/3557542626 shutdown_connections 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f4936550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4570390 0x5565f4936a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 --2- 172.21.15.44:0/3557542626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4936010 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.576+0000 7f610564f740 1 -- 172.21.15.44:0/3557542626 >> 172.21.15.44:0/3557542626 conn(0x5565f4585460 msgr2=0x5565f4933f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.577+0000 7f610564f740 1 -- 172.21.15.44:0/3557542626 shutdown_connections 2024-07-26T08:39:37.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.577+0000 7f610564f740 1 -- 172.21.15.44:0/3557542626 wait complete. 2024-07-26T08:39:37.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.577+0000 7f610564f740 1 -- start start 2024-07-26T08:39:37.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.578+0000 7f610564f740 1 Processor -- start 2024-07-26T08:39:37.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f45681b0 0x5565f4949db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f494a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f494a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5565f46ff700 con 0x5565f45681b0 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5565f48386a0 con 0x5565f4591070 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5565f494cf20 con 0x5565f4570390 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f610564f740 10 client.0 _ll_register_callbacks cb 0x5565f4848150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f60fca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f494a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f60fca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f494a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:46964/0 (socket says 172.21.15.44:46964) 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f60fca00640 1 -- 172.21.15.44:0/2306463432 learned_addr learned my addr 172.21.15.44:0/2306463432 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:37.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60f7400640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f494a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.579+0000 7f60f7e00640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f45681b0 0x5565f4949db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60fca00640 1 -- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 msgr2=0x5565f494a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60fca00640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f494a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60fca00640 1 -- 172.21.15.44:0/2306463432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f45681b0 msgr2=0x5565f4949db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60fca00640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f45681b0 0x5565f4949db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60fca00640 1 -- 172.21.15.44:0/2306463432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f484f840 con 0x5565f4591070 2024-07-26T08:39:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.580+0000 7f60f7400640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f494a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:39:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.581+0000 7f60fca00640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f494a830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f60f000ef40 tx=0x7f60f001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0f0dcb3a6b9c62 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.581+0000 7f60e1600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:39:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.581+0000 7f60e1600640 1 -- 172.21.15.44:0/2306463432 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0034070 con 0x5565f4591070 2024-07-26T08:39:37.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.581+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f000dc70 con 0x5565f4591070 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.581+0000 7f60e1600640 1 -- 172.21.15.44:0/2306463432 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f60f00166b0 con 0x5565f4591070 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.582+0000 7f60e1600640 1 -- 172.21.15.44:0/2306463432 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f60f0040050 con 0x5565f4591070 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.582+0000 7f610564f740 10 client.25568 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.582+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5565f4958ab0 con 0x5565f4591070 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.582+0000 7f60e1600640 1 -- 172.21.15.44:0/2306463432 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f000dc70 con 0x5565f4591070 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.582+0000 7f60e1600640 1 -- 172.21.15.44:0/2306463432 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f60f000b040 con 0x5565f4591070 2024-07-26T08:39:37.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.582+0000 7f610564f740 10 client.25568 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:37.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.583+0000 7f610564f740 10 client.25568 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-07-26T08:39:37.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.583+0000 7f610564f740 20 client.25568 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:37.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.583+0000 7f610564f740 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f454f9f0 0x5565f495be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.583+0000 7f610564f740 4 client.25568 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:39:37.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.583+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565f48aeb20 con 0x5565f454f9f0 2024-07-26T08:39:37.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.583+0000 7f60f7e00640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f454f9f0 0x5565f495be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.584+0000 7f60f7e00640 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f454f9f0 0x5565f495be70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fdf26db6f63c3285 server_cookie=514cb706c1d6d704 in_seq=0 out_seq=0 2024-07-26T08:39:37.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.584+0000 7f60e1600640 10 client.25568 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:39:37.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.585+0000 7f60e1600640 1 -- 172.21.15.44:0/2306463432 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5565f48aeb20 con 0x5565f454f9f0 2024-07-26T08:39:37.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.585+0000 7f60e1600640 10 client.25568 handle_command_reply: tid=0 2024-07-26T08:39:37.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.585+0000 7f610564f740 1 client.25568 shutdown 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 msgr2=0x5565f494a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f494a830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f60f000ef40 tx=0x7f60f001d000 comp rx=0 tx=0).stop 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 shutdown_connections 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f494a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f454f9f0 0x5565f495be70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f494a830 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 --2- 172.21.15.44:0/2306463432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f45681b0 0x5565f4949db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 >> 172.21.15.44:0/2306463432 conn(0x5565f4585460 msgr2=0x5565f4932540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f60e1600640 0 client.25568 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:39:37.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 shutdown_connections 2024-07-26T08:39:37.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.586+0000 7f610564f740 1 -- 172.21.15.44:0/2306463432 wait complete. 2024-07-26T08:39:37.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.587+0000 7f610564f740 20 client.25568 trim_cache size 0 max 16384 2024-07-26T08:39:37.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 Processor -- start 2024-07-26T08:39:37.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 -- start start 2024-07-26T08:39:37.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f45681b0 0x5565f4937a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f49558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4933550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81ae5c0 con 0x5565f4591070 2024-07-26T08:39:37.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81b0100 con 0x5565f45681b0 2024-07-26T08:39:37.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.601+0000 7f610564f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81aec90 con 0x5565f4570390 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.602+0000 7f60fca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4933550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.602+0000 7f60fca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4933550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50046/0 (socket says 172.21.15.44:50046) 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.602+0000 7f60fca00640 1 -- 172.21.15.44:0/2338497178 learned_addr learned my addr 172.21.15.44:0/2338497178 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.602+0000 7f60f7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f45681b0 0x5565f4937a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.602+0000 7f60f7400640 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f49558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.603+0000 7f60fca00640 1 -- 172.21.15.44:0/2338497178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 msgr2=0x5565f49558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.603+0000 7f60fca00640 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f49558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.603+0000 7f60fca00640 1 -- 172.21.15.44:0/2338497178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f45681b0 msgr2=0x5565f4937a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.603+0000 7f60fca00640 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f45681b0 0x5565f4937a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.603+0000 7f60fca00640 1 -- 172.21.15.44:0/2338497178 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f4934420 con 0x5565f4591070 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.603+0000 7f60fca00640 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4933550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f60f000d3b0 tx=0x7f60f0009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2439f390208e51c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.604+0000 7f60e3e00640 1 -- 172.21.15.44:0/2338497178 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f003d020 con 0x5565f4591070 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.604+0000 7f60e3e00640 1 -- 172.21.15.44:0/2338497178 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f00179e0 con 0x5565f4591070 2024-07-26T08:39:37.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.604+0000 7f60e3e00640 1 -- 172.21.15.44:0/2338497178 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0034040 con 0x5565f4591070 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 -- 172.21.15.44:0/2338497178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 msgr2=0x5565f4933550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4933550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f60f000d3b0 tx=0x7f60f0009ab0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 -- 172.21.15.44:0/2338497178 shutdown_connections 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f4570390 0x5565f49558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4591070 0x5565f4933550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 --2- 172.21.15.44:0/2338497178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f45681b0 0x5565f4937a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 -- 172.21.15.44:0/2338497178 >> 172.21.15.44:0/2338497178 conn(0x5565f4585460 msgr2=0x5565f49551b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 -- 172.21.15.44:0/2338497178 shutdown_connections 2024-07-26T08:39:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.605+0000 7f610564f740 1 -- 172.21.15.44:0/2338497178 wait complete. 2024-07-26T08:39:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.606+0000 7f610564f740 1 -- start start 2024-07-26T08:39:37.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.606+0000 7f610564f740 1 Processor -- start 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.606+0000 7f610564f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f495c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f610564f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 0x5565f495c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f610564f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f49c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f610564f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81afa30 con 0x5565f4570390 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f610564f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81af360 con 0x5565f4591070 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f610564f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f60f81ac5a0 con 0x5565f45681b0 2024-07-26T08:39:37.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f610564f740 10 client.0 _ll_register_callbacks cb 0x7f60f0023f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f60f7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 0x5565f495c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f60f7400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 0x5565f495c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50048/0 (socket says 172.21.15.44:50048) 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f60f7400640 1 -- 172.21.15.44:0/2793416955 learned_addr learned my addr 172.21.15.44:0/2793416955 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.607+0000 7f60f7e00640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f495c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60fca00640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f49c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60f7400640 1 -- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 msgr2=0x5565f495c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60f7400640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f495c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60f7400640 1 -- 172.21.15.44:0/2793416955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 msgr2=0x5565f49c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60f7400640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f49c23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60f7400640 1 -- 172.21.15.44:0/2793416955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f49c28f0 con 0x5565f4570390 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60f7400640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 0x5565f495c6f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f60e40aa400 tx=0x7f60e40adfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edea511f2d421623 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.608+0000 7f60e1600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.609+0000 7f60e1600640 1 -- 172.21.15.44:0/2793416955 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e4019620 con 0x5565f4570390 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.609+0000 7f60e1600640 1 -- 172.21.15.44:0/2793416955 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f60e400e0d0 con 0x5565f4570390 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.609+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f00179e0 con 0x5565f4570390 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.609+0000 7f60e1600640 1 -- 172.21.15.44:0/2793416955 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f60e40ad5e0 con 0x5565f4570390 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.609+0000 7f610564f740 10 client.15447 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:37.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.609+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5565f49c3a40 con 0x5565f4570390 2024-07-26T08:39:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.610+0000 7f60e1600640 1 -- 172.21.15.44:0/2793416955 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60f00179e0 con 0x5565f4570390 2024-07-26T08:39:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.610+0000 7f60e1600640 1 -- 172.21.15.44:0/2793416955 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f60e40ad0e0 con 0x5565f4570390 2024-07-26T08:39:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.610+0000 7f610564f740 10 client.15447 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.610+0000 7f610564f740 10 client.15447 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-07-26T08:39:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.610+0000 7f610564f740 20 client.15447 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:37.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.610+0000 7f610564f740 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f4524da0 0x5565f49c6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.611+0000 7f610564f740 4 client.15447 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:39:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.611+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565f48aa5c0 con 0x5565f4524da0 2024-07-26T08:39:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.611+0000 7f60f7e00640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f4524da0 0x5565f49c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.612+0000 7f60f7e00640 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f4524da0 0x5565f49c6180 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a2072d8c8264c381 server_cookie=29de655d6828d3c6 in_seq=0 out_seq=0 2024-07-26T08:39:37.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.612+0000 7f60e1600640 10 client.15447 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:39:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.612+0000 7f60e1600640 1 -- 172.21.15.44:0/2793416955 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565f48aa5c0 con 0x5565f4524da0 2024-07-26T08:39:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.612+0000 7f60e1600640 10 client.15447 handle_command_reply: tid=0 2024-07-26T08:39:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.612+0000 7f610564f740 1 client.15447 shutdown 2024-07-26T08:39:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 msgr2=0x5565f495c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 0x5565f495c6f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f60e40aa400 tx=0x7f60e40adfe0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 shutdown_connections 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x5565f4524da0 0x5565f49c6180 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565f4591070 0x5565f49c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565f4570390 0x5565f495c6f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 --2- 172.21.15.44:0/2793416955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5565f45681b0 0x5565f495c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 >> 172.21.15.44:0/2793416955 conn(0x5565f4585460 msgr2=0x5565f4937760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:37.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f60e1600640 0 client.15447 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:39:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 shutdown_connections 2024-07-26T08:39:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.613+0000 7f610564f740 1 -- 172.21.15.44:0/2793416955 wait complete. 2024-07-26T08:39:37.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.614+0000 7f610564f740 20 client.15447 trim_cache size 0 max 16384 2024-07-26T08:39:37.616 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.614+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f60dc0b3dc0 msgr2=0x7f60dc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.614+0000 7f60fd400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f60dc0b3dc0 0x7f60dc0b62d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60f001c000 tx=0x7f60f0016a30 comp rx=0 tx=0).stop 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f8108d20 msgr2=0x7f60f819a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f8108d20 0x7f60f819a2b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f60f806b360 tx=0x7f60e400b690 comp rx=0 tx=0).stop 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 shutdown_connections 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f60dc0b3dc0 0x7f60dc0b62d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f8109670 0x7f60f819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60f8108d20 0x7f60f819a2b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 --2- 172.21.15.44:0/3405950214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60f81033e0 0x7f60f8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 >> 172.21.15.44:0/3405950214 conn(0x7f60f80fee30 msgr2=0x7f60f8101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 shutdown_connections 2024-07-26T08:39:37.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:37.615+0000 7f60fd400640 1 -- 172.21.15.44:0/3405950214 wait complete. 2024-07-26T08:39:38.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:38 smithi171 ceph-mon[37789]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T08:39:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:38 smithi044 ceph-mon[32404]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T08:39:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:38 smithi116 ceph-mon[37097]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T08:39:38.574 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T08:39:39.013 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:40 smithi044 ceph-mon[32404]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T08:39:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:40 smithi116 ceph-mon[37097]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T08:39:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:40 smithi171 ceph-mon[37789]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T08:39:41.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.339+0000 7f2ddd200640 1 -- 172.21.15.44:0/1938387062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd81027e0 msgr2=0x7f2dd810f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.340+0000 7f2ddd200640 1 --2- 172.21.15.44:0/1938387062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd81027e0 0x7f2dd810f500 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009900 tx=0x7f2dc402a480 comp rx=0 tx=0).stop 2024-07-26T08:39:41.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 -- 172.21.15.44:0/1938387062 shutdown_connections 2024-07-26T08:39:41.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 --2- 172.21.15.44:0/1938387062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd810fa40 0x7f2dd8111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 --2- 172.21.15.44:0/1938387062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd81027e0 0x7f2dd810f500 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 --2- 172.21.15.44:0/1938387062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8101e90 0x7f2dd81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 -- 172.21.15.44:0/1938387062 >> 172.21.15.44:0/1938387062 conn(0x7f2dd80fdcf0 msgr2=0x7f2dd8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:41.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 -- 172.21.15.44:0/1938387062 shutdown_connections 2024-07-26T08:39:41.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.341+0000 7f2ddd200640 1 -- 172.21.15.44:0/1938387062 wait complete. 2024-07-26T08:39:41.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.343+0000 7f2ddd200640 1 Processor -- start 2024-07-26T08:39:41.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.343+0000 7f2ddd200640 1 -- start start 2024-07-26T08:39:41.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2ddd200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 0x7f2dd819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2ddd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd81027e0 0x7f2dd819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2ddd200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd810fa40 0x7f2dd81a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2ddd200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81154d0 con 0x7f2dd810fa40 2024-07-26T08:39:41.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2ddd200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8115350 con 0x7f2dd81027e0 2024-07-26T08:39:41.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2ddd200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8115650 con 0x7f2dd8101e90 2024-07-26T08:39:41.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2dd7400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 0x7f2dd819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2dd7400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 0x7f2dd819bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:49806/0 (socket says 172.21.15.44:49806) 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2dd7400640 1 -- 172.21.15.44:0/546272355 learned_addr learned my addr 172.21.15.44:0/546272355 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd810fa40 0x7f2dd81a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.344+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd81027e0 0x7f2dd819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd6a00640 1 -- 172.21.15.44:0/546272355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 msgr2=0x7f2dd819bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 0x7f2dd819bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd6a00640 1 -- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd810fa40 msgr2=0x7f2dd81a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd810fa40 0x7f2dd81a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd6a00640 1 -- 172.21.15.44:0/546272355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd81a1e70 con 0x7f2dd81027e0 2024-07-26T08:39:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd7400640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 0x7f2dd819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.345+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd810fa40 0x7f2dd81a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.346+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd81027e0 0x7f2dd819c4a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2dc402aa70 tx=0x7f2dc4003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.346+0000 7f2dcfe00640 1 -- 172.21.15.44:0/546272355 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc403f070 con 0x7f2dd81027e0 2024-07-26T08:39:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.346+0000 7f2dcfe00640 1 -- 172.21.15.44:0/546272355 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dc4031b50 con 0x7f2dd81027e0 2024-07-26T08:39:41.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.346+0000 7f2dcfe00640 1 -- 172.21.15.44:0/546272355 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4035030 con 0x7f2dd81027e0 2024-07-26T08:39:41.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.346+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd81a2100 con 0x7f2dd81027e0 2024-07-26T08:39:41.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.346+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd81a2560 con 0x7f2dd81027e0 2024-07-26T08:39:41.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.350+0000 7f2de5452740 1 Processor -- start 2024-07-26T08:39:41.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.350+0000 7f2de5452740 1 -- start start 2024-07-26T08:39:41.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2de5452740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738da010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2de5452740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738da550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2de5452740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738daa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2de5452740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dc73850bd0 con 0x55dc73535070 2024-07-26T08:39:41.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2de5452740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dc738400d0 con 0x55dc73514390 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2de5452740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55dc73844110 con 0x55dc7350c1b0 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738da010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738da010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50064/0 (socket says 172.21.15.44:50064) 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2dd7e00640 1 -- 172.21.15.44:0/2254233488 learned_addr learned my addr 172.21.15.44:0/2254233488 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.351+0000 7f2dd7400640 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738da550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dd7400640 1 -- 172.21.15.44:0/2254233488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 msgr2=0x55dc738daa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dd7400640 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738daa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dd7400640 1 -- 172.21.15.44:0/2254233488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 msgr2=0x55dc738da010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dd7400640 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738da010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dd7400640 1 -- 172.21.15.44:0/2254233488 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc735a0650 con 0x55dc7350c1b0 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dd7400640 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738da550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8010550 tx=0x7f2dc8010a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcbec8312f9b0833 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dccc00640 1 -- 172.21.15.44:0/2254233488 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800e7d0 con 0x55dc7350c1b0 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dccc00640 1 -- 172.21.15.44:0/2254233488 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dc800ee20 con 0x55dc7350c1b0 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.352+0000 7f2dccc00640 1 -- 172.21.15.44:0/2254233488 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80193d0 con 0x55dc7350c1b0 2024-07-26T08:39:41.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.353+0000 7f2dcfe00640 1 -- 172.21.15.44:0/546272355 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f2dc4042070 con 0x7f2dd81027e0 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 -- 172.21.15.44:0/2254233488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 msgr2=0x55dc738da550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738da550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8010550 tx=0x7f2dc8010a10 comp rx=0 tx=0).stop 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 -- 172.21.15.44:0/2254233488 shutdown_connections 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738da550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 --2- 172.21.15.44:0/2254233488 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738da010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 -- 172.21.15.44:0/2254233488 >> 172.21.15.44:0/2254233488 conn(0x55dc73529460 msgr2=0x55dc738d7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 -- 172.21.15.44:0/2254233488 shutdown_connections 2024-07-26T08:39:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.354+0000 7f2de5452740 1 -- 172.21.15.44:0/2254233488 wait complete. 2024-07-26T08:39:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.355+0000 7f2de5452740 1 -- start start 2024-07-26T08:39:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.355+0000 7f2dcfe00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2dbc0b3da0 0x7f2dbc0b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.356+0000 7f2de5452740 1 Processor -- start 2024-07-26T08:39:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.356+0000 7f2dcfe00640 1 -- 172.21.15.44:0/546272355 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2dc41048b0 con 0x7f2dd81027e0 2024-07-26T08:39:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.356+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2dbc0b3da0 0x7f2dbc0b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc7350c1b0 0x55dc738e5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc73535070 0x55dc738e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dc736a3700 con 0x55dc7350c1b0 2024-07-26T08:39:41.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dc737dc6a0 con 0x55dc73514390 2024-07-26T08:39:41.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55dc738e87f0 con 0x55dc73535070 2024-07-26T08:39:41.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2de5452740 10 client.0 _ll_register_callbacks cb 0x55dc737ec150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.357+0000 7f2dd6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738e5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47024/0 (socket says 172.21.15.44:47024) 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 -- 172.21.15.44:0/3623137541 learned_addr learned my addr 172.21.15.44:0/3623137541 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc7350c1b0 0x55dc738e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2dbc0b3da0 0x7f2dbc0b62b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00128c0 tx=0x7f2dd0009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc73535070 0x55dc738e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 -- 172.21.15.44:0/3623137541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc73535070 msgr2=0x55dc738e60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc73535070 0x55dc738e60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 -- 172.21.15.44:0/3623137541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc7350c1b0 msgr2=0x55dc738e5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc7350c1b0 0x55dc738e5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.358+0000 7f2dd6a00640 1 -- 172.21.15.44:0/3623137541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc737f3840 con 0x55dc73514390 2024-07-26T08:39:41.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.359+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc7350c1b0 0x55dc738e5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.359+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738e5b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2dc402a9a0 tx=0x7f2dc40caae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f9440015dc5a4f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.359+0000 7f2dc1600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.359+0000 7f2dc1600640 1 -- 172.21.15.44:0/3623137541 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40cb7d0 con 0x55dc73514390 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.359+0000 7f2dc1600640 1 -- 172.21.15.44:0/3623137541 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2dc40c7360 con 0x55dc73514390 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.359+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dc800ee20 con 0x55dc73514390 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.360+0000 7f2dc1600640 1 -- 172.21.15.44:0/3623137541 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2dc40cb0d0 con 0x55dc73514390 2024-07-26T08:39:41.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.360+0000 7f2de5452740 10 client.25580 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:41.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.360+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc738f44b0 con 0x55dc73514390 2024-07-26T08:39:41.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.360+0000 7f2dc1600640 1 -- 172.21.15.44:0/3623137541 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dc800ee20 con 0x55dc73514390 2024-07-26T08:39:41.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2dc1600640 1 -- 172.21.15.44:0/3623137541 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2dc40d3cf0 con 0x55dc73514390 2024-07-26T08:39:41.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2de5452740 10 client.25580 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:41.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2de5452740 10 client.25580 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-07-26T08:39:41.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2de5452740 20 client.25580 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2de5452740 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734f39f0 0x55dc738f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2de5452740 4 client.25580 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.361+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc73852b20 con 0x55dc734f39f0 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.362+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734f39f0 0x55dc738f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.362+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734f39f0 0x55dc738f7ad0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a876ed598deffbb server_cookie=b7ddd206289ad32c in_seq=0 out_seq=0 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.363+0000 7f2dc1600640 10 client.25580 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:39:41.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.363+0000 7f2dc1600640 1 -- 172.21.15.44:0/3623137541 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc73852b20 con 0x55dc734f39f0 2024-07-26T08:39:41.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.363+0000 7f2dc1600640 10 client.25580 handle_command_reply: tid=0 2024-07-26T08:39:41.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.363+0000 7f2de5452740 1 client.25580 shutdown 2024-07-26T08:39:41.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 msgr2=0x55dc738e5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738e5b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2dc402a9a0 tx=0x7f2dc40caae0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 shutdown_connections 2024-07-26T08:39:41.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734f39f0 0x55dc738f7ad0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc73535070 0x55dc738e60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738e5b80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 --2- 172.21.15.44:0/3623137541 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc7350c1b0 0x55dc738e5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 >> 172.21.15.44:0/3623137541 conn(0x55dc73529460 msgr2=0x55dc738dd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:41.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2dc1600640 0 client.25580 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:39:41.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.364+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 shutdown_connections 2024-07-26T08:39:41.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.365+0000 7f2de5452740 1 -- 172.21.15.44:0/3623137541 wait complete. 2024-07-26T08:39:41.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.365+0000 7f2de5452740 20 client.25580 trim_cache size 0 max 16384 2024-07-26T08:39:41.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 Processor -- start 2024-07-26T08:39:41.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 -- start start 2024-07-26T08:39:41.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738d7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 0x55dc738e7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73535070 0x55dc738db750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81b2780 con 0x55dc73514390 2024-07-26T08:39:41.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81b42c0 con 0x55dc73535070 2024-07-26T08:39:41.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.379+0000 7f2de5452740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81b2e50 con 0x55dc7350c1b0 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 0x55dc738e7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd7400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738d7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 0x55dc738e7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50082/0 (socket says 172.21.15.44:50082) 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 -- 172.21.15.44:0/31251825 learned_addr learned my addr 172.21.15.44:0/31251825 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73535070 0x55dc738db750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 -- 172.21.15.44:0/31251825 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 msgr2=0x55dc738d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738d7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 -- 172.21.15.44:0/31251825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73535070 msgr2=0x55dc738db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73535070 0x55dc738db750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.380+0000 7f2dd6a00640 1 -- 172.21.15.44:0/31251825 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc738f21d0 con 0x55dc73514390 2024-07-26T08:39:41.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.381+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 0x55dc738e7520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40cb290 tx=0x7f2dc40cab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1791242f95791e0e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:41.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.381+0000 7f2dc3e00640 1 -- 172.21.15.44:0/31251825 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc40d4180 con 0x55dc73514390 2024-07-26T08:39:41.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.381+0000 7f2dc3e00640 1 -- 172.21.15.44:0/31251825 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dc40cf6d0 con 0x55dc73514390 2024-07-26T08:39:41.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.381+0000 7f2dc3e00640 1 -- 172.21.15.44:0/31251825 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc403bb70 con 0x55dc73514390 2024-07-26T08:39:41.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 -- 172.21.15.44:0/31251825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 msgr2=0x55dc738e7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 0x55dc738e7520 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40cb290 tx=0x7f2dc40cab50 comp rx=0 tx=0).stop 2024-07-26T08:39:41.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 -- 172.21.15.44:0/31251825 shutdown_connections 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73535070 0x55dc738db750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73514390 0x55dc738e7520 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 --2- 172.21.15.44:0/31251825 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738d7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 -- 172.21.15.44:0/31251825 >> 172.21.15.44:0/31251825 conn(0x55dc73529460 msgr2=0x55dc738db040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 -- 172.21.15.44:0/31251825 shutdown_connections 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.383+0000 7f2de5452740 1 -- 172.21.15.44:0/31251825 wait complete. 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.384+0000 7f2de5452740 1 -- start start 2024-07-26T08:39:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.384+0000 7f2de5452740 1 Processor -- start 2024-07-26T08:39:41.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.384+0000 7f2de5452740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738ee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.384+0000 7f2de5452740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.384+0000 7f2de5452740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738eba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2de5452740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81b3bf0 con 0x55dc73535070 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2de5452740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81b3520 con 0x55dc73514390 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2de5452740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81b0760 con 0x55dc7350c1b0 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2de5452740 10 client.0 _ll_register_callbacks cb 0x7f2dc40c7000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738eba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50086/0 (socket says 172.21.15.44:50086) 2024-07-26T08:39:41.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2dd7e00640 1 -- 172.21.15.44:0/3693313855 learned_addr learned my addr 172.21.15.44:0/3693313855 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.385+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd6a00640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd7e00640 1 -- 172.21.15.44:0/3693313855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 msgr2=0x55dc738ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd7e00640 1 -- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 msgr2=0x55dc738eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738eb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd7e00640 1 -- 172.21.15.44:0/3693313855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc738ec000 con 0x55dc73535070 2024-07-26T08:39:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dd7e00640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738eba10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000fb80 tx=0x7f2dd0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5730c48f28e4320 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:41.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.386+0000 7f2dc1600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:39:41.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.387+0000 7f2dc1600640 1 -- 172.21.15.44:0/3693313855 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd002e060 con 0x55dc73535070 2024-07-26T08:39:41.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.387+0000 7f2dc1600640 1 -- 172.21.15.44:0/3693313855 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2dd0004540 con 0x55dc73535070 2024-07-26T08:39:41.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.387+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dc40cf6d0 con 0x55dc73535070 2024-07-26T08:39:41.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.387+0000 7f2dc1600640 1 -- 172.21.15.44:0/3693313855 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2dd0035050 con 0x55dc73535070 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.387+0000 7f2de5452740 10 client.15462 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.387+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc73968860 con 0x55dc73535070 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.388+0000 7f2dc1600640 1 -- 172.21.15.44:0/3693313855 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dc40cf6d0 con 0x55dc73535070 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.388+0000 7f2dc1600640 1 -- 172.21.15.44:0/3693313855 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2dd001d410 con 0x55dc73535070 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.388+0000 7f2de5452740 10 client.15462 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.389+0000 7f2de5452740 10 client.15462 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-07-26T08:39:41.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.389+0000 7f2de5452740 20 client.15462 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:41.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.389+0000 7f2de5452740 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734c8da0 0x55dc7396b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:41.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.389+0000 7f2de5452740 4 client.15462 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:39:41.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.389+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc7384e5c0 con 0x55dc734c8da0 2024-07-26T08:39:41.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.389+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734c8da0 0x55dc7396b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:41.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.390+0000 7f2dc1600640 10 client.15462 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:39:41.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.390+0000 7f2dd7400640 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734c8da0 0x55dc7396b0d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34cf2f6549255d4f server_cookie=c495cdd4fba4d8b5 in_seq=0 out_seq=0 2024-07-26T08:39:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.390+0000 7f2dc1600640 1 -- 172.21.15.44:0/3693313855 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc7384e5c0 con 0x55dc734c8da0 2024-07-26T08:39:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.390+0000 7f2dc1600640 10 client.15462 handle_command_reply: tid=0 2024-07-26T08:39:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.390+0000 7f2de5452740 1 client.15462 shutdown 2024-07-26T08:39:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 msgr2=0x55dc738eba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738eba10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000fb80 tx=0x7f2dd0004ab0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 shutdown_connections 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x55dc734c8da0 0x55dc7396b0d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dc73535070 0x55dc738eba10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc73514390 0x55dc738eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 --2- 172.21.15.44:0/3693313855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55dc7350c1b0 0x55dc738ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 >> 172.21.15.44:0/3693313855 conn(0x55dc73529460 msgr2=0x55dc738db990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2dc1600640 0 client.15462 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 shutdown_connections 2024-07-26T08:39:41.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 1 -- 172.21.15.44:0/3693313855 wait complete. 2024-07-26T08:39:41.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.391+0000 7f2de5452740 20 client.15462 trim_cache size 0 max 16384 2024-07-26T08:39:41.394 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.392+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2dbc0b3da0 msgr2=0x7f2dbc0b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.392+0000 7f2ddd200640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2dbc0b3da0 0x7f2dbc0b62b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2dd00128c0 tx=0x7f2dd0009500 comp rx=0 tx=0).stop 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.392+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd81027e0 msgr2=0x7f2dd819c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.392+0000 7f2ddd200640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd81027e0 0x7f2dd819c4a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2dc402aa70 tx=0x7f2dc4003b90 comp rx=0 tx=0).stop 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 shutdown_connections 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f2dbc0b3da0 0x7f2dbc0b62b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2dd810fa40 0x7f2dd81a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd81027e0 0x7f2dd819c4a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 --2- 172.21.15.44:0/546272355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2dd8101e90 0x7f2dd819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 >> 172.21.15.44:0/546272355 conn(0x7f2dd80fdcf0 msgr2=0x7f2dd8100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 shutdown_connections 2024-07-26T08:39:41.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:41.393+0000 7f2ddd200640 1 -- 172.21.15.44:0/546272355 wait complete. 2024-07-26T08:39:42.322 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T08:39:42.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:42 smithi044 ceph-mon[32404]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:42 smithi116 ceph-mon[37097]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:42 smithi171 ceph-mon[37789]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:42.783 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:44 smithi044 ceph-mon[32404]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:39:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:44 smithi116 ceph-mon[37097]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:39:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:44 smithi171 ceph-mon[37789]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:39:45.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.149+0000 7fde0d800640 1 -- 172.21.15.44:0/433182141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810af30 msgr2=0x7fde0810b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.149+0000 7fde0d800640 1 --2- 172.21.15.44:0/433182141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810af30 0x7fde0810b340 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009860 tx=0x7fddf402a340 comp rx=0 tx=0).stop 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.150+0000 7fde0d800640 1 -- 172.21.15.44:0/433182141 shutdown_connections 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.150+0000 7fde0d800640 1 --2- 172.21.15.44:0/433182141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810b880 0x7fde081120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.150+0000 7fde0d800640 1 --2- 172.21.15.44:0/433182141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810af30 0x7fde0810b340 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.150+0000 7fde0d800640 1 --2- 172.21.15.44:0/433182141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde08105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.150+0000 7fde0d800640 1 -- 172.21.15.44:0/433182141 >> 172.21.15.44:0/433182141 conn(0x7fde08101060 msgr2=0x7fde081034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.151+0000 7fde0d800640 1 -- 172.21.15.44:0/433182141 shutdown_connections 2024-07-26T08:39:45.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.151+0000 7fde0d800640 1 -- 172.21.15.44:0/433182141 wait complete. 2024-07-26T08:39:45.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.152+0000 7fde0d800640 1 Processor -- start 2024-07-26T08:39:45.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 -- start start 2024-07-26T08:39:45.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde0819bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 0x7fde0819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810b880 0x7fde081a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde081154f0 con 0x7fde0810b880 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde08115370 con 0x7fde0810af30 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.153+0000 7fde0d800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde08115670 con 0x7fde081055f0 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.154+0000 7fde07e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde0819bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.154+0000 7fde07400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 0x7fde0819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.154+0000 7fde07e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde0819bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:45294/0 (socket says 172.21.15.44:45294) 2024-07-26T08:39:45.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.154+0000 7fde07e00640 1 -- 172.21.15.44:0/947926568 learned_addr learned my addr 172.21.15.44:0/947926568 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:45.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.154+0000 7fde07400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 0x7fde0819c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:43420/0 (socket says 172.21.15.44:43420) 2024-07-26T08:39:45.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.154+0000 7fde0ce00640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810b880 0x7fde081a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07e00640 1 -- 172.21.15.44:0/947926568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 msgr2=0x7fde0819c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07e00640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 0x7fde0819c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07e00640 1 -- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810b880 msgr2=0x7fde081a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07e00640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810b880 0x7fde081a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07e00640 1 -- 172.21.15.44:0/947926568 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde081a1d10 con 0x7fde081055f0 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07400640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 0x7fde0819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:45.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.155+0000 7fde07e00640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde0819bda0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fddf800e9f0 tx=0x7fddf800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.156+0000 7fde04c00640 1 -- 172.21.15.44:0/947926568 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf800cd80 con 0x7fde081055f0 2024-07-26T08:39:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.156+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde081a2000 con 0x7fde081055f0 2024-07-26T08:39:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.156+0000 7fde04c00640 1 -- 172.21.15.44:0/947926568 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf80108e0 con 0x7fde081055f0 2024-07-26T08:39:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.156+0000 7fde04c00640 1 -- 172.21.15.44:0/947926568 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8018980 con 0x7fde081055f0 2024-07-26T08:39:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.157+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde081a8760 con 0x7fde081055f0 2024-07-26T08:39:45.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.160+0000 7fde15a1a740 1 Processor -- start 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 -- start start 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcc8a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc8ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edcc8b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559edcbf8bd0 con 0x559edc8b41b0 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559edcbe80d0 con 0x559edc8bc390 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde15a1a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559edcbec110 con 0x559edc8dd070 2024-07-26T08:39:45.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.161+0000 7fde0ce00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcc8a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcc8a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:45304/0 (socket says 172.21.15.44:45304) 2024-07-26T08:39:45.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 -- 172.21.15.44:0/4045112140 learned_addr learned my addr 172.21.15.44:0/4045112140 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:45.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde07400640 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edcc8b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 -- 172.21.15.44:0/4045112140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 msgr2=0x559edcc8b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:45.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edcc8b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 -- 172.21.15.44:0/4045112140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 msgr2=0x559edcc8ad60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:39:45.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc8ad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.162+0000 7fde0ce00640 1 -- 172.21.15.44:0/4045112140 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559edc948650 con 0x559edc8dd070 2024-07-26T08:39:45.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.163+0000 7fde04c00640 1 -- 172.21.15.44:0/947926568 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fddf8010430 con 0x7fde081055f0 2024-07-26T08:39:45.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.163+0000 7fde0ce00640 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcc8a820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde00009f60 tx=0x7fde00009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb704447c167b455 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:45.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.164+0000 7fddfd600640 1 -- 172.21.15.44:0/4045112140 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0001d070 con 0x559edc8dd070 2024-07-26T08:39:45.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.164+0000 7fddfd600640 1 -- 172.21.15.44:0/4045112140 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde0000daa0 con 0x559edc8dd070 2024-07-26T08:39:45.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.164+0000 7fddfd600640 1 -- 172.21.15.44:0/4045112140 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00018c00 con 0x559edc8dd070 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 -- 172.21.15.44:0/4045112140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 msgr2=0x559edcc8a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcc8a820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde00009f60 tx=0x7fde00009f90 comp rx=0 tx=0).stop 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 -- 172.21.15.44:0/4045112140 shutdown_connections 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edcc8b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc8ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 --2- 172.21.15.44:0/4045112140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcc8a820 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde15a1a740 1 -- 172.21.15.44:0/4045112140 >> 172.21.15.44:0/4045112140 conn(0x559edc8d1460 msgr2=0x559edcc88710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:45.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.165+0000 7fde04c00640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fddec0b3d40 0x7fddec0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.166+0000 7fde04c00640 1 -- 172.21.15.44:0/947926568 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fddf80e4750 con 0x7fde081055f0 2024-07-26T08:39:45.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.166+0000 7fde15a1a740 1 -- 172.21.15.44:0/4045112140 shutdown_connections 2024-07-26T08:39:45.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.166+0000 7fde15a1a740 1 -- 172.21.15.44:0/4045112140 wait complete. 2024-07-26T08:39:45.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.166+0000 7fde07400640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fddec0b3d40 0x7fddec0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.167+0000 7fde15a1a740 1 -- start start 2024-07-26T08:39:45.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.167+0000 7fde15a1a740 1 Processor -- start 2024-07-26T08:39:45.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde07400640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fddec0b3d40 0x7fddec0b6250 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fddf402a860 tx=0x7fddf402c040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:45.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc848c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8bc390 0x559edcc84e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8dd070 0x559edcc85340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559edca4b700 con 0x559edc8b41b0 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559edcb846a0 con 0x559edc8dd070 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559edcc87b50 con 0x559edc8bc390 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde15a1a740 10 client.0 _ll_register_callbacks cb 0x559edcb94150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde07e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc848c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde07e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc848c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41498/0 (socket says 172.21.15.44:41498) 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.168+0000 7fde07e00640 1 -- 172.21.15.44:0/3276331033 learned_addr learned my addr 172.21.15.44:0/3276331033 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.169+0000 7fde07400640 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8bc390 0x559edcc84e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.169+0000 7fde07e00640 1 -- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8bc390 msgr2=0x559edcc84e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.169+0000 7fde07e00640 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8bc390 0x559edcc84e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.169+0000 7fde07e00640 1 -- 172.21.15.44:0/3276331033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8dd070 msgr2=0x559edcc85340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.169+0000 7fde07e00640 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8dd070 0x559edcc85340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.169+0000 7fde07e00640 1 -- 172.21.15.44:0/3276331033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559edcb9b840 con 0x559edc8b41b0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fde07e00640 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc848c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fddf800e9c0 tx=0x7fddf80107e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6363f1f25fb47425 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fddf2000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fddf2000640 1 -- 172.21.15.44:0/3276331033 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf80aafd0 con 0x559edc8b41b0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fddf2000640 1 -- 172.21.15.44:0/3276331033 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fddf80e4360 con 0x559edc8b41b0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde0000daa0 con 0x559edc8b41b0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fddf2000640 1 -- 172.21.15.44:0/3276331033 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fddf80af5d0 con 0x559edc8b41b0 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fde15a1a740 10 client.15471 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:45.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.170+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559edccad400 con 0x559edc8b41b0 2024-07-26T08:39:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.171+0000 7fddf2000640 1 -- 172.21.15.44:0/3276331033 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde0000daa0 con 0x559edc8b41b0 2024-07-26T08:39:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fddf2000640 1 -- 172.21.15.44:0/3276331033 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fddf80c5b10 con 0x559edc8b41b0 2024-07-26T08:39:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fde15a1a740 10 client.15471 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fde15a1a740 10 client.15471 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T08:39:45.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fde15a1a740 20 client.15471 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:45.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edc89b9f0 0x559edccb0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fde15a1a740 4 client.15471 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:39:45.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.172+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559edcbfab20 con 0x559edc89b9f0 2024-07-26T08:39:45.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.173+0000 7fde07400640 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edc89b9f0 0x559edccb0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.173+0000 7fde07400640 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edc89b9f0 0x559edccb0960 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4db9e3c5974b3dea server_cookie=e5cc877216727c8 in_seq=0 out_seq=0 2024-07-26T08:39:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.173+0000 7fddf2000640 10 client.15471 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:39:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.174+0000 7fddf2000640 1 -- 172.21.15.44:0/3276331033 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559edcbfab20 con 0x559edc89b9f0 2024-07-26T08:39:45.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.174+0000 7fddf2000640 10 client.15471 handle_command_reply: tid=0 2024-07-26T08:39:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.174+0000 7fde15a1a740 1 client.15471 shutdown 2024-07-26T08:39:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 msgr2=0x559edcc848c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc848c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fddf800e9c0 tx=0x7fddf80107e0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 shutdown_connections 2024-07-26T08:39:45.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edc89b9f0 0x559edccb0960 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8bc390 0x559edcc84e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8dd070 0x559edcc85340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3276331033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8b41b0 0x559edcc848c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 >> 172.21.15.44:0/3276331033 conn(0x559edc8d1460 msgr2=0x559edcc8c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:45.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.175+0000 7fddf2000640 0 client.15471 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:39:45.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.176+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 shutdown_connections 2024-07-26T08:39:45.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.176+0000 7fde15a1a740 1 -- 172.21.15.44:0/3276331033 wait complete. 2024-07-26T08:39:45.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.176+0000 7fde15a1a740 20 client.15471 trim_cache size 0 max 16384 2024-07-26T08:39:45.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 Processor -- start 2024-07-26T08:39:45.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 -- start start 2024-07-26T08:39:45.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8b41b0 0x559edcc86680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edccb6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8dd070 0x559edccb6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde081ae2d0 con 0x559edc8dd070 2024-07-26T08:39:45.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde081afe10 con 0x559edc8bc390 2024-07-26T08:39:45.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.204+0000 7fde15a1a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde081ae9a0 con 0x559edc8b41b0 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edccb6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8b41b0 0x559edcc86680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edccb6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:43446/0 (socket says 172.21.15.44:43446) 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 -- 172.21.15.44:0/3267622074 learned_addr learned my addr 172.21.15.44:0/3267622074 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 -- 172.21.15.44:0/3267622074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8b41b0 msgr2=0x559edcc86680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8b41b0 0x559edcc86680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 -- 172.21.15.44:0/3267622074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8dd070 msgr2=0x559edccb6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:45.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde0ce00640 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8dd070 0x559edccb6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8dd070 0x559edccb6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde07400640 1 -- 172.21.15.44:0/3267622074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559edcc89000 con 0x559edc8bc390 2024-07-26T08:39:45.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.205+0000 7fde0ce00640 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8dd070 0x559edccb6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:39:45.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.206+0000 7fde07400640 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edccb6120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fddf4004150 tx=0x7fddf403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62ab7d1b4698a842 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:45.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.206+0000 7fddfcc00640 1 -- 172.21.15.44:0/3267622074 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf4037040 con 0x559edc8bc390 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.206+0000 7fddfcc00640 1 -- 172.21.15.44:0/3267622074 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf4043040 con 0x559edc8bc390 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.206+0000 7fddfcc00640 1 -- 172.21.15.44:0/3267622074 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf4003bd0 con 0x559edc8bc390 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 -- 172.21.15.44:0/3267622074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 msgr2=0x559edccb6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edccb6120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fddf4004150 tx=0x7fddf403d000 comp rx=0 tx=0).stop 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 -- 172.21.15.44:0/3267622074 shutdown_connections 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc8dd070 0x559edccb6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8bc390 0x559edccb6120 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fddf4004150 tx=0x7fddf403d000 comp rx=0 tx=0).stop 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 --2- 172.21.15.44:0/3267622074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8b41b0 0x559edcc86680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 -- 172.21.15.44:0/3267622074 >> 172.21.15.44:0/3267622074 conn(0x559edc8d1460 msgr2=0x559edccaa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:45.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.207+0000 7fde15a1a740 1 -- 172.21.15.44:0/3267622074 shutdown_connections 2024-07-26T08:39:45.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.208+0000 7fde15a1a740 1 -- 172.21.15.44:0/3267622074 wait complete. 2024-07-26T08:39:45.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.209+0000 7fde15a1a740 1 -- start start 2024-07-26T08:39:45.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.209+0000 7fde15a1a740 1 Processor -- start 2024-07-26T08:39:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.209+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8b41b0 0x559edcca6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.209+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcca4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.209+0000 7fde15a1a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 0x559edcca4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde15a1a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde081af740 con 0x559edc870da0 2024-07-26T08:39:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde15a1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde081af070 con 0x559edc8b41b0 2024-07-26T08:39:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde15a1a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde081ac2b0 con 0x559edc8dd070 2024-07-26T08:39:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde15a1a740 10 client.0 _ll_register_callbacks cb 0x7fddf80aa740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde0ce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 0x559edcca4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde0ce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 0x559edcca4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41504/0 (socket says 172.21.15.44:41504) 2024-07-26T08:39:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.210+0000 7fde0ce00640 1 -- 172.21.15.44:0/29960455 learned_addr learned my addr 172.21.15.44:0/29960455 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:45.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde07e00640 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8b41b0 0x559edcca6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde0ce00640 1 -- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 msgr2=0x559edcca4430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde0ce00640 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcca4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde0ce00640 1 -- 172.21.15.44:0/29960455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8b41b0 msgr2=0x559edcca6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde0ce00640 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8b41b0 0x559edcca6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde0ce00640 1 -- 172.21.15.44:0/29960455 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559edcca4eb0 con 0x559edc870da0 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fde0ce00640 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 0x559edcca4970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fde0000dda0 tx=0x7fde0000ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae34f4abf4857d9a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:45.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.211+0000 7fddf2000640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:39:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fddf2000640 1 -- 172.21.15.44:0/29960455 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00026030 con 0x559edc870da0 2024-07-26T08:39:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fddf2000640 1 -- 172.21.15.44:0/29960455 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fde00015040 con 0x559edc870da0 2024-07-26T08:39:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddf4043040 con 0x559edc870da0 2024-07-26T08:39:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fddf2000640 1 -- 172.21.15.44:0/29960455 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fde00030050 con 0x559edc870da0 2024-07-26T08:39:45.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fde15a1a740 10 client.15474 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559edcd18c90 con 0x559edc870da0 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.212+0000 7fddf2000640 1 -- 172.21.15.44:0/29960455 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf4043040 con 0x559edc870da0 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fddf2000640 1 -- 172.21.15.44:0/29960455 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fde0000b040 con 0x559edc870da0 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fde15a1a740 10 client.15474 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fde15a1a740 10 client.15474 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fde15a1a740 20 client.15474 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fde15a1a740 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edcd190f0 0x559edcd1b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fde15a1a740 4 client.15474 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:39:45.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.213+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559edcbf65c0 con 0x559edcd190f0 2024-07-26T08:39:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.214+0000 7fde07e00640 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edcd190f0 0x559edcd1b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:45.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.215+0000 7fde07e00640 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edcd190f0 0x559edcd1b510 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=572972d067bd41c6 server_cookie=6b9ce9ad66f10b48 in_seq=0 out_seq=0 2024-07-26T08:39:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.215+0000 7fddf2000640 10 client.15474 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:39:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fddf2000640 1 -- 172.21.15.44:0/29960455 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559edcbf65c0 con 0x559edcd190f0 2024-07-26T08:39:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fddf2000640 10 client.15474 handle_command_reply: tid=0 2024-07-26T08:39:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 client.15474 shutdown 2024-07-26T08:39:45.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 msgr2=0x559edcca4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 0x559edcca4970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fde0000dda0 tx=0x7fde0000ddd0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 shutdown_connections 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x559edcd190f0 0x559edcd1b510 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559edc8dd070 0x559edcca4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559edc870da0 0x559edcca4970 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.216+0000 7fde15a1a740 1 --2- 172.21.15.44:0/29960455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559edc8b41b0 0x559edcca6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.217+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 >> 172.21.15.44:0/29960455 conn(0x559edc8d1460 msgr2=0x559edccaa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.217+0000 7fddf2000640 0 client.15474 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:39:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.217+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 shutdown_connections 2024-07-26T08:39:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.217+0000 7fde15a1a740 1 -- 172.21.15.44:0/29960455 wait complete. 2024-07-26T08:39:45.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.217+0000 7fde15a1a740 20 client.15474 trim_cache size 0 max 16384 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fddec0b3d40 msgr2=0x7fddec0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fddec0b3d40 0x7fddec0b6250 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fddf402a860 tx=0x7fddf402c040 comp rx=0 tx=0).stop 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 msgr2=0x7fde0819bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde0819bda0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fddf800e9f0 tx=0x7fddf800eeb0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 shutdown_connections 2024-07-26T08:39:45.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fddec0b3d40 0x7fddec0b6250 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde0810b880 0x7fde081a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0810af30 0x7fde0819c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 --2- 172.21.15.44:0/947926568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde081055f0 0x7fde0819bda0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.218+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 >> 172.21.15.44:0/947926568 conn(0x7fde08101060 msgr2=0x7fde0810a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.219+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 shutdown_connections 2024-07-26T08:39:45.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:45.219+0000 7fde0d800640 1 -- 172.21.15.44:0/947926568 wait complete. 2024-07-26T08:39:46.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T08:39:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:46 smithi044 ceph-mon[32404]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:46 smithi116 ceph-mon[37097]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:46 smithi171 ceph-mon[37789]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:46.604 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:48.438 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:48 smithi044 ceph-mon[32404]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:39:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:48 smithi116 ceph-mon[37097]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:39:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:48 smithi171 ceph-mon[37789]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:39:48.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 -- 172.21.15.44:0/475020759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 msgr2=0x7f1c40102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:48.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 --2- 172.21.15.44:0/475020759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c40102220 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009900 tx=0x7f1c3002a480 comp rx=0 tx=0).stop 2024-07-26T08:39:48.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 -- 172.21.15.44:0/475020759 shutdown_connections 2024-07-26T08:39:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 --2- 172.21.15.44:0/475020759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c40102760 0x7f1c4010f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 --2- 172.21.15.44:0/475020759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4010f9c0 0x7f1c40111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 --2- 172.21.15.44:0/475020759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c40102220 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 -- 172.21.15.44:0/475020759 >> 172.21.15.44:0/475020759 conn(0x7f1c400fdcf0 msgr2=0x7f1c40100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 -- 172.21.15.44:0/475020759 shutdown_connections 2024-07-26T08:39:48.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.959+0000 7f1c45000640 1 -- 172.21.15.44:0/475020759 wait complete. 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.961+0000 7f1c45000640 1 Processor -- start 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.961+0000 7f1c45000640 1 -- start start 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.961+0000 7f1c45000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c4010c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.961+0000 7f1c45000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c40102760 0x7f1c4010ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.961+0000 7f1c45000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4010f9c0 0x7f1c4010d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c45000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401154e0 con 0x7f1c4010f9c0 2024-07-26T08:39:48.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c45000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c40115360 con 0x7f1c40101e10 2024-07-26T08:39:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c45000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1c40115660 con 0x7f1c40102760 2024-07-26T08:39:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c3ea00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c40102760 0x7f1c4010ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4010f9c0 0x7f1c4010d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c3f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c4010c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c3f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c4010c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:43472/0 (socket says 172.21.15.44:43472) 2024-07-26T08:39:48.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.962+0000 7f1c3f400640 1 -- 172.21.15.44:0/2308651705 learned_addr learned my addr 172.21.15.44:0/2308651705 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.963+0000 7f1c3f400640 1 -- 172.21.15.44:0/2308651705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c40102760 msgr2=0x7f1c4010ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.963+0000 7f1c3f400640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c40102760 0x7f1c4010ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.963+0000 7f1c3f400640 1 -- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4010f9c0 msgr2=0x7f1c4010d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:48.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.963+0000 7f1c3f400640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4010f9c0 0x7f1c4010d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.963+0000 7f1c3f400640 1 -- 172.21.15.44:0/2308651705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c401ac3e0 con 0x7f1c40101e10 2024-07-26T08:39:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.964+0000 7f1c3f400640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c4010c950 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c30005ab0 tx=0x7f1c30003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.964+0000 7f1c37e00640 1 -- 172.21.15.44:0/2308651705 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3003f070 con 0x7f1c40101e10 2024-07-26T08:39:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.964+0000 7f1c37e00640 1 -- 172.21.15.44:0/2308651705 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c30031d60 con 0x7f1c40101e10 2024-07-26T08:39:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.964+0000 7f1c37e00640 1 -- 172.21.15.44:0/2308651705 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c30035030 con 0x7f1c40101e10 2024-07-26T08:39:48.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.964+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c401ac670 con 0x7f1c40101e10 2024-07-26T08:39:48.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.965+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c401acba0 con 0x7f1c40101e10 2024-07-26T08:39:48.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.969+0000 7f1c37e00640 1 -- 172.21.15.44:0/2308651705 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f1c30042070 con 0x7f1c40101e10 2024-07-26T08:39:48.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.969+0000 7f1c4d2bb740 1 Processor -- start 2024-07-26T08:39:48.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 -- start start 2024-07-26T08:39:48.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 0x55fe6f665030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2971b0 0x55fe6f665570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f29f390 0x55fe6f665ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fe6f5dbbf0 con 0x55fe6f29f390 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe6f5cb0f0 con 0x55fe6f2971b0 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.970+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fe6f5cf130 con 0x55fe6f2c0070 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 0x55fe6f665030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2971b0 0x55fe6f665570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 0x55fe6f665030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:45364/0 (socket says 172.21.15.44:45364) 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 -- 172.21.15.44:0/3951305595 learned_addr learned my addr 172.21.15.44:0/3951305595 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f29f390 0x55fe6f665ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 -- 172.21.15.44:0/3951305595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2971b0 msgr2=0x55fe6f665570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2971b0 0x55fe6f665570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 -- 172.21.15.44:0/3951305595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f29f390 msgr2=0x55fe6f665ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f29f390 0x55fe6f665ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.971+0000 7f1c3fe00640 1 -- 172.21.15.44:0/3951305595 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe6f32b650 con 0x55fe6f2c0070 2024-07-26T08:39:48.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.972+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 0x55fe6f665030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1c380039f0 tx=0x7f1c38003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=609f8d45936b11f2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:48.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.972+0000 7f1c34c00640 1 -- 172.21.15.44:0/3951305595 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3801a470 con 0x55fe6f2c0070 2024-07-26T08:39:48.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.972+0000 7f1c34c00640 1 -- 172.21.15.44:0/3951305595 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c38015070 con 0x55fe6f2c0070 2024-07-26T08:39:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.972+0000 7f1c34c00640 1 -- 172.21.15.44:0/3951305595 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c380107e0 con 0x55fe6f2c0070 2024-07-26T08:39:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.972+0000 7f1c37e00640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1c240b3dc0 0x7f1c240b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.972+0000 7f1c37e00640 1 -- 172.21.15.44:0/2308651705 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f1c30104be0 con 0x7f1c40101e10 2024-07-26T08:39:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c3f400640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1c240b3dc0 0x7f1c240b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/3951305595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 msgr2=0x55fe6f665030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:48.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 0x55fe6f665030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1c380039f0 tx=0x7f1c38003eb0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/3951305595 shutdown_connections 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f29f390 0x55fe6f665ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2971b0 0x55fe6f665570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/3951305595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2c0070 0x55fe6f665030 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.973+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/3951305595 >> 172.21.15.44:0/3951305595 conn(0x55fe6f2b4460 msgr2=0x55fe6f662f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.974+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/3951305595 shutdown_connections 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.974+0000 7f1c3f400640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1c240b3dc0 0x7f1c240b62d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55fe6f663cb0 tx=0x7f1c30108690 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:48.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.975+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/3951305595 wait complete. 2024-07-26T08:39:48.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.975+0000 7f1c4d2bb740 1 -- start start 2024-07-26T08:39:48.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.976+0000 7f1c4d2bb740 1 Processor -- start 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.976+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f6705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.976+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f670ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.976+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f671020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.976+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fe6f42e700 con 0x55fe6f2c0070 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe6f5676c0 con 0x55fe6f29f390 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fe6f673810 con 0x55fe6f2971b0 2024-07-26T08:39:48.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c4d2bb740 10 client.0 _ll_register_callbacks cb 0x55fe6f577170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:48.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f6705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f6705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:45366/0 (socket says 172.21.15.44:45366) 2024-07-26T08:39:48.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c3fe00640 1 -- 172.21.15.44:0/4153655380 learned_addr learned my addr 172.21.15.44:0/4153655380 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:48.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.977+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f671020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f670ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3fe00640 1 -- 172.21.15.44:0/4153655380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 msgr2=0x55fe6f670ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f670ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3fe00640 1 -- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 msgr2=0x55fe6f671020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f671020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3fe00640 1 -- 172.21.15.44:0/4153655380 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe6f57e860 con 0x55fe6f2971b0 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.978+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f6705a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c38011e90 tx=0x7f1c38010440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2a2b64686521c04 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:48.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.981+0000 7f1c29600640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:39:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.981+0000 7f1c29600640 1 -- 172.21.15.44:0/4153655380 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802e060 con 0x55fe6f2971b0 2024-07-26T08:39:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.981+0000 7f1c29600640 1 -- 172.21.15.44:0/4153655380 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f1c38010580 con 0x55fe6f2971b0 2024-07-26T08:39:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.981+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c38015070 con 0x55fe6f2971b0 2024-07-26T08:39:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.982+0000 7f1c29600640 1 -- 172.21.15.44:0/4153655380 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1c38005b30 con 0x55fe6f2971b0 2024-07-26T08:39:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.982+0000 7f1c4d2bb740 10 client.25612 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:48.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.982+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe6f67f410 con 0x55fe6f2971b0 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.982+0000 7f1c29600640 1 -- 172.21.15.44:0/4153655380 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c38015070 con 0x55fe6f2971b0 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.983+0000 7f1c29600640 1 -- 172.21.15.44:0/4153655380 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f1c38029950 con 0x55fe6f2971b0 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c4d2bb740 10 client.25612 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c4d2bb740 10 client.25612 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c4d2bb740 20 client.25612 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f27e9f0 0x55fe6f6827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c4d2bb740 4 client.25612 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:39:48.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe6f5ddb40 con 0x55fe6f27e9f0 2024-07-26T08:39:48.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.984+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f27e9f0 0x55fe6f6827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:48.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.985+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f27e9f0 0x55fe6f6827d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=565214767433ece5 server_cookie=cc2e245fa1f6c408 in_seq=0 out_seq=0 2024-07-26T08:39:48.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.985+0000 7f1c29600640 10 client.25612 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:39:48.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.986+0000 7f1c29600640 1 -- 172.21.15.44:0/4153655380 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe6f5ddb40 con 0x55fe6f27e9f0 2024-07-26T08:39:48.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.986+0000 7f1c29600640 10 client.25612 handle_command_reply: tid=0 2024-07-26T08:39:48.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.986+0000 7f1c4d2bb740 1 client.25612 shutdown 2024-07-26T08:39:48.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 msgr2=0x55fe6f6705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f6705a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c38011e90 tx=0x7f1c38010440 comp rx=0 tx=0).stop 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 shutdown_connections 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f27e9f0 0x55fe6f6827d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f671020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f670ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/4153655380 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f6705a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 >> 172.21.15.44:0/4153655380 conn(0x55fe6f2b4460 msgr2=0x55fe6f6681c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c29600640 0 client.25612 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:39:48.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.987+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 shutdown_connections 2024-07-26T08:39:48.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.988+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/4153655380 wait complete. 2024-07-26T08:39:48.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:48.988+0000 7f1c4d2bb740 20 client.25612 trim_cache size 0 max 16384 2024-07-26T08:39:49.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.002+0000 7f1c4d2bb740 1 Processor -- start 2024-07-26T08:39:49.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.002+0000 7f1c4d2bb740 1 -- start start 2024-07-26T08:39:49.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f672480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f661560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f666770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b6e50 con 0x55fe6f2c0070 2024-07-26T08:39:49.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b8990 con 0x55fe6f29f390 2024-07-26T08:39:49.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b7520 con 0x55fe6f2971b0 2024-07-26T08:39:49.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f672480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f666770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f666770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41542/0 (socket says 172.21.15.44:41542) 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.003+0000 7f1c3fe00640 1 -- 172.21.15.44:0/2090114489 learned_addr learned my addr 172.21.15.44:0/2090114489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f661560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3fe00640 1 -- 172.21.15.44:0/2090114489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 msgr2=0x55fe6f661560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f661560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3fe00640 1 -- 172.21.15.44:0/2090114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 msgr2=0x55fe6f666770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f666770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3fe00640 1 -- 172.21.15.44:0/2090114489 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe6f667e90 con 0x55fe6f2971b0 2024-07-26T08:39:49.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.004+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f672480 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009b60 tx=0x7f1c38003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aef2d806d6ca865 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:49.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.005+0000 7f1c2be00640 1 -- 172.21.15.44:0/2090114489 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802e070 con 0x55fe6f2971b0 2024-07-26T08:39:49.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.005+0000 7f1c2be00640 1 -- 172.21.15.44:0/2090114489 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c38015040 con 0x55fe6f2971b0 2024-07-26T08:39:49.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.005+0000 7f1c2be00640 1 -- 172.21.15.44:0/2090114489 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c38029b30 con 0x55fe6f2971b0 2024-07-26T08:39:49.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/2090114489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 msgr2=0x55fe6f672480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:49.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f672480 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009b60 tx=0x7f1c38003eb0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/2090114489 shutdown_connections 2024-07-26T08:39:49.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2c0070 0x55fe6f666770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f29f390 0x55fe6f661560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/2090114489 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f2971b0 0x55fe6f672480 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/2090114489 >> 172.21.15.44:0/2090114489 conn(0x55fe6f2b4460 msgr2=0x55fe6f666060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/2090114489 shutdown_connections 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.006+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/2090114489 wait complete. 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.007+0000 7f1c4d2bb740 1 -- start start 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.007+0000 7f1c4d2bb740 1 Processor -- start 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2971b0 0x55fe6f685010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f29f390 0x55fe6f6855f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2c0070 0x55fe6f6e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b82c0 con 0x55fe6f2971b0 2024-07-26T08:39:49.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b7bf0 con 0x55fe6f2c0070 2024-07-26T08:39:49.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1c401b4e30 con 0x55fe6f29f390 2024-07-26T08:39:49.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c4d2bb740 10 client.0 _ll_register_callbacks cb 0x7f1c38011580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:49.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2971b0 0x55fe6f685010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c3fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2971b0 0x55fe6f685010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41556/0 (socket says 172.21.15.44:41556) 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c3fe00640 1 -- 172.21.15.44:0/707164244 learned_addr learned my addr 172.21.15.44:0/707164244 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.008+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2c0070 0x55fe6f6e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f29f390 0x55fe6f6855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3fe00640 1 -- 172.21.15.44:0/707164244 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f29f390 msgr2=0x55fe6f6855f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f29f390 0x55fe6f6855f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3fe00640 1 -- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2971b0 msgr2=0x55fe6f685010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2971b0 0x55fe6f685010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3fe00640 1 -- 172.21.15.44:0/707164244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe6f6e3360 con 0x55fe6f2c0070 2024-07-26T08:39:49.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.009+0000 7f1c3fe00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2c0070 0x55fe6f6e2cf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009df0 tx=0x7f1c38013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a42dfad2fca7b2f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:49.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.010+0000 7f1c29600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.010+0000 7f1c29600640 1 -- 172.21.15.44:0/707164244 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802e030 con 0x55fe6f2c0070 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.010+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c38015040 con 0x55fe6f2c0070 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.010+0000 7f1c29600640 1 -- 172.21.15.44:0/707164244 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f1c38029670 con 0x55fe6f2c0070 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.010+0000 7f1c29600640 1 -- 172.21.15.44:0/707164244 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1c38004940 con 0x55fe6f2c0070 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.010+0000 7f1c4d2bb740 10 client.25613 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.011+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe6f6ea6a0 con 0x55fe6f2c0070 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.011+0000 7f1c29600640 1 -- 172.21.15.44:0/707164244 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c38015040 con 0x55fe6f2c0070 2024-07-26T08:39:49.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.011+0000 7f1c29600640 1 -- 172.21.15.44:0/707164244 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f1c3800fc90 con 0x55fe6f2c0070 2024-07-26T08:39:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.012+0000 7f1c4d2bb740 10 client.25613 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.012+0000 7f1c4d2bb740 10 client.25613 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T08:39:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.012+0000 7f1c4d2bb740 20 client.25613 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.012+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f253da0 0x55fe6f6ece00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.012+0000 7f1c4d2bb740 4 client.25613 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:39:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.012+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe6f5d95e0 con 0x55fe6f253da0 2024-07-26T08:39:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.013+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f253da0 0x55fe6f6ece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.013+0000 7f1c3ea00640 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f253da0 0x55fe6f6ece00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91482efd6eebae58 server_cookie=176a4422259843b in_seq=0 out_seq=0 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.013+0000 7f1c29600640 10 client.25613 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c29600640 1 -- 172.21.15.44:0/707164244 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe6f5d95e0 con 0x55fe6f253da0 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c29600640 10 client.25613 handle_command_reply: tid=0 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 client.25613 shutdown 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2c0070 msgr2=0x55fe6f6e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2c0070 0x55fe6f6e2cf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009df0 tx=0x7f1c38013040 comp rx=0 tx=0).stop 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 shutdown_connections 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x55fe6f253da0 0x55fe6f6ece00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c29600640 0 client.25613 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe6f29f390 0x55fe6f6855f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe6f2c0070 0x55fe6f6e2cf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 --2- 172.21.15.44:0/707164244 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fe6f2971b0 0x55fe6f685010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.014+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 >> 172.21.15.44:0/707164244 conn(0x55fe6f2b4460 msgr2=0x55fe6f67c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.015+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 shutdown_connections 2024-07-26T08:39:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.015+0000 7f1c4d2bb740 1 -- 172.21.15.44:0/707164244 wait complete. 2024-07-26T08:39:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.015+0000 7f1c4d2bb740 20 client.25613 trim_cache size 0 max 16384 2024-07-26T08:39:49.018 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:39:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1c240b3dc0 msgr2=0x7f1c240b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1c240b3dc0 0x7f1c240b62d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55fe6f663cb0 tx=0x7f1c30108690 comp rx=0 tx=0).stop 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 msgr2=0x7f1c4010c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c4010c950 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c30005ab0 tx=0x7f1c30003c80 comp rx=0 tx=0).stop 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 shutdown_connections 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f1c240b3dc0 0x7f1c240b62d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c40102760 0x7f1c4010ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c4010f9c0 0x7f1c4010d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 --2- 172.21.15.44:0/2308651705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c40101e10 0x7f1c4010c950 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.016+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 >> 172.21.15.44:0/2308651705 conn(0x7f1c400fdcf0 msgr2=0x7f1c40100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.017+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 shutdown_connections 2024-07-26T08:39:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:49.017+0000 7f1c45000640 1 -- 172.21.15.44:0/2308651705 wait complete. 2024-07-26T08:39:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:50 smithi044 ceph-mon[32404]: pgmap v517: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:39:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:50 smithi116 ceph-mon[37097]: pgmap v517: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:39:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:50 smithi171 ceph-mon[37789]: pgmap v517: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:39:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:52 smithi044 ceph-mon[32404]: pgmap v518: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:52 smithi116 ceph-mon[37097]: pgmap v518: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:52 smithi171 ceph-mon[37789]: pgmap v518: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:52.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T08:39:53.309 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:54.145 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:54 smithi044 ceph-mon[32404]: pgmap v519: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:39:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:54 smithi116 ceph-mon[37097]: pgmap v519: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:39:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:54 smithi171 ceph-mon[37789]: pgmap v519: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:39:55.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.679+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3516378923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd4108d20 msgr2=0x7f7dd4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.679+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3516378923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd4108d20 0x7f7dd4109130 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009900 tx=0x7f7dbc02a410 comp rx=0 tx=0).stop 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3516378923 shutdown_connections 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3516378923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd4109670 0x7f7dd410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3516378923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd4108d20 0x7f7dd4109130 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3516378923 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dd41033e0 0x7f7dd41037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3516378923 >> 172.21.15.44:0/3516378923 conn(0x7f7dd40fee70 msgr2=0x7f7dd41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3516378923 shutdown_connections 2024-07-26T08:39:55.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.680+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3516378923 wait complete. 2024-07-26T08:39:55.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.681+0000 7f7dd3e00640 1 Processor -- start 2024-07-26T08:39:55.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.681+0000 7f7dd3e00640 1 -- start start 2024-07-26T08:39:55.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 0x7f7dd4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dd4108d20 0x7f7dd4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd4109670 0x7f7dd4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd4113350 con 0x7f7dd41033e0 2024-07-26T08:39:55.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41131d0 con 0x7f7dd4109670 2024-07-26T08:39:55.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41134d0 con 0x7f7dd4108d20 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 0x7f7dd4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 0x7f7dd4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35704/0 (socket says 172.21.15.44:35704) 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd2a00640 1 -- 172.21.15.44:0/3209982165 learned_addr learned my addr 172.21.15.44:0/3209982165 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dca000640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dd4108d20 0x7f7dd4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.682+0000 7f7dd3400640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd4109670 0x7f7dd4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd2a00640 1 -- 172.21.15.44:0/3209982165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dd4108d20 msgr2=0x7f7dd4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dd4108d20 0x7f7dd4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd2a00640 1 -- 172.21.15.44:0/3209982165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd4109670 msgr2=0x7f7dd4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd4109670 0x7f7dd4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd2a00640 1 -- 172.21.15.44:0/3209982165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd41a8060 con 0x7f7dd41033e0 2024-07-26T08:39:55.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 0x7f7dd4075140 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e9f0 tx=0x7f7dc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dcbe00640 1 -- 172.21.15.44:0/3209982165 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400cd80 con 0x7f7dd41033e0 2024-07-26T08:39:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dcbe00640 1 -- 172.21.15.44:0/3209982165 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc40054d0 con 0x7f7dd41033e0 2024-07-26T08:39:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.683+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd41a8350 con 0x7f7dd41033e0 2024-07-26T08:39:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.684+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd41a8810 con 0x7f7dd41033e0 2024-07-26T08:39:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.684+0000 7f7dcbe00640 1 -- 172.21.15.44:0/3209982165 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4010690 con 0x7f7dd41033e0 2024-07-26T08:39:55.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.687+0000 7f7dcbe00640 1 -- 172.21.15.44:0/3209982165 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f7dc4004e90 con 0x7f7dd41033e0 2024-07-26T08:39:55.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.688+0000 7f7de0643740 1 Processor -- start 2024-07-26T08:39:55.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.688+0000 7f7de0643740 1 -- start start 2024-07-26T08:39:55.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dcbe00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f7db80b3d40 0x7f7db80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7de0643740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12b0070 0x558af1655030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7de0643740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 0x558af1655570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7de0643740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af128f390 0x558af1655ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7de0643740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558af15cbbf0 con 0x558af12b0070 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7de0643740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558af15bb0f0 con 0x558af12871b0 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7de0643740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558af15bf130 con 0x558af128f390 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dcbe00640 1 -- 172.21.15.44:0/3209982165 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f7dc40e47d0 con 0x7f7dd41033e0 2024-07-26T08:39:55.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12b0070 0x558af1655030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dca000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 0x558af1655570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dca000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 0x558af1655570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:34960/0 (socket says 172.21.15.44:34960) 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dca000640 1 -- 172.21.15.44:0/2194213889 learned_addr learned my addr 172.21.15.44:0/2194213889 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.689+0000 7f7dd3400640 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af128f390 0x558af1655ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd3400640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f7db80b3d40 0x7f7db80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2194213889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af128f390 msgr2=0x558af1655ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af128f390 0x558af1655ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2194213889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 msgr2=0x558af1655570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 0x558af1655570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2194213889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af131b650 con 0x558af12b0070 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dca000640 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 0x558af1655570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12b0070 0x558af1655030 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40e1b30 tx=0x7f7dc40a6fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f596912109ea1b9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.690+0000 7f7dc3e00640 1 -- 172.21.15.44:0/2194213889 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40aab00 con 0x558af12b0070 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.691+0000 7f7dc3e00640 1 -- 172.21.15.44:0/2194213889 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc40ab6f0 con 0x558af12b0070 2024-07-26T08:39:55.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.691+0000 7f7dc3e00640 1 -- 172.21.15.44:0/2194213889 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40e1db0 con 0x558af12b0070 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 -- 172.21.15.44:0/2194213889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12b0070 msgr2=0x558af1655030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7dd3400640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f7db80b3d40 0x7f7db80b6250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009fd0 tx=0x7f7dcc009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12b0070 0x558af1655030 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40e1b30 tx=0x7f7dc40a6fe0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 -- 172.21.15.44:0/2194213889 shutdown_connections 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af128f390 0x558af1655ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12871b0 0x558af1655570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 --2- 172.21.15.44:0/2194213889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12b0070 0x558af1655030 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 -- 172.21.15.44:0/2194213889 >> 172.21.15.44:0/2194213889 conn(0x558af12a4460 msgr2=0x558af1652f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 -- 172.21.15.44:0/2194213889 shutdown_connections 2024-07-26T08:39:55.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.692+0000 7f7de0643740 1 -- 172.21.15.44:0/2194213889 wait complete. 2024-07-26T08:39:55.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.693+0000 7f7de0643740 1 -- start start 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.693+0000 7f7de0643740 1 Processor -- start 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1660630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1660b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af16610b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558af141e700 con 0x558af128f390 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558af15576c0 con 0x558af12b0070 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558af1663810 con 0x558af12871b0 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7de0643740 10 client.0 _ll_register_callbacks cb 0x558af1567170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1660630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1660630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:46974/0 (socket says 172.21.15.44:46974) 2024-07-26T08:39:55.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7dd2a00640 1 -- 172.21.15.44:0/1864891234 learned_addr learned my addr 172.21.15.44:0/1864891234 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.694+0000 7f7dca000640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1660b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd3400640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af16610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd2a00640 1 -- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 msgr2=0x558af16610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af16610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd2a00640 1 -- 172.21.15.44:0/1864891234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 msgr2=0x558af1660b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1660b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd2a00640 1 -- 172.21.15.44:0/1864891234 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af156e860 con 0x558af12871b0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1660630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40e1b90 tx=0x7f7dc4004830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb65b3f3b68a382 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.695+0000 7f7dc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7dc0c00640 1 -- 172.21.15.44:0/1864891234 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40c5060 con 0x558af12871b0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7dc0c00640 1 -- 172.21.15.44:0/1864891234 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f7dc40e4370 con 0x558af12871b0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc40ab6f0 con 0x558af12871b0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7dc0c00640 1 -- 172.21.15.44:0/1864891234 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7dc40cc020 con 0x558af12871b0 2024-07-26T08:39:55.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7de0643740 10 client.25627 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558af166f440 con 0x558af12871b0 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7dc0c00640 1 -- 172.21.15.44:0/1864891234 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc40ab6f0 con 0x558af12871b0 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7dc0c00640 1 -- 172.21.15.44:0/1864891234 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7dc40b3c10 con 0x558af12871b0 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7de0643740 10 client.25627 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7de0643740 10 client.25627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.696+0000 7f7de0643740 20 client.25627 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.697+0000 7f7de0643740 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af126e9f0 0x558af16727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.697+0000 7f7de0643740 4 client.25627 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.697+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558af15cdb40 con 0x558af126e9f0 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.697+0000 7f7dca000640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af126e9f0 0x558af16727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.698+0000 7f7dca000640 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af126e9f0 0x558af16727a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cce0e37e4d77572 server_cookie=dbea1ab15d57fde0 in_seq=0 out_seq=0 2024-07-26T08:39:55.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.698+0000 7f7dc0c00640 10 client.25627 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.698+0000 7f7dc0c00640 1 -- 172.21.15.44:0/1864891234 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558af15cdb40 con 0x558af126e9f0 2024-07-26T08:39:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.698+0000 7f7dc0c00640 10 client.25627 handle_command_reply: tid=0 2024-07-26T08:39:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.698+0000 7f7de0643740 1 client.25627 shutdown 2024-07-26T08:39:55.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 msgr2=0x558af1660630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1660630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40e1b90 tx=0x7f7dc4004830 comp rx=0 tx=0).stop 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 shutdown_connections 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af126e9f0 0x558af16727a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af16610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1660b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 --2- 172.21.15.44:0/1864891234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1660630 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 >> 172.21.15.44:0/1864891234 conn(0x558af12a4460 msgr2=0x558af1658210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7dc0c00640 0 client.25627 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 shutdown_connections 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.699+0000 7f7de0643740 1 -- 172.21.15.44:0/1864891234 wait complete. 2024-07-26T08:39:55.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.700+0000 7f7de0643740 20 client.25627 trim_cache size 0 max 16384 2024-07-26T08:39:55.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 Processor -- start 2024-07-26T08:39:55.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 -- start start 2024-07-26T08:39:55.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1662510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1651560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af1656770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41b2960 con 0x558af128f390 2024-07-26T08:39:55.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41b44a0 con 0x558af12b0070 2024-07-26T08:39:55.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7de0643740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41b3030 con 0x558af12871b0 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.714+0000 7f7dca000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1651560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1662510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dca000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1651560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35738/0 (socket says 172.21.15.44:35738) 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dca000640 1 -- 172.21.15.44:0/315226112 learned_addr learned my addr 172.21.15.44:0/315226112 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd3400640 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af1656770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 -- 172.21.15.44:0/315226112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 msgr2=0x558af1656770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af1656770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 -- 172.21.15.44:0/315226112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 msgr2=0x558af1651560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1651560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 -- 172.21.15.44:0/315226112 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af16618f0 con 0x558af12871b0 2024-07-26T08:39:55.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dca000640 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1651560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:55.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.715+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1662510 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40a7070 tx=0x7f7dc40a6fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa284c62eef18b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:55.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.716+0000 7f7dc3400640 1 -- 172.21.15.44:0/315226112 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40c5070 con 0x558af12871b0 2024-07-26T08:39:55.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.716+0000 7f7dc3400640 1 -- 172.21.15.44:0/315226112 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc40aab00 con 0x558af12871b0 2024-07-26T08:39:55.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.716+0000 7f7dc3400640 1 -- 172.21.15.44:0/315226112 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4087070 con 0x558af12871b0 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 -- 172.21.15.44:0/315226112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 msgr2=0x558af1662510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1662510 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40a7070 tx=0x7f7dc40a6fe0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 -- 172.21.15.44:0/315226112 shutdown_connections 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af12b0070 0x558af1656770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af128f390 0x558af1651560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 --2- 172.21.15.44:0/315226112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12871b0 0x558af1662510 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 -- 172.21.15.44:0/315226112 >> 172.21.15.44:0/315226112 conn(0x558af12a4460 msgr2=0x558af1656060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 -- 172.21.15.44:0/315226112 shutdown_connections 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 -- 172.21.15.44:0/315226112 wait complete. 2024-07-26T08:39:55.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.717+0000 7f7de0643740 1 -- start start 2024-07-26T08:39:55.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 Processor -- start 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 0x558af1675740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af128f390 0x558af1675d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12b0070 0x558af16d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41b3dd0 con 0x558af12871b0 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41b3700 con 0x558af128f390 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41b0940 con 0x558af12b0070 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.718+0000 7f7de0643740 10 client.0 _ll_register_callbacks cb 0x7f7dc40046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 0x558af1675740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 0x558af1675740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35746/0 (socket says 172.21.15.44:35746) 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2408617089 learned_addr learned my addr 172.21.15.44:0/2408617089 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:39:55.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dca000640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af128f390 0x558af1675d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd3400640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12b0070 0x558af16d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2408617089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12b0070 msgr2=0x558af16d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12b0070 0x558af16d33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af128f390 msgr2=0x558af1675d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af128f390 0x558af1675d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dd2a00640 1 -- 172.21.15.44:0/2408617089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af16d3a20 con 0x558af12871b0 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.719+0000 7f7dca000640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af128f390 0x558af1675d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7dd2a00640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 0x558af1675740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40aacf0 tx=0x7f7dc4004830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4548238bf36e8c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:39:55.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7dc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:39:55.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7dc0c00640 1 -- 172.21.15.44:0/2408617089 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4087070 con 0x558af12871b0 2024-07-26T08:39:55.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7dc0c00640 1 -- 172.21.15.44:0/2408617089 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f7dc40c5b30 con 0x558af12871b0 2024-07-26T08:39:55.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc40aab00 con 0x558af12871b0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7dc0c00640 1 -- 172.21.15.44:0/2408617089 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7dc40c5040 con 0x558af12871b0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7de0643740 10 client.15507 fetch_fsmap learned FSMap version 30 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.720+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558af16d3a20 con 0x558af12871b0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7dc0c00640 1 -- 172.21.15.44:0/2408617089 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc40aab00 con 0x558af12871b0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7dc0c00640 1 -- 172.21.15.44:0/2408617089 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7dc40b5060 con 0x558af12871b0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7de0643740 10 client.15507 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7de0643740 10 client.15507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7de0643740 20 client.15507 populate_metadata read hostname 'smithi044' 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7de0643740 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af1243da0 0x558af16dd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7de0643740 4 client.15507 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558af15c95e0 con 0x558af1243da0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.721+0000 7f7dca000640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af1243da0 0x558af16dd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.722+0000 7f7dca000640 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af1243da0 0x558af16dd170 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d314c12d22dd4703 server_cookie=e203e85f550d826e in_seq=0 out_seq=0 2024-07-26T08:39:55.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.722+0000 7f7dc0c00640 10 client.15507 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.722+0000 7f7dc0c00640 1 -- 172.21.15.44:0/2408617089 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558af15c95e0 con 0x558af1243da0 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.722+0000 7f7dc0c00640 10 client.15507 handle_command_reply: tid=0 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.722+0000 7f7de0643740 1 client.15507 shutdown 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 msgr2=0x558af1675740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 0x558af1675740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40aacf0 tx=0x7f7dc4004830 comp rx=0 tx=0).stop 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 shutdown_connections 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x558af1243da0 0x558af16dd170 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558af12b0070 0x558af16d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558af128f390 0x558af1675d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 --2- 172.21.15.44:0/2408617089 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558af12871b0 0x558af1675740 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 >> 172.21.15.44:0/2408617089 conn(0x558af12a4460 msgr2=0x558af1662250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7dc0c00640 0 client.15507 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:39:55.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 shutdown_connections 2024-07-26T08:39:55.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 1 -- 172.21.15.44:0/2408617089 wait complete. 2024-07-26T08:39:55.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.723+0000 7f7de0643740 20 client.15507 trim_cache size 0 max 16384 2024-07-26T08:39:55.727 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f7db80b3d40 msgr2=0x7f7db80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f7db80b3d40 0x7f7db80b6250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009fd0 tx=0x7f7dcc009500 comp rx=0 tx=0).stop 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 msgr2=0x7f7dd4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 0x7f7dd4075140 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e9f0 tx=0x7f7dc400edf0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 shutdown_connections 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f7db80b3d40 0x7f7db80b6250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dd4108d20 0x7f7dd4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd4109670 0x7f7dd4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 --2- 172.21.15.44:0/3209982165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7dd41033e0 0x7f7dd4075140 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 >> 172.21.15.44:0/3209982165 conn(0x7f7dd40fee70 msgr2=0x7f7dd4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:39:55.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 shutdown_connections 2024-07-26T08:39:55.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:39:55.724+0000 7f7dd3e00640 1 -- 172.21.15.44:0/3209982165 wait complete. 2024-07-26T08:39:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:56 smithi044 ceph-mon[32404]: pgmap v520: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:56 smithi116 ceph-mon[37097]: pgmap v520: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:56 smithi171 ceph-mon[37789]: pgmap v520: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:39:57.701 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T08:39:57.702 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T08:39:58.182 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:39:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:39:58 smithi044 ceph-mon[32404]: pgmap v521: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:39:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:39:58 smithi116 ceph-mon[37097]: pgmap v521: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:39:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:39:58 smithi171 ceph-mon[37789]: pgmap v521: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:40:00.259 INFO:tasks.workunit.client.0.smithi044.stderr:Updating files: 54% (6878/12697) Updating files: 55% (6984/12697) Updating files: 56% (7111/12697) Updating files: 57% (7238/12697) Updating files: 58% (7365/12697) Updating files: 59% (7492/12697) Updating files: 60% (7619/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 88% (11284/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-26T08:40:00.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:00 smithi044 ceph-mon[32404]: pgmap v522: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:40:00.391 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:00 smithi044 ceph-mon[32404]: overall HEALTH_OK 2024-07-26T08:40:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:00 smithi116 ceph-mon[37097]: pgmap v522: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:40:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:00 smithi116 ceph-mon[37097]: overall HEALTH_OK 2024-07-26T08:40:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:00 smithi171 ceph-mon[37789]: pgmap v522: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:40:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:00 smithi171 ceph-mon[37789]: overall HEALTH_OK 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 -- 172.21.15.44:0/2926140691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4810f940 msgr2=0x7fea48111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 --2- 172.21.15.44:0/2926140691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4810f940 0x7fea48111d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fea40009be0 tx=0x7fea4002a6c0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 -- 172.21.15.44:0/2926140691 shutdown_connections 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 --2- 172.21.15.44:0/2926140691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea4810f940 0x7fea48111d80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 --2- 172.21.15.44:0/2926140691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea48075b30 0x7fea4810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 --2- 172.21.15.44:0/2926140691 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea480751e0 0x7fea480755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 -- 172.21.15.44:0/2926140691 >> 172.21.15.44:0/2926140691 conn(0x7fea480fee70 msgr2=0x7fea481012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 -- 172.21.15.44:0/2926140691 shutdown_connections 2024-07-26T08:40:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.590+0000 7fea4ca00640 1 -- 172.21.15.44:0/2926140691 wait complete. 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.591+0000 7fea4ca00640 1 Processor -- start 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.591+0000 7fea4ca00640 1 -- start start 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea4ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea480751e0 0x7fea48106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea4ca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 0x7fea481030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea4ca00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea4810f940 0x7fea481037d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea4ca00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea481154d0 con 0x7fea48075b30 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea4ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea48115350 con 0x7fea480751e0 2024-07-26T08:40:00.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea4ca00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fea48115650 con 0x7fea4810f940 2024-07-26T08:40:00.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea45e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 0x7fea481030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea46800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea480751e0 0x7fea48106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea45e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 0x7fea481030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35766/0 (socket says 172.21.15.44:35766) 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea45e00640 1 -- 172.21.15.44:0/739241615 learned_addr learned my addr 172.21.15.44:0/739241615 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea46800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea480751e0 0x7fea48106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:35000/0 (socket says 172.21.15.44:35000) 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.592+0000 7fea47200640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea4810f940 0x7fea481037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea45e00640 1 -- 172.21.15.44:0/739241615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea4810f940 msgr2=0x7fea481037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea45e00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea4810f940 0x7fea481037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea45e00640 1 -- 172.21.15.44:0/739241615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea480751e0 msgr2=0x7fea48106dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea45e00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea480751e0 0x7fea48106dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea45e00640 1 -- 172.21.15.44:0/739241615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea481ac410 con 0x7fea48075b30 2024-07-26T08:40:00.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea45e00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 0x7fea481030b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea3000e9a0 tx=0x7fea3000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:00.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea3f400640 1 -- 172.21.15.44:0/739241615 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3000cd80 con 0x7fea48075b30 2024-07-26T08:40:00.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea481ac640 con 0x7fea48075b30 2024-07-26T08:40:00.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea3f400640 1 -- 172.21.15.44:0/739241615 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea300108e0 con 0x7fea48075b30 2024-07-26T08:40:00.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.593+0000 7fea3f400640 1 -- 172.21.15.44:0/739241615 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea30018960 con 0x7fea48075b30 2024-07-26T08:40:00.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.594+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea481acb80 con 0x7fea48075b30 2024-07-26T08:40:00.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.597+0000 7fea54038740 1 Processor -- start 2024-07-26T08:40:00.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.597+0000 7fea54038740 1 -- start start 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.597+0000 7fea54038740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4be2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.597+0000 7fea54038740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d4815490 0x55f1d4be3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea54038740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d481d670 0x55f1d4be39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea54038740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4b59be0 con 0x55f1d481d670 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea54038740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4b490e0 con 0x55f1d4815490 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea54038740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4b4d120 con 0x55f1d483e350 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4be2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4be2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:47014/0 (socket says 172.21.15.44:47014) 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 -- 172.21.15.44:0/1591024618 learned_addr learned my addr 172.21.15.44:0/1591024618 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea46800640 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d4815490 0x55f1d4be3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 -- 172.21.15.44:0/1591024618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d4815490 msgr2=0x55f1d4be3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d4815490 0x55f1d4be3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 -- 172.21.15.44:0/1591024618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d481d670 msgr2=0x55f1d4be39d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d481d670 0x55f1d4be39d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.598+0000 7fea47200640 1 -- 172.21.15.44:0/1591024618 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d48a9930 con 0x55f1d483e350 2024-07-26T08:40:00.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.599+0000 7fea47200640 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4be2f50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fea4002acb0 tx=0x7fea40030420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14422c5137ceda7e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:00.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.599+0000 7fea37400640 1 -- 172.21.15.44:0/1591024618 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40041030 con 0x55f1d483e350 2024-07-26T08:40:00.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.599+0000 7fea37400640 1 -- 172.21.15.44:0/1591024618 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea40003e20 con 0x55f1d483e350 2024-07-26T08:40:00.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.599+0000 7fea37400640 1 -- 172.21.15.44:0/1591024618 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40036040 con 0x55f1d483e350 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.599+0000 7fea3f400640 1 -- 172.21.15.44:0/739241615 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fea30010a90 con 0x7fea48075b30 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 -- 172.21.15.44:0/1591024618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 msgr2=0x55f1d4be2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4be2f50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fea4002acb0 tx=0x7fea40030420 comp rx=0 tx=0).stop 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 -- 172.21.15.44:0/1591024618 shutdown_connections 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d481d670 0x55f1d4be39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d4815490 0x55f1d4be3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 --2- 172.21.15.44:0/1591024618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4be2f50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.600+0000 7fea54038740 1 -- 172.21.15.44:0/1591024618 >> 172.21.15.44:0/1591024618 conn(0x55f1d4832740 msgr2=0x55f1d4be0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.601+0000 7fea54038740 1 -- 172.21.15.44:0/1591024618 shutdown_connections 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.601+0000 7fea54038740 1 -- 172.21.15.44:0/1591024618 wait complete. 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.601+0000 7fea54038740 1 -- start start 2024-07-26T08:40:00.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.601+0000 7fea3f400640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fea2c0b3c50 0x7fea2c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 Processor -- start 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea3f400640 1 -- 172.21.15.44:0/739241615 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fea300e46e0 con 0x7fea48075b30 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea47200640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fea2c0b3c50 0x7fea2c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4bee650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4beeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4bfa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f1d49ac890 con 0x55f1d4815490 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4ae5320 con 0x55f1d481d670 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.602+0000 7fea54038740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4bf16d0 con 0x55f1d483e350 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea54038740 10 client.0 _ll_register_callbacks cb 0x55f1d4b11000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fea2c0b3c50 0x7fea2c0b6160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fea40009600 tx=0x7fea40004ab0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4bfa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea46800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4bee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4bfa9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:47030/0 (socket says 172.21.15.44:47030) 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 -- 172.21.15.44:0/1176963730 learned_addr learned my addr 172.21.15.44:0/1176963730 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea45e00640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4beeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 -- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 msgr2=0x55f1d4beeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4beeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 -- 172.21.15.44:0/1176963730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 msgr2=0x55f1d4bee650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4bee650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea47200640 1 -- 172.21.15.44:0/1176963730 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d4bef410 con 0x55f1d483e350 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea45e00640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4beeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:00.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.603+0000 7fea46800640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4bee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.604+0000 7fea47200640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4bfa9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fea40030720 tx=0x7fea40030420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c5191180cd4f04a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.604+0000 7fea2be00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.604+0000 7fea2be00640 1 -- 172.21.15.44:0/1176963730 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40056070 con 0x55f1d483e350 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.604+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea40003e20 con 0x55f1d483e350 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.604+0000 7fea2be00640 1 -- 172.21.15.44:0/1176963730 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fea40030a30 con 0x55f1d483e350 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea2be00640 1 -- 172.21.15.44:0/1176963730 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fea40062050 con 0x55f1d483e350 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 10 client.25645 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:00.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1d4bfd5f0 con 0x55f1d483e350 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea2be00640 1 -- 172.21.15.44:0/1176963730 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea40003e20 con 0x55f1d483e350 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea2be00640 1 -- 172.21.15.44:0/1176963730 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fea400314d0 con 0x55f1d483e350 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 10 client.25645 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 20 client.25645 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47fccd0 0x55f1d4c00950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 4 client.25645 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.605+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d4b5bb30 con 0x55f1d47fccd0 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.606+0000 7fea46800640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47fccd0 0x55f1d4c00950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.606+0000 7fea46800640 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47fccd0 0x55f1d4c00950 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c82fa4883f13a6b server_cookie=70124444be388480 in_seq=0 out_seq=0 2024-07-26T08:40:00.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.606+0000 7fea2be00640 10 client.25645 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:00.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.607+0000 7fea2be00640 1 -- 172.21.15.44:0/1176963730 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f1d4b5bb30 con 0x55f1d47fccd0 2024-07-26T08:40:00.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.607+0000 7fea2be00640 10 client.25645 handle_command_reply: tid=0 2024-07-26T08:40:00.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.607+0000 7fea54038740 1 client.25645 shutdown 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 msgr2=0x55f1d4bfa9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4bfa9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fea40030720 tx=0x7fea40030420 comp rx=0 tx=0).stop 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 shutdown_connections 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47fccd0 0x55f1d4c00950 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4bfa9c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4beeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 --2- 172.21.15.44:0/1176963730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4bee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea2be00640 0 client.25645 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 >> 172.21.15.44:0/1176963730 conn(0x55f1d4832740 msgr2=0x55f1d4be60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 shutdown_connections 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 1 -- 172.21.15.44:0/1176963730 wait complete. 2024-07-26T08:40:00.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.608+0000 7fea54038740 20 client.25645 trim_cache size 0 max 16384 2024-07-26T08:40:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.627+0000 7fea54038740 1 Processor -- start 2024-07-26T08:40:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.627+0000 7fea54038740 1 -- start start 2024-07-26T08:40:00.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea54038740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d4815490 0x55f1d4bedf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea54038740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4bf0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea54038740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 0x55f1d4bfa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea54038740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea481b6d80 con 0x55f1d483e350 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea54038740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea481b88c0 con 0x55f1d481d670 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea54038740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fea481b7450 con 0x55f1d4815490 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea47200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 0x55f1d4bfa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea47200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 0x55f1d4bfa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35774/0 (socket says 172.21.15.44:35774) 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea46800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d4815490 0x55f1d4bedf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea47200640 1 -- 172.21.15.44:0/1279146286 learned_addr learned my addr 172.21.15.44:0/1279146286 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:00.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.628+0000 7fea45e00640 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4bf0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea46800640 1 -- 172.21.15.44:0/1279146286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 msgr2=0x55f1d4bf0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea46800640 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4bf0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea46800640 1 -- 172.21.15.44:0/1279146286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 msgr2=0x55f1d4bfa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea46800640 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 0x55f1d4bfa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea46800640 1 -- 172.21.15.44:0/1279146286 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d4bef910 con 0x55f1d4815490 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea47200640 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 0x55f1d4bfa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:00.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea46800640 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d4815490 0x55f1d4bedf00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55f1d4ae5380 tx=0x7fea38005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc7cba464f63c101 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea36a00640 1 -- 172.21.15.44:0/1279146286 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38004320 con 0x55f1d4815490 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea36a00640 1 -- 172.21.15.44:0/1279146286 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea380044c0 con 0x55f1d4815490 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.629+0000 7fea36a00640 1 -- 172.21.15.44:0/1279146286 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38010930 con 0x55f1d4815490 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 -- 172.21.15.44:0/1279146286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d4815490 msgr2=0x55f1d4bedf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d4815490 0x55f1d4bedf00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55f1d4ae5380 tx=0x7fea38005f90 comp rx=0 tx=0).stop 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 -- 172.21.15.44:0/1279146286 shutdown_connections 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d483e350 0x55f1d4bfa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4bf0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 --2- 172.21.15.44:0/1279146286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d4815490 0x55f1d4bedf00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.630+0000 7fea54038740 1 -- 172.21.15.44:0/1279146286 >> 172.21.15.44:0/1279146286 conn(0x55f1d4832740 msgr2=0x55f1d4be45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.631+0000 7fea54038740 1 -- 172.21.15.44:0/1279146286 shutdown_connections 2024-07-26T08:40:00.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.631+0000 7fea54038740 1 -- 172.21.15.44:0/1279146286 wait complete. 2024-07-26T08:40:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.631+0000 7fea54038740 1 -- start start 2024-07-26T08:40:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.631+0000 7fea54038740 1 Processor -- start 2024-07-26T08:40:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4c04020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4c014f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4c01a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea481b81f0 con 0x55f1d4815490 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea481b7b20 con 0x55f1d481d670 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fea481b4d60 con 0x55f1d483e350 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea54038740 10 client.0 _ll_register_callbacks cb 0x7fea4004a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea46800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4c04020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea45e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4c014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea46800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4c04020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35780/0 (socket says 172.21.15.44:35780) 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea46800640 1 -- 172.21.15.44:0/3891250687 learned_addr learned my addr 172.21.15.44:0/3891250687 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.632+0000 7fea47200640 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4c01a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea46800640 1 -- 172.21.15.44:0/3891250687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 msgr2=0x55f1d4c01a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea46800640 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4c01a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea46800640 1 -- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 msgr2=0x55f1d4c014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea46800640 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4c014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea46800640 1 -- 172.21.15.44:0/3891250687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d4c021a0 con 0x55f1d4815490 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea46800640 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4c04020 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fea38012580 tx=0x7fea380104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb11b678a3bfc7a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea2be00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.633+0000 7fea2be00640 1 -- 172.21.15.44:0/3891250687 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38004320 con 0x55f1d4815490 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.634+0000 7fea2be00640 1 -- 172.21.15.44:0/3891250687 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fea38005030 con 0x55f1d4815490 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.634+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f1d4b575d0 con 0x55f1d4815490 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.634+0000 7fea2be00640 1 -- 172.21.15.44:0/3891250687 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fea3803c050 con 0x55f1d4815490 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.634+0000 7fea54038740 10 client.15522 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:00.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.634+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1d4c68e10 con 0x55f1d4815490 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.634+0000 7fea2be00640 1 -- 172.21.15.44:0/3891250687 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f1d4b575d0 con 0x55f1d4815490 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea2be00640 1 -- 172.21.15.44:0/3891250687 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fea380106b0 con 0x55f1d4815490 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea54038740 10 client.15522 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea54038740 10 client.15522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea54038740 20 client.15522 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea54038740 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47d2080 0x55f1d4c6b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea54038740 4 client.15522 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T08:40:00.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1d4b54d10 con 0x55f1d47d2080 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea45e00640 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47d2080 0x55f1d4c6b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea45e00640 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47d2080 0x55f1d4c6b5d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f0d18c58d779146 server_cookie=6161a8fb1fb4f5b0 in_seq=0 out_seq=0 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.635+0000 7fea2be00640 10 client.15522 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea2be00640 1 -- 172.21.15.44:0/3891250687 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1d4b54d10 con 0x55f1d47d2080 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea2be00640 10 client.15522 handle_command_reply: tid=0 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 client.15522 shutdown 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 msgr2=0x55f1d4c04020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4c04020 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fea38012580 tx=0x7fea380104c0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 shutdown_connections 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55f1d47d2080 0x55f1d4c6b5d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d483e350 0x55f1d4c01a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d481d670 0x55f1d4c014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea2be00640 0 client.15522 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 --2- 172.21.15.44:0/3891250687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f1d4815490 0x55f1d4c04020 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.637+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 >> 172.21.15.44:0/3891250687 conn(0x55f1d4832740 msgr2=0x55f1d4bf04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.638+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 shutdown_connections 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.638+0000 7fea54038740 1 -- 172.21.15.44:0/3891250687 wait complete. 2024-07-26T08:40:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.638+0000 7fea54038740 20 client.15522 trim_cache size 0 max 16384 2024-07-26T08:40:00.642 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:40:00.642 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-07-26T08:40:00.642 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c589f5ca-bb76-41a1-9d2f-a4a552371d4e", 2024-07-26T08:40:00.642 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-07-26T08:40:00.642 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fea2c0b3c50 msgr2=0x7fea2c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fea2c0b3c50 0x7fea2c0b6160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fea40009600 tx=0x7fea40004ab0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 msgr2=0x7fea481030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 0x7fea481030b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea3000e9a0 tx=0x7fea3000ee60 comp rx=0 tx=0).stop 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 shutdown_connections 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fea2c0b3c50 0x7fea2c0b6160 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea4810f940 0x7fea481037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea48075b30 0x7fea481030b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 --2- 172.21.15.44:0/739241615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea480751e0 0x7fea48106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 >> 172.21.15.44:0/739241615 conn(0x7fea480fee70 msgr2=0x7fea48109450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 shutdown_connections 2024-07-26T08:40:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:00.639+0000 7fea4ca00640 1 -- 172.21.15.44:0/739241615 wait complete. 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.737 INFO:tasks.workunit.client.0.smithi044.stderr: git switch -c 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr:Or undo this operation with: 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr: git switch - 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:00.738 INFO:tasks.workunit.client.0.smithi044.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-26T08:40:00.746 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T08:40:00.842 INFO:tasks.workunit.client.0.smithi044.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T08:40:00.869 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T08:40:00.870 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T08:40:01.918 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T08:40:02.267 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T08:40:02.371 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T08:40:02.393 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:02.460 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T08:40:02.465 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T08:40:02.465 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T08:40:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:02 smithi044 ceph-mon[32404]: scrub queued for path: / 2024-07-26T08:40:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:02 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:02 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:02 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:02 smithi044 ceph-mon[32404]: pgmap v523: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:02 smithi116 ceph-mon[37097]: scrub queued for path: / 2024-07-26T08:40:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:02 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:02 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:02 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:02 smithi116 ceph-mon[37097]: pgmap v523: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:02.562 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T08:40:02.574 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-07-26T08:40:02.574 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T08:40:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:02 smithi171 ceph-mon[37789]: scrub queued for path: / 2024-07-26T08:40:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:02 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:02 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:02 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:02 smithi171 ceph-mon[37789]: pgmap v523: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:40:02.631 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-07-26T08:40:02.632 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-07-26T08:40:02.633 DEBUG:teuthology.orchestra.run.smithi044:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-07-26T08:40:02.701 INFO:tasks.workunit.client.0.smithi044.stderr:+ set -e 2024-07-26T08:40:02.701 INFO:tasks.workunit.client.0.smithi044.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-07-26T08:40:02.707 INFO:tasks.workunit.client.0.smithi044.stderr:Cloning into 'xfstests-dev'... 2024-07-26T08:40:04.422 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:04 smithi044 ceph-mon[32404]: pgmap v524: 129 pgs: 129 active+clean; 1.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 34 KiB/s wr, 27 op/s 2024-07-26T08:40:04.422 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:04 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:40:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:04 smithi116 ceph-mon[37097]: pgmap v524: 129 pgs: 129 active+clean; 1.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 34 KiB/s wr, 27 op/s 2024-07-26T08:40:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:04 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:40:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:04 smithi171 ceph-mon[37789]: pgmap v524: 129 pgs: 129 active+clean; 1.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 34 KiB/s wr, 27 op/s 2024-07-26T08:40:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:04 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:40:04.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 -- 172.21.15.44:0/3433261200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac10f940 msgr2=0x7f21ac111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 --2- 172.21.15.44:0/3433261200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac10f940 0x7f21ac111d80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009be0 tx=0x7f21a402a6c0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 -- 172.21.15.44:0/3433261200 shutdown_connections 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 --2- 172.21.15.44:0/3433261200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac10f940 0x7f21ac111d80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 --2- 172.21.15.44:0/3433261200 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac075b30 0x7f21ac10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 --2- 172.21.15.44:0/3433261200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0751e0 0x7f21ac0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 -- 172.21.15.44:0/3433261200 >> 172.21.15.44:0/3433261200 conn(0x7f21ac0fee70 msgr2=0x7f21ac1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 -- 172.21.15.44:0/3433261200 shutdown_connections 2024-07-26T08:40:04.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.746+0000 7f21abe00640 1 -- 172.21.15.44:0/3433261200 wait complete. 2024-07-26T08:40:04.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.748+0000 7f21abe00640 1 Processor -- start 2024-07-26T08:40:04.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.748+0000 7f21abe00640 1 -- start start 2024-07-26T08:40:04.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21abe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac0751e0 0x7f21ac19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21abe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac075b30 0x7f21ac19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac10f940 0x7f21ac1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21abe00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac1154d0 con 0x7f21ac0751e0 2024-07-26T08:40:04.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21abe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac115350 con 0x7f21ac10f940 2024-07-26T08:40:04.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21abe00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac115650 con 0x7f21ac075b30 2024-07-26T08:40:04.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21aaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac0751e0 0x7f21ac19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21aaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac0751e0 0x7f21ac19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40194/0 (socket says 172.21.15.44:40194) 2024-07-26T08:40:04.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.749+0000 7f21aaa00640 1 -- 172.21.15.44:0/4067436830 learned_addr learned my addr 172.21.15.44:0/4067436830 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:04.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21aa000640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac075b30 0x7f21ac19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21ab400640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac10f940 0x7f21ac1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21aa000640 1 -- 172.21.15.44:0/4067436830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac10f940 msgr2=0x7f21ac1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21aa000640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac10f940 0x7f21ac1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21aa000640 1 -- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac0751e0 msgr2=0x7f21ac19be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21aa000640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac0751e0 0x7f21ac19be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.750+0000 7f21aa000640 1 -- 172.21.15.44:0/4067436830 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac1a1dd0 con 0x7f21ac075b30 2024-07-26T08:40:04.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.751+0000 7f21aa000640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac075b30 0x7f21ac19c400 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f21ac06b130 tx=0x7f219800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:04.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.751+0000 7f21b0a00640 1 -- 172.21.15.44:0/4067436830 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198004440 con 0x7f21ac075b30 2024-07-26T08:40:04.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.751+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac1a20c0 con 0x7f21ac075b30 2024-07-26T08:40:04.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.751+0000 7f21b0a00640 1 -- 172.21.15.44:0/4067436830 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f219800cd70 con 0x7f21ac075b30 2024-07-26T08:40:04.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.751+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac1a8820 con 0x7f21ac075b30 2024-07-26T08:40:04.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.751+0000 7f21b0a00640 1 -- 172.21.15.44:0/4067436830 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198019480 con 0x7f21ac075b30 2024-07-26T08:40:04.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.755+0000 7f21b8a65740 1 Processor -- start 2024-07-26T08:40:04.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.755+0000 7f21b8a65740 1 -- start start 2024-07-26T08:40:04.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210df5070 0x55a211199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 0x55a21119a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a21119aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21b8a65740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a211110b90 con 0x55a210df5070 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21b8a65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a211100090 con 0x55a210dd4390 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21b8a65740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a2111040d0 con 0x55a210dcc1b0 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21aaa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 0x55a21119a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21aaa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 0x55a21119a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:40826/0 (socket says 172.21.15.44:40826) 2024-07-26T08:40:04.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.756+0000 7f21aaa00640 1 -- 172.21.15.44:0/3627345748 learned_addr learned my addr 172.21.15.44:0/3627345748 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:04.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21ab400640 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210df5070 0x55a211199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21b0a00640 1 -- 172.21.15.44:0/4067436830 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f2198019620 con 0x7f21ac075b30 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21aaa00640 1 -- 172.21.15.44:0/3627345748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 msgr2=0x55a21119aa50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21aaa00640 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a21119aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21aaa00640 1 -- 172.21.15.44:0/3627345748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210df5070 msgr2=0x55a211199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21aaa00640 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210df5070 0x55a211199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21aaa00640 1 -- 172.21.15.44:0/3627345748 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a210e60650 con 0x55a210dcc1b0 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.757+0000 7f21aaa00640 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 0x55a21119a510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f219c009dc0 tx=0x7f219c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75374b9cadaef2dd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.758+0000 7f2197e00640 1 -- 172.21.15.44:0/3627345748 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c005220 con 0x55a210dcc1b0 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.758+0000 7f2197e00640 1 -- 172.21.15.44:0/3627345748 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f219c004c50 con 0x55a210dcc1b0 2024-07-26T08:40:04.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.758+0000 7f2197e00640 1 -- 172.21.15.44:0/3627345748 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c019890 con 0x55a210dcc1b0 2024-07-26T08:40:04.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b0a00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21900b3dc0 0x7f21900b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 -- 172.21.15.44:0/3627345748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 msgr2=0x55a21119a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 0x55a21119a510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f219c009dc0 tx=0x7f219c011810 comp rx=0 tx=0).stop 2024-07-26T08:40:04.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 -- 172.21.15.44:0/3627345748 shutdown_connections 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dcc1b0 0x55a21119a510 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a21119aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 --2- 172.21.15.44:0/3627345748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210df5070 0x55a211199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.759+0000 7f21b8a65740 1 -- 172.21.15.44:0/3627345748 >> 172.21.15.44:0/3627345748 conn(0x55a210de9460 msgr2=0x55a211197ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.760+0000 7f21ab400640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21900b3dc0 0x7f21900b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.760+0000 7f21b0a00640 1 -- 172.21.15.44:0/4067436830 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f21980e4620 con 0x7f21ac075b30 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.760+0000 7f21b8a65740 1 -- 172.21.15.44:0/3627345748 shutdown_connections 2024-07-26T08:40:04.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.760+0000 7f21b8a65740 1 -- 172.21.15.44:0/3627345748 wait complete. 2024-07-26T08:40:04.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.761+0000 7f21b8a65740 1 -- start start 2024-07-26T08:40:04.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.761+0000 7f21ab400640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21900b3dc0 0x7f21900b62d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009370 tx=0x7f21a4004a40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:04.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.761+0000 7f21b8a65740 1 Processor -- start 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111aed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a210f63700 con 0x55a210dcc1b0 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a21109c660 con 0x55a210dd4390 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a2111a87b0 con 0x55a210df5070 2024-07-26T08:40:04.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.762+0000 7f21b8a65740 10 client.0 _ll_register_callbacks cb 0x55a2110ac110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:04.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:40836/0 (socket says 172.21.15.44:40836) 2024-07-26T08:40:04.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 -- 172.21.15.44:0/2783662732 learned_addr learned my addr 172.21.15.44:0/2783662732 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:04.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21aa000640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21aaa00640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 -- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 msgr2=0x55a2111aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 -- 172.21.15.44:0/2783662732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 msgr2=0x55a2111ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.763+0000 7f21ab400640 1 -- 172.21.15.44:0/2783662732 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2110b3800 con 0x55a210df5070 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.764+0000 7f21aa000640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.764+0000 7f21ab400640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b3d30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009be0 tx=0x7f21a402a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9a01230f19816f1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:04.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.764+0000 7f2194c00640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:40:04.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.765+0000 7f2194c00640 1 -- 172.21.15.44:0/2783662732 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a404f020 con 0x55a210df5070 2024-07-26T08:40:04.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.765+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f219c004c50 con 0x55a210df5070 2024-07-26T08:40:04.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.765+0000 7f2194c00640 1 -- 172.21.15.44:0/2783662732 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f21a4030970 con 0x55a210df5070 2024-07-26T08:40:04.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.765+0000 7f2194c00640 1 -- 172.21.15.44:0/2783662732 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f21a405a050 con 0x55a210df5070 2024-07-26T08:40:04.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.766+0000 7f21b8a65740 10 client.25663 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:04.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.766+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a2111bccb0 con 0x55a210df5070 2024-07-26T08:40:04.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.766+0000 7f2194c00640 1 -- 172.21.15.44:0/2783662732 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f219c004c50 con 0x55a210df5070 2024-07-26T08:40:04.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f2194c00640 1 -- 172.21.15.44:0/2783662732 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f21a404b480 con 0x55a210df5070 2024-07-26T08:40:04.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21b8a65740 10 client.25663 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:04.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21b8a65740 10 client.25663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:04.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21b8a65740 20 client.25663 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:04.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21b8a65740 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210db39f0 0x55a2111c0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21b8a65740 4 client.25663 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:04.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a211112ae0 con 0x55a210db39f0 2024-07-26T08:40:04.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.767+0000 7f21aaa00640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210db39f0 0x55a2111c0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.768+0000 7f21aaa00640 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210db39f0 0x55a2111c0270 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba44703d5d3ab688 server_cookie=a5cc3bfbefbe021a in_seq=0 out_seq=0 2024-07-26T08:40:04.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.768+0000 7f2194c00640 10 client.25663 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:04.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.769+0000 7f2194c00640 1 -- 172.21.15.44:0/2783662732 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a211112ae0 con 0x55a210db39f0 2024-07-26T08:40:04.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.769+0000 7f2194c00640 10 client.25663 handle_command_reply: tid=0 2024-07-26T08:40:04.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.769+0000 7f21b8a65740 1 client.25663 shutdown 2024-07-26T08:40:04.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.769+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 msgr2=0x55a2111b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.769+0000 7f21b8a65740 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b3d30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009be0 tx=0x7f21a402a6c0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 shutdown_connections 2024-07-26T08:40:04.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210db39f0 0x55a2111c0270 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b3d30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 --2- 172.21.15.44:0/2783662732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 >> 172.21.15.44:0/2783662732 conn(0x55a210de9460 msgr2=0x55a21119d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f2194c00640 0 client.25663 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 shutdown_connections 2024-07-26T08:40:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 1 -- 172.21.15.44:0/2783662732 wait complete. 2024-07-26T08:40:04.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.770+0000 7f21b8a65740 20 client.25663 trim_cache size 0 max 16384 2024-07-26T08:40:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 Processor -- start 2024-07-26T08:40:04.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 -- start start 2024-07-26T08:40:04.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a211196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f218c00ff30 con 0x55a210dcc1b0 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f218c011a70 con 0x55a210dd4390 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.805+0000 7f21b8a65740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f218c010600 con 0x55a210df5070 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.806+0000 7f21aa000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.806+0000 7f21aa000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111a9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:44872/0 (socket says 172.21.15.44:44872) 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.806+0000 7f21aa000640 1 -- 172.21.15.44:0/490378900 learned_addr learned my addr 172.21.15.44:0/490378900 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.806+0000 7f21aaa00640 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a211196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.806+0000 7f21ab400640 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aa000640 1 -- 172.21.15.44:0/490378900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 msgr2=0x55a2111b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aa000640 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aa000640 1 -- 172.21.15.44:0/490378900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 msgr2=0x55a211196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aa000640 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a211196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aa000640 1 -- 172.21.15.44:0/490378900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2111af5d0 con 0x55a210dd4390 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aaa00640 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a211196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.807+0000 7f21aa000640 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111a9a60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f21980a73e0 tx=0x7f21980aa6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0e3ad3d83793e80 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.808+0000 7f2197400640 1 -- 172.21.15.44:0/490378900 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21980c4030 con 0x55a210dd4390 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.808+0000 7f2197400640 1 -- 172.21.15.44:0/490378900 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2198009510 con 0x55a210dd4390 2024-07-26T08:40:04.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.808+0000 7f2197400640 1 -- 172.21.15.44:0/490378900 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21980ad040 con 0x55a210dd4390 2024-07-26T08:40:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 -- 172.21.15.44:0/490378900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 msgr2=0x55a2111a9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111a9a60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f21980a73e0 tx=0x7f21980aa6c0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 -- 172.21.15.44:0/490378900 shutdown_connections 2024-07-26T08:40:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210df5070 0x55a2111b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210dd4390 0x55a2111a9a60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 --2- 172.21.15.44:0/490378900 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a211196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.809+0000 7f21b8a65740 1 -- 172.21.15.44:0/490378900 >> 172.21.15.44:0/490378900 conn(0x55a210de9460 msgr2=0x55a2111afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.810+0000 7f21b8a65740 1 -- 172.21.15.44:0/490378900 shutdown_connections 2024-07-26T08:40:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.810+0000 7f21b8a65740 1 -- 172.21.15.44:0/490378900 wait complete. 2024-07-26T08:40:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.811+0000 7f21b8a65740 1 -- start start 2024-07-26T08:40:04.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.811+0000 7f21b8a65740 1 Processor -- start 2024-07-26T08:40:04.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dd4390 0x55a2111b3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210df5070 0x55a2112269d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f218c0113a0 con 0x55a210dcc1b0 2024-07-26T08:40:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f218c010cd0 con 0x55a210df5070 2024-07-26T08:40:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f218c00df10 con 0x55a210dd4390 2024-07-26T08:40:04.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21b8a65740 10 client.0 _ll_register_callbacks cb 0x7f219c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21aaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21aaa00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40232/0 (socket says 172.21.15.44:40232) 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21aaa00640 1 -- 172.21.15.44:0/330006979 learned_addr learned my addr 172.21.15.44:0/330006979 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.812+0000 7f21aa000640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dd4390 0x55a2111b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21ab400640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210df5070 0x55a2112269d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21aaa00640 1 -- 172.21.15.44:0/330006979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dd4390 msgr2=0x55a2111b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21aaa00640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dd4390 0x55a2111b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21aaa00640 1 -- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210df5070 msgr2=0x55a2112269d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21aaa00640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210df5070 0x55a2112269d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21aaa00640 1 -- 172.21.15.44:0/330006979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2111badb0 con 0x55a210dcc1b0 2024-07-26T08:40:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21aa000640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dd4390 0x55a2111b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:40:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.813+0000 7f21ab400640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210df5070 0x55a2112269d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.814+0000 7f21aaa00640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111b3310 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f219c01b040 tx=0x7f219c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e39c5c04ce718fd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.814+0000 7f2194c00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.814+0000 7f2194c00640 1 -- 172.21.15.44:0/330006979 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c005090 con 0x55a210dcc1b0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.814+0000 7f2194c00640 1 -- 172.21.15.44:0/330006979 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f219c005230 con 0x55a210dcc1b0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.814+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2198009510 con 0x55a210dcc1b0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.815+0000 7f2194c00640 1 -- 172.21.15.44:0/330006979 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f219c039050 con 0x55a210dcc1b0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.815+0000 7f21b8a65740 10 client.15540 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.815+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a211227db0 con 0x55a210dcc1b0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.815+0000 7f2194c00640 1 -- 172.21.15.44:0/330006979 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2198009510 con 0x55a210dcc1b0 2024-07-26T08:40:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f2194c00640 1 -- 172.21.15.44:0/330006979 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f219c0196d0 con 0x55a210dcc1b0 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f21b8a65740 10 client.15540 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f21b8a65740 10 client.15540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f21b8a65740 20 client.15540 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f21b8a65740 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210d88da0 0x55a21122a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f21b8a65740 4 client.15540 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.816+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a211198060 con 0x55a210d88da0 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.817+0000 7f21aa000640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210d88da0 0x55a21122a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:04.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.817+0000 7f21aa000640 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210d88da0 0x55a21122a2e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cc9ff386fff7814 server_cookie=63b5f2df6de1fffb in_seq=0 out_seq=0 2024-07-26T08:40:04.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.817+0000 7f2194c00640 10 client.15540 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.818+0000 7f2194c00640 1 -- 172.21.15.44:0/330006979 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a211198060 con 0x55a210d88da0 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.818+0000 7f2194c00640 10 client.15540 handle_command_reply: tid=0 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.818+0000 7f21b8a65740 1 client.15540 shutdown 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.818+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 msgr2=0x55a2111b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.818+0000 7f21b8a65740 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111b3310 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f219c01b040 tx=0x7f219c0095c0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 shutdown_connections 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x55a210d88da0 0x55a21122a2e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a210dd4390 0x55a2111b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a210df5070 0x55a2112269d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 --2- 172.21.15.44:0/330006979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a210dcc1b0 0x55a2111b3310 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 >> 172.21.15.44:0/330006979 conn(0x55a210de9460 msgr2=0x55a211196c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f2194c00640 0 client.15540 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.819+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 shutdown_connections 2024-07-26T08:40:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.820+0000 7f21b8a65740 1 -- 172.21.15.44:0/330006979 wait complete. 2024-07-26T08:40:04.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.820+0000 7f21b8a65740 20 client.15540 trim_cache size 0 max 16384 2024-07-26T08:40:04.823 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:40:04.823 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-07-26T08:40:04.823 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-07-26T08:40:04.823 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21900b3dc0 msgr2=0x7f21900b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21900b3dc0 0x7f21900b62d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009370 tx=0x7f21a4004a40 comp rx=0 tx=0).stop 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac075b30 msgr2=0x7f21ac19c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac075b30 0x7f21ac19c400 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f21ac06b130 tx=0x7f219800ee50 comp rx=0 tx=0).stop 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 shutdown_connections 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f21900b3dc0 0x7f21900b62d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21ac075b30 0x7f21ac19c400 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac10f940 0x7f21ac1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 --2- 172.21.15.44:0/4067436830 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f21ac0751e0 0x7f21ac19be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:04.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.821+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 >> 172.21.15.44:0/4067436830 conn(0x7f21ac0fee70 msgr2=0x7f21ac1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:04.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.822+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 shutdown_connections 2024-07-26T08:40:04.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:04.822+0000 7f21abe00640 1 -- 172.21.15.44:0/4067436830 wait complete. 2024-07-26T08:40:05.787 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T08:40:05.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T08:40:06.295 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:06.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:06 smithi044 ceph-mon[32404]: pgmap v525: 129 pgs: 129 active+clean; 1.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 24 op/s 2024-07-26T08:40:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:06 smithi116 ceph-mon[37097]: pgmap v525: 129 pgs: 129 active+clean; 1.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 24 op/s 2024-07-26T08:40:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:06 smithi171 ceph-mon[37789]: pgmap v525: 129 pgs: 129 active+clean; 1.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 24 op/s 2024-07-26T08:40:07.481 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:07.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:07.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:08.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:08 smithi171 ceph-mon[37789]: pgmap v526: 129 pgs: 129 active+clean; 11 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 828 KiB/s wr, 26 op/s 2024-07-26T08:40:08.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:08.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:08.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:08 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:40:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:08 smithi044 ceph-mon[32404]: pgmap v526: 129 pgs: 129 active+clean; 11 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 828 KiB/s wr, 26 op/s 2024-07-26T08:40:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:08 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:40:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:08 smithi116 ceph-mon[37097]: pgmap v526: 129 pgs: 129 active+clean; 11 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 828 KiB/s wr, 26 op/s 2024-07-26T08:40:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:08 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 -- 172.21.15.44:0/1348076011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010ba70 msgr2=0x7f42401122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 --2- 172.21.15.44:0/1348076011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010ba70 0x7f42401122a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4238009860 tx=0x7f423802a340 comp rx=0 tx=0).stop 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 -- 172.21.15.44:0/1348076011 shutdown_connections 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 --2- 172.21.15.44:0/1348076011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010ba70 0x7f42401122a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 --2- 172.21.15.44:0/1348076011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010b120 0x7f424010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 --2- 172.21.15.44:0/1348076011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42401057e0 0x7f4240105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 -- 172.21.15.44:0/1348076011 >> 172.21.15.44:0/1348076011 conn(0x7f42401011f0 msgr2=0x7f4240103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:08.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 -- 172.21.15.44:0/1348076011 shutdown_connections 2024-07-26T08:40:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.745+0000 7f4248400640 1 -- 172.21.15.44:0/1348076011 wait complete. 2024-07-26T08:40:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 Processor -- start 2024-07-26T08:40:08.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 -- start start 2024-07-26T08:40:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42401057e0 0x7f424019c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010b120 0x7f424019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 0x7f42401a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4240115720 con 0x7f42401057e0 2024-07-26T08:40:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42401155a0 con 0x7f424010ba70 2024-07-26T08:40:08.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.747+0000 7f4248400640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f42401158a0 con 0x7f424010b120 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 0x7f42401a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 0x7f42401a19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:44900/0 (socket says 172.21.15.44:44900) 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 -- 172.21.15.44:0/2365277638 learned_addr learned my addr 172.21.15.44:0/2365277638 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4245a00640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42401057e0 0x7f424019c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4245000640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010b120 0x7f424019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 -- 172.21.15.44:0/2365277638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010b120 msgr2=0x7f424019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010b120 0x7f424019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 -- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42401057e0 msgr2=0x7f424019c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42401057e0 0x7f424019c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:08.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.748+0000 7f4246400640 1 -- 172.21.15.44:0/2365277638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42401a2120 con 0x7f424010ba70 2024-07-26T08:40:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.749+0000 7f4246400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 0x7f42401a19f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f423802a930 tx=0x7f4238032d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.749+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238042070 con 0x7f424010ba70 2024-07-26T08:40:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.749+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42401a23b0 con 0x7f424010ba70 2024-07-26T08:40:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.749+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4238030d50 con 0x7f424010ba70 2024-07-26T08:40:08.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.749+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238037040 con 0x7f424010ba70 2024-07-26T08:40:08.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.749+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42401a8b70 con 0x7f424010ba70 2024-07-26T08:40:08.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.751+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4240105bf0 con 0x7f424010ba70 2024-07-26T08:40:08.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.753+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f4238045070 con 0x7f424010ba70 2024-07-26T08:40:08.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.754+0000 7f423e000640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42240b3dc0 0x7f42240b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:08.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.755+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4238103900 con 0x7f424010ba70 2024-07-26T08:40:08.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.755+0000 7f4245a00640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42240b3dc0 0x7f42240b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:08.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.756+0000 7f4245a00640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42240b3dc0 0x7f42240b62d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4230005fd0 tx=0x7f4230010040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:08.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.756+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f42380ca470 con 0x7f424010ba70 2024-07-26T08:40:08.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.997+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f424010d860 con 0x7f424010ba70 2024-07-26T08:40:08.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:08.998+0000 7f423e000640 1 -- 172.21.15.44:0/2365277638 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7f42380ce9c0 con 0x7f424010ba70 2024-07-26T08:40:09.000 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-07-26T08:40:09.000 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":30,"btime":"2024-07-26T08:38:32:520674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/2787895500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":2787895500},{"type":"v1","addr":"172.21.15.44:6835","nonce":2787895500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/4084470430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":4084470430},{"type":"v1","addr":"172.21.15.44:6841","nonce":4084470430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T08:33:06.110522+0000","modified":"2024-07-26T08:38:32.022353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24701,"mds_1":24686,"mds_2":24679,"mds_3":24668,"mds_4":24661},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6833/4260884949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":4260884949},{"type":"v1","addr":"172.21.15.171:6833","nonce":4260884949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2097534847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2097534847},{"type":"v1","addr":"172.21.15.116:6833","nonce":2097534847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24638":{"gid":24638,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.44:6837/3938332919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3938332919},{"type":"v1","addr":"172.21.15.44:6837","nonce":3938332919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24650":{"gid":24650,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1052813979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1052813979},{"type":"v1","addr":"172.21.15.116:6835","nonce":1052813979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24655":{"gid":24655,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/372796843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":372796843},{"type":"v1","addr":"172.21.15.171:6835","nonce":372796843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24661":{"gid":24661,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.171:6837/3462225872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":3462225872},{"type":"v1","addr":"172.21.15.171:6837","nonce":3462225872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24668":{"gid":24668,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.116:6837/2715773427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2715773427},{"type":"v1","addr":"172.21.15.116:6837","nonce":2715773427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24679":{"gid":24679,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.171:6839/2065417578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":2065417578},{"type":"v1","addr":"172.21.15.171:6839","nonce":2065417578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":30,"addr":"172.21.15.44:6839/1144072288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1144072288},{"type":"v1","addr":"172.21.15.44:6839","nonce":1144072288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24701":{"gid":24701,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6839/3153223603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":3153223603},{"type":"v1","addr":"172.21.15.116:6839","nonce":3153223603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24701,"qdb_cluster":[24701,24686,24679,24668,24661]},"id":1}]} 2024-07-26T08:40:09.001 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 30 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42240b3dc0 msgr2=0x7f42240b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42240b3dc0 0x7f42240b62d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4230005fd0 tx=0x7f4230010040 comp rx=0 tx=0).stop 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 msgr2=0x7f42401a19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 0x7f42401a19f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f423802a930 tx=0x7f4238032d40 comp rx=0 tx=0).stop 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 shutdown_connections 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f42240b3dc0 0x7f42240b62d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f424010b120 0x7f424019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424010ba70 0x7f42401a19f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 --2- 172.21.15.44:0/2365277638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f42401057e0 0x7f424019c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.001+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 >> 172.21.15.44:0/2365277638 conn(0x7f42401011f0 msgr2=0x7f4240104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.002+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 shutdown_connections 2024-07-26T08:40:09.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:09.002+0000 7f4248400640 1 -- 172.21.15.44:0/2365277638 wait complete. 2024-07-26T08:40:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:09 smithi044 ceph-mon[32404]: from='client.? 172.21.15.44:0/2365277638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:40:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:09 smithi116 ceph-mon[37097]: from='client.? 172.21.15.44:0/2365277638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:40:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:09 smithi171 ceph-mon[37789]: from='client.? 172.21.15.44:0/2365277638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T08:40:09.975 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T08:40:10.490 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:10 smithi044 ceph-mon[32404]: pgmap v527: 129 pgs: 129 active+clean; 11 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 859 KiB/s wr, 29 op/s 2024-07-26T08:40:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:10 smithi116 ceph-mon[37097]: pgmap v527: 129 pgs: 129 active+clean; 11 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 859 KiB/s wr, 29 op/s 2024-07-26T08:40:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:10 smithi171 ceph-mon[37789]: pgmap v527: 129 pgs: 129 active+clean; 11 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 859 KiB/s wr, 29 op/s 2024-07-26T08:40:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:12 smithi044 ceph-mon[32404]: pgmap v528: 129 pgs: 129 active+clean; 15 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-07-26T08:40:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:12 smithi116 ceph-mon[37097]: pgmap v528: 129 pgs: 129 active+clean; 15 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-07-26T08:40:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:12 smithi171 ceph-mon[37789]: pgmap v528: 129 pgs: 129 active+clean; 15 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 1.2 MiB/s wr, 27 op/s 2024-07-26T08:40:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 -- 172.21.15.44:0/3734711011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c810b120 msgr2=0x7f88c810b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 --2- 172.21.15.44:0/3734711011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c810b120 0x7f88c810b530 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009900 tx=0x7f88b402a3f0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 -- 172.21.15.44:0/3734711011 shutdown_connections 2024-07-26T08:40:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 --2- 172.21.15.44:0/3734711011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 0x7f88c81122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 --2- 172.21.15.44:0/3734711011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c810b120 0x7f88c810b530 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 --2- 172.21.15.44:0/3734711011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c81057e0 0x7f88c8105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.935+0000 7f88cf800640 1 -- 172.21.15.44:0/3734711011 >> 172.21.15.44:0/3734711011 conn(0x7f88c81011f0 msgr2=0x7f88c8103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.936+0000 7f88cf800640 1 -- 172.21.15.44:0/3734711011 shutdown_connections 2024-07-26T08:40:12.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.936+0000 7f88cf800640 1 -- 172.21.15.44:0/3734711011 wait complete. 2024-07-26T08:40:12.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.937+0000 7f88cf800640 1 Processor -- start 2024-07-26T08:40:12.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.937+0000 7f88cf800640 1 -- start start 2024-07-26T08:40:12.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cf800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c81057e0 0x7f88c807b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cf800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 0x7f88c8077350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cf800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 0x7f88c8077a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cf800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88c8115700 con 0x7f88c81057e0 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cf800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88c8115580 con 0x7f88c810b120 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cf800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f88c8115880 con 0x7f88c810ba70 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c81057e0 0x7f88c807b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 0x7f88c8077350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 0x7f88c8077350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47386/0 (socket says 172.21.15.44:47386) 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88cd800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 0x7f88c8077a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.938+0000 7f88c7e00640 1 -- 172.21.15.44:0/1036710686 learned_addr learned my addr 172.21.15.44:0/1036710686 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:12.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cd800640 1 -- 172.21.15.44:0/1036710686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 msgr2=0x7f88c8077350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cd800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 0x7f88c8077350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cd800640 1 -- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c81057e0 msgr2=0x7f88c807b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cd800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c81057e0 0x7f88c807b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cd800640 1 -- 172.21.15.44:0/1036710686 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c81ac760 con 0x7f88c810ba70 2024-07-26T08:40:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 0x7f88c8077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:12.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cd800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 0x7f88c8077a70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009fd0 tx=0x7f88c000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88c5600640 1 -- 172.21.15.44:0/1036710686 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c000ce20 con 0x7f88c810ba70 2024-07-26T08:40:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.939+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c81ac990 con 0x7f88c810ba70 2024-07-26T08:40:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.940+0000 7f88c5600640 1 -- 172.21.15.44:0/1036710686 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88c0005020 con 0x7f88c810ba70 2024-07-26T08:40:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.940+0000 7f88c5600640 1 -- 172.21.15.44:0/1036710686 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0010690 con 0x7f88c810ba70 2024-07-26T08:40:12.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.940+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c81acf20 con 0x7f88c810ba70 2024-07-26T08:40:12.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.943+0000 7f88d6632740 1 Processor -- start 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88c5600640 1 -- 172.21.15.44:0/1036710686 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f88c00054d0 con 0x7f88c810ba70 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 -- start start 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15656c070 0x55c156910e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156911370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c1569118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c156887bd0 con 0x55c1565431b0 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c1568770d0 con 0x55c15656c070 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.944+0000 7f88d6632740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c15687b110 con 0x55c15654b390 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156911370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156911370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52374/0 (socket says 172.21.15.44:52374) 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 -- 172.21.15.44:0/1714890417 learned_addr learned my addr 172.21.15.44:0/1714890417 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15656c070 0x55c156910e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c1569118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 -- 172.21.15.44:0/1714890417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 msgr2=0x55c1569118b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c1569118b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 -- 172.21.15.44:0/1714890417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15656c070 msgr2=0x55c156910e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15656c070 0x55c156910e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.945+0000 7f88cce00640 1 -- 172.21.15.44:0/1714890417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1565d7650 con 0x55c1565431b0 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88cce00640 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156911370 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88bc009c90 tx=0x7f88bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195497a69c0ea6e9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88c5600640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f88b00b3d40 0x7f88b00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88ba000640 1 -- 172.21.15.44:0/1714890417 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc004230 con 0x55c1565431b0 2024-07-26T08:40:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88ba000640 1 -- 172.21.15.44:0/1714890417 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88bc0043d0 con 0x55c1565431b0 2024-07-26T08:40:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88ba000640 1 -- 172.21.15.44:0/1714890417 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc019640 con 0x55c1565431b0 2024-07-26T08:40:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88c5600640 1 -- 172.21.15.44:0/1036710686 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f88c00e4a00 con 0x7f88c810ba70 2024-07-26T08:40:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.946+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f88b00b3d40 0x7f88b00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 -- 172.21.15.44:0/1714890417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 msgr2=0x55c156911370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156911370 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88bc009c90 tx=0x7f88bc011810 comp rx=0 tx=0).stop 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f88b00b3d40 0x7f88b00b6250 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88c8103a50 tx=0x7f88b4032f40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 -- 172.21.15.44:0/1714890417 shutdown_connections 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c1569118b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156911370 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 --2- 172.21.15.44:0/1714890417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15656c070 0x55c156910e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 -- 172.21.15.44:0/1714890417 >> 172.21.15.44:0/1714890417 conn(0x55c156560460 msgr2=0x55c15690ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.947+0000 7f88d6632740 1 -- 172.21.15.44:0/1714890417 shutdown_connections 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.948+0000 7f88d6632740 1 -- 172.21.15.44:0/1714890417 wait complete. 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.948+0000 7f88d6632740 1 -- start start 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.948+0000 7f88d6632740 1 Processor -- start 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 0x55c15691c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c15691c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c15656c070 0x55c15691cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c1566da700 con 0x55c15656c070 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c1568136a0 con 0x55c1565431b0 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c15691f610 con 0x55c15654b390 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88d6632740 10 client.0 _ll_register_callbacks cb 0x55c156823150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88cce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 0x55c15691c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88cce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 0x55c15691c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47410/0 (socket says 172.21.15.44:47410) 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88cd800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c15656c070 0x55c15691cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.949+0000 7f88cce00640 1 -- 172.21.15.44:0/1620923553 learned_addr learned my addr 172.21.15.44:0/1620923553 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c15691c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88cce00640 1 -- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 msgr2=0x55c15691c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88cce00640 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c15691c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88cce00640 1 -- 172.21.15.44:0/1620923553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c15656c070 msgr2=0x55c15691cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88cce00640 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c15656c070 0x55c15691cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88cce00640 1 -- 172.21.15.44:0/1620923553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c15682a840 con 0x55c1565431b0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88cce00640 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 0x55c15691c330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0097b0 tx=0x7f88bc003df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8b3fa43fd1af4e6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.950+0000 7f88aea00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.951+0000 7f88aea00640 1 -- 172.21.15.44:0/1620923553 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc028020 con 0x55c1565431b0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.951+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88bc0043d0 con 0x55c1565431b0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.951+0000 7f88aea00640 1 -- 172.21.15.44:0/1620923553 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f88bc0037f0 con 0x55c1565431b0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.951+0000 7f88aea00640 1 -- 172.21.15.44:0/1620923553 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f88bc033050 con 0x55c1565431b0 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.951+0000 7f88d6632740 10 client.25682 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:12.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.951+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c15692b170 con 0x55c1565431b0 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88aea00640 1 -- 172.21.15.44:0/1620923553 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88bc0043d0 con 0x55c1565431b0 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88aea00640 1 -- 172.21.15.44:0/1620923553 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f88bc018070 con 0x55c1565431b0 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88d6632740 10 client.25682 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88d6632740 10 client.25682 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88d6632740 20 client.25682 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88d6632740 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c15652a9f0 0x55c15692e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88d6632740 4 client.25682 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.952+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c156889b20 con 0x55c15652a9f0 2024-07-26T08:40:12.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.953+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c15652a9f0 0x55c15692e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.953+0000 7f88c7e00640 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c15652a9f0 0x55c15692e4d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3f93d8de08b75055 server_cookie=63b0b0e2bb98b779 in_seq=0 out_seq=0 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.953+0000 7f88aea00640 10 client.25682 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.954+0000 7f88aea00640 1 -- 172.21.15.44:0/1620923553 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c156889b20 con 0x55c15652a9f0 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.954+0000 7f88aea00640 10 client.25682 handle_command_reply: tid=0 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.954+0000 7f88d6632740 1 client.25682 shutdown 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 msgr2=0x55c15691c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 0x55c15691c330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0097b0 tx=0x7f88bc003df0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 shutdown_connections 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15654b390 0x55c15691c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c15652a9f0 0x55c15692e4d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c15656c070 0x55c15691cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 --2- 172.21.15.44:0/1620923553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1565431b0 0x55c15691c330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88aea00640 0 client.25682 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 >> 172.21.15.44:0/1620923553 conn(0x55c156560460 msgr2=0x55c156913fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 shutdown_connections 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 1 -- 172.21.15.44:0/1620923553 wait complete. 2024-07-26T08:40:12.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.955+0000 7f88d6632740 20 client.25682 trim_cache size 0 max 16384 2024-07-26T08:40:12.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 Processor -- start 2024-07-26T08:40:12.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 -- start start 2024-07-26T08:40:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156912930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c156933c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c1569341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88c81b7030 con 0x55c1565431b0 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88c81b8b70 con 0x55c15654b390 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.970+0000 7f88d6632740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f88c81b7700 con 0x55c15656c070 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156912930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c156933c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156912930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52388/0 (socket says 172.21.15.44:52388) 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88cce00640 1 -- 172.21.15.44:0/4165277558 learned_addr learned my addr 172.21.15.44:0/4165277558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88cd800640 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c1569341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88c7e00640 1 -- 172.21.15.44:0/4165277558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 msgr2=0x55c1569341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88c7e00640 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c1569341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88c7e00640 1 -- 172.21.15.44:0/4165277558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 msgr2=0x55c156912930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88c7e00640 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156912930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.971+0000 7f88c7e00640 1 -- 172.21.15.44:0/4165277558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c15690a1b0 con 0x55c15654b390 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.972+0000 7f88c7e00640 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c156933c70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f88b4031440 tx=0x7f88b4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8767a26bac144c02 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.972+0000 7f88b9600640 1 -- 172.21.15.44:0/4165277558 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4046070 con 0x55c15654b390 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.972+0000 7f88b9600640 1 -- 172.21.15.44:0/4165277558 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88b40044c0 con 0x55c15654b390 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.972+0000 7f88b9600640 1 -- 172.21.15.44:0/4165277558 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4035040 con 0x55c15654b390 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 -- 172.21.15.44:0/4165277558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 msgr2=0x55c156933c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c156933c70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f88b4031440 tx=0x7f88b4009260 comp rx=0 tx=0).stop 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 -- 172.21.15.44:0/4165277558 shutdown_connections 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c1569341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c156933c70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 --2- 172.21.15.44:0/4165277558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156912930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 -- 172.21.15.44:0/4165277558 >> 172.21.15.44:0/4165277558 conn(0x55c156560460 msgr2=0x55c15691e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 -- 172.21.15.44:0/4165277558 shutdown_connections 2024-07-26T08:40:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 -- 172.21.15.44:0/4165277558 wait complete. 2024-07-26T08:40:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.973+0000 7f88d6632740 1 -- start start 2024-07-26T08:40:12.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 Processor -- start 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156924ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c1569203d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c156920910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88c81b84a0 con 0x55c1565431b0 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88c81b7dd0 con 0x55c15654b390 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f88c81b5010 con 0x55c15656c070 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.974+0000 7f88d6632740 10 client.0 _ll_register_callbacks cb 0x7f88bc009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156924ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cd800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c156920910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156924ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52392/0 (socket says 172.21.15.44:52392) 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 -- 172.21.15.44:0/428398561 learned_addr learned my addr 172.21.15.44:0/428398561 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88c7e00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c1569203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 -- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 msgr2=0x55c156920910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c156920910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 -- 172.21.15.44:0/428398561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 msgr2=0x55c1569203d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c1569203d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cce00640 1 -- 172.21.15.44:0/428398561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c156920fd0 con 0x55c1565431b0 2024-07-26T08:40:12.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88cd800640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c156920910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:12.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.975+0000 7f88c7e00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c1569203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:12.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88cce00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156924ed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88bc004ca0 tx=0x7f88bc009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc3ed65d11f29e7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:12.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88aea00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:12.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88aea00640 1 -- 172.21.15.44:0/428398561 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc03b070 con 0x55c1565431b0 2024-07-26T08:40:12.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88aea00640 1 -- 172.21.15.44:0/428398561 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f88bc0043d0 con 0x55c1565431b0 2024-07-26T08:40:12.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c1568855c0 con 0x55c1565431b0 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88aea00640 1 -- 172.21.15.44:0/428398561 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f88bc044050 con 0x55c1565431b0 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88d6632740 10 client.15564 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.976+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c156996ab0 con 0x55c1565431b0 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88aea00640 1 -- 172.21.15.44:0/428398561 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c1568855c0 con 0x55c1565431b0 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88aea00640 1 -- 172.21.15.44:0/428398561 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f88bc03e020 con 0x55c1565431b0 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88d6632740 10 client.15564 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88d6632740 10 client.15564 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88d6632740 20 client.15564 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88d6632740 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c1564ffda0 0x55c156999200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88d6632740 4 client.15564 mds_command: new command op to 24661 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:40:12.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.977+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 --> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c156882d00 con 0x55c1564ffda0 2024-07-26T08:40:12.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.978+0000 7f88c7e00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c1564ffda0 0x55c156999200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:12.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.978+0000 7f88c7e00640 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c1564ffda0 0x55c156999200 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=30bffc3e55939adf server_cookie=96a65fffaf3a2613 in_seq=0 out_seq=0 2024-07-26T08:40:12.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.978+0000 7f88aea00640 10 client.15564 ms_handle_connect on v2:172.21.15.171:6836/3462225872 2024-07-26T08:40:12.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.978+0000 7f88aea00640 1 -- 172.21.15.44:0/428398561 <== mds.4 v2:172.21.15.171:6836/3462225872 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c156882d00 con 0x55c1564ffda0 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.978+0000 7f88aea00640 10 client.15564 handle_command_reply: tid=0 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.978+0000 7f88d6632740 1 client.15564 shutdown 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 msgr2=0x55c156924ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156924ed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88bc004ca0 tx=0x7f88bc009400 comp rx=0 tx=0).stop 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 shutdown_connections 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c15656c070 0x55c156920910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.171:6836/3462225872,v1:172.21.15.171:6837/3462225872] conn(0x55c1564ffda0 0x55c156999200 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c15654b390 0x55c1569203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 --2- 172.21.15.44:0/428398561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c1565431b0 0x55c156924ed0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 >> 172.21.15.44:0/428398561 conn(0x55c156560460 msgr2=0x55c15691d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88aea00640 0 client.15564 ms_handle_reset on v2:172.21.15.171:6836/3462225872 2024-07-26T08:40:12.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 shutdown_connections 2024-07-26T08:40:12.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 1 -- 172.21.15.44:0/428398561 wait complete. 2024-07-26T08:40:12.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.979+0000 7f88d6632740 20 client.15564 trim_cache size 0 max 16384 2024-07-26T08:40:12.985 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f88b00b3d40 msgr2=0x7f88b00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f88b00b3d40 0x7f88b00b6250 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88c8103a50 tx=0x7f88b4032f40 comp rx=0 tx=0).stop 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 msgr2=0x7f88c8077a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 0x7f88c8077a70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009fd0 tx=0x7f88c000ef10 comp rx=0 tx=0).stop 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 shutdown_connections 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f88b00b3d40 0x7f88b00b6250 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88c810ba70 0x7f88c8077a70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88c810b120 0x7f88c8077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 --2- 172.21.15.44:0/1036710686 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88c81057e0 0x7f88c807b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 >> 172.21.15.44:0/1036710686 conn(0x7f88c81011f0 msgr2=0x7f88c81098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 shutdown_connections 2024-07-26T08:40:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:12.980+0000 7f88cf800640 1 -- 172.21.15.44:0/1036710686 wait complete. 2024-07-26T08:40:14.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T08:40:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:14 smithi044 ceph-mon[32404]: pgmap v529: 129 pgs: 129 active+clean; 16 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-07-26T08:40:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:14 smithi116 ceph-mon[37097]: pgmap v529: 129 pgs: 129 active+clean; 16 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-07-26T08:40:14.532 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:14 smithi171 ceph-mon[37789]: pgmap v529: 129 pgs: 129 active+clean; 16 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-07-26T08:40:16.510 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:16 smithi044 ceph-mon[32404]: pgmap v530: 129 pgs: 129 active+clean; 16 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-07-26T08:40:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:16 smithi116 ceph-mon[37097]: pgmap v530: 129 pgs: 129 active+clean; 16 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-07-26T08:40:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:16 smithi171 ceph-mon[37789]: pgmap v530: 129 pgs: 129 active+clean; 16 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 1.2 MiB/s wr, 15 op/s 2024-07-26T08:40:16.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.898+0000 7f291ca00640 1 -- 172.21.15.44:0/749280764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2918102510 msgr2=0x7f2918102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.898+0000 7f291ca00640 1 --2- 172.21.15.44:0/749280764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2918102510 0x7f2918102920 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2908009900 tx=0x7f290802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 -- 172.21.15.44:0/749280764 shutdown_connections 2024-07-26T08:40:16.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 --2- 172.21.15.44:0/749280764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29181087a0 0x7f291810efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 --2- 172.21.15.44:0/749280764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918107e50 0x7f2918108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 --2- 172.21.15.44:0/749280764 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2918102510 0x7f2918102920 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 -- 172.21.15.44:0/749280764 >> 172.21.15.44:0/749280764 conn(0x7f29180790e0 msgr2=0x7f29180794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:16.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 -- 172.21.15.44:0/749280764 shutdown_connections 2024-07-26T08:40:16.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.899+0000 7f291ca00640 1 -- 172.21.15.44:0/749280764 wait complete. 2024-07-26T08:40:16.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.900+0000 7f291ca00640 1 Processor -- start 2024-07-26T08:40:16.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.900+0000 7f291ca00640 1 -- start start 2024-07-26T08:40:16.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f291ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918102510 0x7f2918199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f291ca00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2918107e50 0x7f291819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f291ca00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 0x7f291819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f291ca00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29181135b0 con 0x7f29181087a0 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f291ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2918113430 con 0x7f2918102510 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f291ca00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2918113730 con 0x7f2918107e50 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f2916800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 0x7f291819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f2915e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918102510 0x7f2918199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f2916800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 0x7f291819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52408/0 (socket says 172.21.15.44:52408) 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.901+0000 7f2916800640 1 -- 172.21.15.44:0/3404942731 learned_addr learned my addr 172.21.15.44:0/3404942731 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f290fe00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2918107e50 0x7f291819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2915e00640 1 -- 172.21.15.44:0/3404942731 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2918107e50 msgr2=0x7f291819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2915e00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2918107e50 0x7f291819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2915e00640 1 -- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 msgr2=0x7f291819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2915e00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 0x7f291819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2915e00640 1 -- 172.21.15.44:0/3404942731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291819fe90 con 0x7f2918102510 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2916800640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 0x7f291819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.902+0000 7f2915e00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918102510 0x7f2918199f20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f290802a9e0 tx=0x7f290803ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.903+0000 7f290ea00640 1 -- 172.21.15.44:0/3404942731 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908035070 con 0x7f2918102510 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.903+0000 7f290ea00640 1 -- 172.21.15.44:0/3404942731 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2908003cb0 con 0x7f2918102510 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.903+0000 7f290ea00640 1 -- 172.21.15.44:0/3404942731 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908031d70 con 0x7f2918102510 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.903+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29181a0120 con 0x7f2918102510 2024-07-26T08:40:16.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.903+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29181a0580 con 0x7f2918102510 2024-07-26T08:40:16.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.906+0000 7f2923818740 1 Processor -- start 2024-07-26T08:40:16.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 -- start start 2024-07-26T08:40:16.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a1884e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1885370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18858b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5568a17fbbd0 con 0x5568a14b71b0 2024-07-26T08:40:16.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568a17eb0d0 con 0x5568a14bf390 2024-07-26T08:40:16.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2923818740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5568a17ef110 con 0x5568a14e0070 2024-07-26T08:40:16.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2916800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a1884e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2916800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a1884e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52212/0 (socket says 172.21.15.44:52212) 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.907+0000 7f2916800640 1 -- 172.21.15.44:0/3807603561 learned_addr learned my addr 172.21.15.44:0/3807603561 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f290fe00640 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18858b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f290ea00640 1 -- 172.21.15.44:0/3404942731 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f29080314a0 con 0x7f2918102510 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f2916800640 1 -- 172.21.15.44:0/3807603561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 msgr2=0x5568a18858b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f2916800640 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18858b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f2916800640 1 -- 172.21.15.44:0/3807603561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 msgr2=0x5568a1885370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f2916800640 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1885370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.908+0000 7f2916800640 1 -- 172.21.15.44:0/3807603561 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568a154b650 con 0x5568a14e0070 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.909+0000 7f2916800640 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a1884e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29100118b0 tx=0x7f2910011d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22dcb1c502c3d901 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.909+0000 7f2907000640 1 -- 172.21.15.44:0/3807603561 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291001e070 con 0x5568a14e0070 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.909+0000 7f2907000640 1 -- 172.21.15.44:0/3807603561 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f291000fe70 con 0x5568a14e0070 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.909+0000 7f2907000640 1 -- 172.21.15.44:0/3807603561 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910019da0 con 0x5568a14e0070 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f290ea00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f28fc0b3dc0 0x7f28fc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 -- 172.21.15.44:0/3807603561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 msgr2=0x5568a1884e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a1884e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29100118b0 tx=0x7f2910011d70 comp rx=0 tx=0).stop 2024-07-26T08:40:16.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f290ea00640 1 -- 172.21.15.44:0/3404942731 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f2908108550 con 0x7f2918102510 2024-07-26T08:40:16.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 -- 172.21.15.44:0/3807603561 shutdown_connections 2024-07-26T08:40:16.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18858b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1885370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 --2- 172.21.15.44:0/3807603561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a1884e30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 -- 172.21.15.44:0/3807603561 >> 172.21.15.44:0/3807603561 conn(0x5568a14d4460 msgr2=0x5568a1882d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 -- 172.21.15.44:0/3807603561 shutdown_connections 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2915e00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f28fc0b3dc0 0x7f28fc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.910+0000 7f2923818740 1 -- 172.21.15.44:0/3807603561 wait complete. 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.911+0000 7f2923818740 1 -- start start 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.911+0000 7f2915e00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f28fc0b3dc0 0x7f28fc0b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x5568a1883ab0 tx=0x7f290802a3f0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:16.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.911+0000 7f2923818740 1 Processor -- start 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1890360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 0x5568a18908a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1890de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5568a164e700 con 0x5568a14b71b0 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568a17876a0 con 0x5568a14e0070 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5568a1893610 con 0x5568a14bf390 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2923818740 10 client.0 _ll_register_callbacks cb 0x5568a1797150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2916800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1890360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f290fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 0x5568a18908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f290fe00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 0x5568a18908a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:52220/0 (socket says 172.21.15.44:52220) 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f290fe00640 1 -- 172.21.15.44:0/405235132 learned_addr learned my addr 172.21.15.44:0/405235132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.912+0000 7f2916800640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1890de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f290fe00640 1 -- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 msgr2=0x5568a1890de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f290fe00640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1890de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f290fe00640 1 -- 172.21.15.44:0/405235132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 msgr2=0x5568a1890360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f290fe00640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1890360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f290fe00640 1 -- 172.21.15.44:0/405235132 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568a179e840 con 0x5568a14bf390 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f2916800640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1890360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f290fe00640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 0x5568a18908a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2900012700 tx=0x7f2900012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a6f3795ad98408a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f28fb400640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f28fb400640 1 -- 172.21.15.44:0/405235132 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290000f920 con 0x5568a14bf390 2024-07-26T08:40:16.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f291000fe70 con 0x5568a14bf390 2024-07-26T08:40:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.913+0000 7f28fb400640 1 -- 172.21.15.44:0/405235132 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f290000fac0 con 0x5568a14bf390 2024-07-26T08:40:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.914+0000 7f28fb400640 1 -- 172.21.15.44:0/405235132 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f290000fcc0 con 0x5568a14bf390 2024-07-26T08:40:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.914+0000 7f2923818740 10 client.25705 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.914+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5568a189f210 con 0x5568a14bf390 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.914+0000 7f28fb400640 1 -- 172.21.15.44:0/405235132 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f291000fe70 con 0x5568a14bf390 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.914+0000 7f28fb400640 1 -- 172.21.15.44:0/405235132 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f29000052f0 con 0x5568a14bf390 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2923818740 10 client.25705 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2923818740 10 client.25705 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2923818740 20 client.25705 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2923818740 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a149e9f0 0x5568a18a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2923818740 4 client.25705 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568a17fdb20 con 0x5568a149e9f0 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.915+0000 7f2916800640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a149e9f0 0x5568a18a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.916+0000 7f2916800640 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a149e9f0 0x5568a18a2780 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=436544c24d7671ce server_cookie=44aa67cf0c623bde in_seq=0 out_seq=0 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.916+0000 7f28fb400640 10 client.25705 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:40:16.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.916+0000 7f28fb400640 1 -- 172.21.15.44:0/405235132 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568a17fdb20 con 0x5568a149e9f0 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.916+0000 7f28fb400640 10 client.25705 handle_command_reply: tid=0 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 client.25705 shutdown 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 msgr2=0x5568a18908a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 0x5568a18908a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2900012700 tx=0x7f2900012bc0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 shutdown_connections 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a149e9f0 0x5568a18a2780 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14bf390 0x5568a18908a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1890de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 --2- 172.21.15.44:0/405235132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a1890360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f28fb400640 0 client.25705 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 >> 172.21.15.44:0/405235132 conn(0x5568a14d4460 msgr2=0x5568a1887fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 shutdown_connections 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 1 -- 172.21.15.44:0/405235132 wait complete. 2024-07-26T08:40:16.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.917+0000 7f2923818740 20 client.25705 trim_cache size 0 max 16384 2024-07-26T08:40:16.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.932+0000 7f2923818740 1 Processor -- start 2024-07-26T08:40:16.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.932+0000 7f2923818740 1 -- start start 2024-07-26T08:40:16.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2923818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a189c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2923818740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2923818740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a18a8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2923818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29181ae6c0 con 0x5568a14b71b0 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2923818740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29181b0200 con 0x5568a14bf390 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2923818740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f29181aed90 con 0x5568a14e0070 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2916800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a189c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2916800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a189c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52426/0 (socket says 172.21.15.44:52426) 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f2916800640 1 -- 172.21.15.44:0/2508149837 learned_addr learned my addr 172.21.15.44:0/2508149837 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:16.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.933+0000 7f290fe00640 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f2916800640 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a18a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f290fe00640 1 -- 172.21.15.44:0/2508149837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 msgr2=0x5568a18a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f290fe00640 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a18a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f290fe00640 1 -- 172.21.15.44:0/2508149837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 msgr2=0x5568a189c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f290fe00640 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a189c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f290fe00640 1 -- 172.21.15.44:0/2508149837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568a1887120 con 0x5568a14bf390 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f2916800640 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a189c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f290fe00640 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18a7f40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2900010430 tx=0x7f2900010530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290b307a170b1eaa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.934+0000 7f2906600640 1 -- 172.21.15.44:0/2508149837 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290000f920 con 0x5568a14bf390 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2906600640 1 -- 172.21.15.44:0/2508149837 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f290000fac0 con 0x5568a14bf390 2024-07-26T08:40:16.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2906600640 1 -- 172.21.15.44:0/2508149837 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900003850 con 0x5568a14bf390 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 -- 172.21.15.44:0/2508149837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 msgr2=0x5568a18a7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18a7f40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2900010430 tx=0x7f2900010530 comp rx=0 tx=0).stop 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 -- 172.21.15.44:0/2508149837 shutdown_connections 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14e0070 0x5568a18a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14bf390 0x5568a18a7f40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 --2- 172.21.15.44:0/2508149837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14b71b0 0x5568a189c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.935+0000 7f2923818740 1 -- 172.21.15.44:0/2508149837 >> 172.21.15.44:0/2508149837 conn(0x5568a14d4460 msgr2=0x5568a188b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.936+0000 7f2923818740 1 -- 172.21.15.44:0/2508149837 shutdown_connections 2024-07-26T08:40:16.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.936+0000 7f2923818740 1 -- 172.21.15.44:0/2508149837 wait complete. 2024-07-26T08:40:16.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.936+0000 7f2923818740 1 -- start start 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.936+0000 7f2923818740 1 Processor -- start 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14b71b0 0x5568a1898c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 0x5568a18940e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1894620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f29181afb30 con 0x5568a14bf390 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29181af460 con 0x5568a14e0070 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f29181ac6a0 con 0x5568a14b71b0 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2923818740 10 client.0 _ll_register_callbacks cb 0x7f2900009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f290fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 0x5568a18940e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f290fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 0x5568a18940e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52428/0 (socket says 172.21.15.44:52428) 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f290fe00640 1 -- 172.21.15.44:0/2932872617 learned_addr learned my addr 172.21.15.44:0/2932872617 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:16.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2916800640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1894620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.937+0000 7f2916800640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14b71b0 0x5568a1898c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f290fe00640 1 -- 172.21.15.44:0/2932872617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14b71b0 msgr2=0x5568a1898c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f290fe00640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14b71b0 0x5568a1898c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f290fe00640 1 -- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 msgr2=0x5568a1894620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f290fe00640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1894620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f290fe00640 1 -- 172.21.15.44:0/2932872617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568a1894b60 con 0x5568a14bf390 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f2916800640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14b71b0 0x5568a1898c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f290fe00640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 0x5568a18940e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2900005710 tx=0x7f2900005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e5435af9bfd320 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f28fb400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f28fb400640 1 -- 172.21.15.44:0/2932872617 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290000f920 con 0x5568a14bf390 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f28fb400640 1 -- 172.21.15.44:0/2932872617 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f29000109f0 con 0x5568a14bf390 2024-07-26T08:40:16.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.938+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f290000fac0 con 0x5568a14bf390 2024-07-26T08:40:16.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.939+0000 7f28fb400640 1 -- 172.21.15.44:0/2932872617 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f290003c050 con 0x5568a14bf390 2024-07-26T08:40:16.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.939+0000 7f2923818740 10 client.15579 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:16.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.939+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5568a190ab10 con 0x5568a14bf390 2024-07-26T08:40:16.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.939+0000 7f28fb400640 1 -- 172.21.15.44:0/2932872617 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f290000fac0 con 0x5568a14bf390 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.939+0000 7f28fb400640 1 -- 172.21.15.44:0/2932872617 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f2900033440 con 0x5568a14bf390 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2923818740 10 client.15579 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2923818740 10 client.15579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2923818740 20 client.15579 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2923818740 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a1473da0 0x5568a190d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2923818740 4 client.15579 mds_command: new command op to 24668 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 --> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568a17f95c0 con 0x5568a1473da0 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2916800640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a1473da0 0x5568a190d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f2916800640 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a1473da0 0x5568a190d120 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=556c417816ccb2e8 server_cookie=bc7bc25a503d4c30 in_seq=0 out_seq=0 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.940+0000 7f28fb400640 10 client.15579 ms_handle_connect on v2:172.21.15.116:6836/2715773427 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f28fb400640 1 -- 172.21.15.44:0/2932872617 <== mds.3 v2:172.21.15.116:6836/2715773427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568a17f95c0 con 0x5568a1473da0 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f28fb400640 10 client.15579 handle_command_reply: tid=0 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 client.15579 shutdown 2024-07-26T08:40:16.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 msgr2=0x5568a18940e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 0x5568a18940e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2900005710 tx=0x7f2900005300 comp rx=0 tx=0).stop 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 shutdown_connections 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:6836/2715773427,v1:172.21.15.116:6837/2715773427] conn(0x5568a1473da0 0x5568a190d120 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568a14e0070 0x5568a1894620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5568a14bf390 0x5568a18940e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 --2- 172.21.15.44:0/2932872617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5568a14b71b0 0x5568a1898c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 >> 172.21.15.44:0/2932872617 conn(0x5568a14d4460 msgr2=0x5568a188ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f28fb400640 0 client.15579 ms_handle_reset on v2:172.21.15.116:6836/2715773427 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 shutdown_connections 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 1 -- 172.21.15.44:0/2932872617 wait complete. 2024-07-26T08:40:16.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.941+0000 7f2923818740 20 client.15579 trim_cache size 0 max 16384 2024-07-26T08:40:16.946 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:40:16.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f28fc0b3dc0 msgr2=0x7f28fc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f28fc0b3dc0 0x7f28fc0b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x5568a1883ab0 tx=0x7f290802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918102510 msgr2=0x7f2918199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:16.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918102510 0x7f2918199f20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f290802a9e0 tx=0x7f290803ac10 comp rx=0 tx=0).stop 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 shutdown_connections 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f28fc0b3dc0 0x7f28fc0b62d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2918107e50 0x7f291819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f29181087a0 0x7f291819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 --2- 172.21.15.44:0/3404942731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2918102510 0x7f2918199f20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 >> 172.21.15.44:0/3404942731 conn(0x7f29180790e0 msgr2=0x7f2918105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.942+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 shutdown_connections 2024-07-26T08:40:16.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:16.943+0000 7f291ca00640 1 -- 172.21.15.44:0/3404942731 wait complete. 2024-07-26T08:40:17.899 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T08:40:18.445 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:18 smithi044 ceph-mon[32404]: pgmap v531: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-26T08:40:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:18 smithi116 ceph-mon[37097]: pgmap v531: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-26T08:40:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:18 smithi171 ceph-mon[37789]: pgmap v531: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-26T08:40:20.498 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:20 smithi044 ceph-mon[32404]: pgmap v532: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 495 KiB/s wr, 18 op/s 2024-07-26T08:40:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:20 smithi116 ceph-mon[37097]: pgmap v532: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 495 KiB/s wr, 18 op/s 2024-07-26T08:40:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:20 smithi171 ceph-mon[37789]: pgmap v532: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 495 KiB/s wr, 18 op/s 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 -- 172.21.15.44:0/2467909966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 msgr2=0x7f9e981059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2467909966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e981059e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009900 tx=0x7f9e8802a410 comp rx=0 tx=0).stop 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 -- 172.21.15.44:0/2467909966 shutdown_connections 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2467909966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 0x7f9e98112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2467909966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 0x7f9e9810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2467909966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e981059e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 -- 172.21.15.44:0/2467909966 >> 172.21.15.44:0/2467909966 conn(0x7f9e98101040 msgr2=0x7f9e98103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:20.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 -- 172.21.15.44:0/2467909966 shutdown_connections 2024-07-26T08:40:20.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.870+0000 7f9e97e00640 1 -- 172.21.15.44:0/2467909966 wait complete. 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 Processor -- start 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 -- start start 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e9819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 0x7f9e9819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 0x7f9e981a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e98115560 con 0x7f9e9810af10 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981153e0 con 0x7f9e981055d0 2024-07-26T08:40:20.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.872+0000 7f9e97e00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981156e0 con 0x7f9e9810b860 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e9819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e9819bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47502/0 (socket says 172.21.15.44:47502) 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 -- 172.21.15.44:0/2116006282 learned_addr learned my addr 172.21.15.44:0/2116006282 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 0x7f9e9819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e97400640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 0x7f9e981a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 -- 172.21.15.44:0/2116006282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 msgr2=0x7f9e981a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 0x7f9e981a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 -- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 msgr2=0x7f9e9819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 0x7f9e9819c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e96a00640 1 -- 172.21.15.44:0/2116006282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e981a1f00 con 0x7f9e981055d0 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e97400640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 0x7f9e981a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.873+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 0x7f9e9819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:20.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.874+0000 7f9e96a00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e9819bf90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9e88005ab0 tx=0x7f9e88003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.874+0000 7f9e8f400640 1 -- 172.21.15.44:0/2116006282 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8803f070 con 0x7f9e981055d0 2024-07-26T08:40:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.874+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e981a2190 con 0x7f9e981055d0 2024-07-26T08:40:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.874+0000 7f9e8f400640 1 -- 172.21.15.44:0/2116006282 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e88031b60 con 0x7f9e981055d0 2024-07-26T08:40:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.874+0000 7f9e8f400640 1 -- 172.21.15.44:0/2116006282 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88035030 con 0x7f9e981055d0 2024-07-26T08:40:20.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.874+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e981a8950 con 0x7f9e981055d0 2024-07-26T08:40:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 Processor -- start 2024-07-26T08:40:20.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 -- start start 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 0x556d0ad48010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a97a1b0 0x556d0ad48550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad48a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556d0acbebd0 con 0x556d0a982390 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d0acae0d0 con 0x556d0a9a3070 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556d0acb2110 con 0x556d0a97a1b0 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.878+0000 7f9e8f400640 1 -- 172.21.15.44:0/2116006282 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f9e88042070 con 0x7f9e981055d0 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 0x556d0ad48010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 0x556d0ad48010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47512/0 (socket says 172.21.15.44:47512) 2024-07-26T08:40:20.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 -- 172.21.15.44:0/2877079843 learned_addr learned my addr 172.21.15.44:0/2877079843 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:20.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e96a00640 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a97a1b0 0x556d0ad48550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad48a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 -- 172.21.15.44:0/2877079843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a97a1b0 msgr2=0x556d0ad48550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a97a1b0 0x556d0ad48550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 -- 172.21.15.44:0/2877079843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 msgr2=0x556d0ad48a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad48a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e97400640 1 -- 172.21.15.44:0/2877079843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0aa0e650 con 0x556d0a9a3070 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.879+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad48a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.880+0000 7f9e97400640 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 0x556d0ad48010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9e90011790 tx=0x7f9e90011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f3ee60c9804f5a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.880+0000 7f9e87e00640 1 -- 172.21.15.44:0/2877079843 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90003c30 con 0x556d0a9a3070 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.880+0000 7f9e87e00640 1 -- 172.21.15.44:0/2877079843 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e9000fe50 con 0x556d0a9a3070 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.880+0000 7f9e87e00640 1 -- 172.21.15.44:0/2877079843 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900193e0 con 0x556d0a9a3070 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9e8f400640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9e7c0b3d40 0x7f9e7c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9e8f400640 1 -- 172.21.15.44:0/2116006282 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9e88104740 con 0x7f9e981055d0 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9e96a00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9e7c0b3d40 0x7f9e7c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2877079843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 msgr2=0x556d0ad48010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 0x556d0ad48010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9e90011790 tx=0x7f9e90011c50 comp rx=0 tx=0).stop 2024-07-26T08:40:20.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2877079843 shutdown_connections 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a97a1b0 0x556d0ad48550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad48a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2877079843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a9a3070 0x556d0ad48010 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2877079843 >> 172.21.15.44:0/2877079843 conn(0x556d0a997460 msgr2=0x556d0ad45f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.881+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2877079843 shutdown_connections 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.882+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2877079843 wait complete. 2024-07-26T08:40:20.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.882+0000 7f9e96a00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9e7c0b3d40 0x7f9e7c0b6250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556d0ad46c90 tx=0x7f9e88009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.882+0000 7f9ea4a2a740 1 -- start start 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.882+0000 7f9ea4a2a740 1 Processor -- start 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad53580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad53ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad54000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556d0ab11700 con 0x556d0a97a1b0 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d0ac4a6a0 con 0x556d0a982390 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556d0ad567f0 con 0x556d0a9a3070 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9ea4a2a740 10 client.0 _ll_register_callbacks cb 0x556d0ac5a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9e97400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad53580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9e8fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad53ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9e8fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad53ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47524/0 (socket says 172.21.15.44:47524) 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.883+0000 7f9e8fe00640 1 -- 172.21.15.44:0/3095973869 learned_addr learned my addr 172.21.15.44:0/3095973869 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e97400640 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad54000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e8fe00640 1 -- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 msgr2=0x556d0ad54000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad54000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e8fe00640 1 -- 172.21.15.44:0/3095973869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 msgr2=0x556d0ad53580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad53580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e8fe00640 1 -- 172.21.15.44:0/3095973869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0ac61840 con 0x556d0a982390 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad53ac0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e800139f0 tx=0x7f9e80013eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83f319198a492bd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.884+0000 7f9e84c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9e84c00640 1 -- 172.21.15.44:0/3095973869 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80011cf0 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9e84c00640 1 -- 172.21.15.44:0/3095973869 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9e80004020 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e9000fe50 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9e84c00640 1 -- 172.21.15.44:0/3095973869 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9e80026050 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9ea4a2a740 10 client.25718 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556d0ad62440 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.885+0000 7f9e84c00640 1 -- 172.21.15.44:0/3095973869 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e9000fe50 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9e84c00640 1 -- 172.21.15.44:0/3095973869 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9e8001b410 con 0x556d0a982390 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9ea4a2a740 10 client.25718 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9ea4a2a740 10 client.25718 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9ea4a2a740 20 client.25718 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:20.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a9619f0 0x556d0ad65a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9ea4a2a740 4 client.25718 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d0acc0b20 con 0x556d0a9619f0 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.886+0000 7f9e97400640 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a9619f0 0x556d0ad65a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.887+0000 7f9e97400640 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a9619f0 0x556d0ad65a90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a63cf0913cdcd19 server_cookie=324711b8e6a0953 in_seq=0 out_seq=0 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.887+0000 7f9e84c00640 10 client.25718 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9e84c00640 1 -- 172.21.15.44:0/3095973869 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d0acc0b20 con 0x556d0a9619f0 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9e84c00640 10 client.25718 handle_command_reply: tid=0 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 client.25718 shutdown 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 msgr2=0x556d0ad53ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad53ac0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e800139f0 tx=0x7f9e80013eb0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 shutdown_connections 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a9619f0 0x556d0ad65a90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad54000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad53ac0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/3095973869 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad53580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9e84c00640 0 client.25718 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:40:20.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.888+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 >> 172.21.15.44:0/3095973869 conn(0x556d0a997460 msgr2=0x556d0ad4b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:20.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.889+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 shutdown_connections 2024-07-26T08:40:20.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.889+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/3095973869 wait complete. 2024-07-26T08:40:20.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.889+0000 7f9ea4a2a740 20 client.25718 trim_cache size 0 max 16384 2024-07-26T08:40:20.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.904+0000 7f9ea4a2a740 1 Processor -- start 2024-07-26T08:40:20.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.904+0000 7f9ea4a2a740 1 -- start start 2024-07-26T08:40:20.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 0x556d0ad55460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad44540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad53c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981b2930 con 0x556d0a982390 2024-07-26T08:40:20.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981b4470 con 0x556d0a97a1b0 2024-07-26T08:40:20.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981b3000 con 0x556d0a9a3070 2024-07-26T08:40:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9e97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 0x556d0ad55460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9e8fe00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad44540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9e97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 0x556d0ad55460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:47534/0 (socket says 172.21.15.44:47534) 2024-07-26T08:40:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9e97400640 1 -- 172.21.15.44:0/2656642093 learned_addr learned my addr 172.21.15.44:0/2656642093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.905+0000 7f9e97400640 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad53c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e8fe00640 1 -- 172.21.15.44:0/2656642093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 msgr2=0x556d0ad53c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad53c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e8fe00640 1 -- 172.21.15.44:0/2656642093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 msgr2=0x556d0ad55460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 0x556d0ad55460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e8fe00640 1 -- 172.21.15.44:0/2656642093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0ad560b0 con 0x556d0a982390 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e97400640 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 0x556d0ad55460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad44540 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e8001b800 tx=0x7f9e80009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d70bc83530f8104 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.906+0000 7f9e87400640 1 -- 172.21.15.44:0/2656642093 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8001b410 con 0x556d0a982390 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.907+0000 7f9e87400640 1 -- 172.21.15.44:0/2656642093 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e8001b5b0 con 0x556d0a982390 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.907+0000 7f9e87400640 1 -- 172.21.15.44:0/2656642093 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80003850 con 0x556d0a982390 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.907+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2656642093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 msgr2=0x556d0ad44540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.907+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad44540 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e8001b800 tx=0x7f9e80009fd0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2656642093 shutdown_connections 2024-07-26T08:40:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad53c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a982390 0x556d0ad44540 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2656642093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a97a1b0 0x556d0ad55460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2656642093 >> 172.21.15.44:0/2656642093 conn(0x556d0a997460 msgr2=0x556d0ad534f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2656642093 shutdown_connections 2024-07-26T08:40:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2656642093 wait complete. 2024-07-26T08:40:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.908+0000 7f9ea4a2a740 1 -- start start 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 Processor -- start 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad59990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad59ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad5a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981b3da0 con 0x556d0a97a1b0 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981b36d0 con 0x556d0a982390 2024-07-26T08:40:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981b0910 con 0x556d0a9a3070 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9ea4a2a740 10 client.0 _ll_register_callbacks cb 0x7f9e80013630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.909+0000 7f9e97400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad59990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad59990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52488/0 (socket says 172.21.15.44:52488) 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 -- 172.21.15.44:0/2275641008 learned_addr learned my addr 172.21.15.44:0/2275641008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad59ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad5a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 -- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 msgr2=0x556d0ad5a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad5a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 -- 172.21.15.44:0/2275641008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 msgr2=0x556d0ad59ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad59ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 -- 172.21.15.44:0/2275641008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0adcd230 con 0x556d0a97a1b0 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e97400640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad59990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e90011790 tx=0x7f9e90005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a125a709d95ed7bb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.910+0000 7f9e84c00640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9e84c00640 1 -- 172.21.15.44:0/2275641008 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90003a30 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9e84c00640 1 -- 172.21.15.44:0/2275641008 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f9e90003bd0 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e8001b5b0 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9e84c00640 1 -- 172.21.15.44:0/2275641008 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9e90003dd0 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9ea4a2a740 10 client.15594 fetch_fsmap learned FSMap version 30 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556d0adce490 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.911+0000 7f9e84c00640 1 -- 172.21.15.44:0/2275641008 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e8001b5b0 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9e84c00640 1 -- 172.21.15.44:0/2275641008 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9e90025040 con 0x556d0a97a1b0 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9ea4a2a740 10 client.15594 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9ea4a2a740 10 client.15594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9ea4a2a740 20 client.15594 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a936da0 0x556d0add09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:20.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9ea4a2a740 4 client.15594 mds_command: new command op to 24679 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:40:20.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 --> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d0acbc5c0 con 0x556d0a936da0 2024-07-26T08:40:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.912+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a936da0 0x556d0add09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9e8fe00640 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a936da0 0x556d0add09f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6bb7ea42400b33d0 server_cookie=34687e889ef87b8 in_seq=0 out_seq=0 2024-07-26T08:40:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9e84c00640 10 client.15594 ms_handle_connect on v2:172.21.15.171:6838/2065417578 2024-07-26T08:40:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9e84c00640 1 -- 172.21.15.44:0/2275641008 <== mds.2 v2:172.21.15.171:6838/2065417578 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d0acbc5c0 con 0x556d0a936da0 2024-07-26T08:40:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9e84c00640 10 client.15594 handle_command_reply: tid=0 2024-07-26T08:40:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9ea4a2a740 1 client.15594 shutdown 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 msgr2=0x556d0ad59990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad59990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e90011790 tx=0x7f9e90005bd0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.913+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 shutdown_connections 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:6838/2065417578,v1:172.21.15.171:6839/2065417578] conn(0x556d0a936da0 0x556d0add09f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d0a9a3070 0x556d0ad5a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d0a982390 0x556d0ad59ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 --2- 172.21.15.44:0/2275641008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556d0a97a1b0 0x556d0ad59990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9e84c00640 0 client.15594 ms_handle_reset on v2:172.21.15.171:6838/2065417578 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 >> 172.21.15.44:0/2275641008 conn(0x556d0a997460 msgr2=0x556d0ad53e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 shutdown_connections 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 1 -- 172.21.15.44:0/2275641008 wait complete. 2024-07-26T08:40:20.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.914+0000 7f9ea4a2a740 20 client.15594 trim_cache size 0 max 16384 2024-07-26T08:40:20.920 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:40:20.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9e7c0b3d40 msgr2=0x7f9e7c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9e7c0b3d40 0x7f9e7c0b6250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556d0ad46c90 tx=0x7f9e88009260 comp rx=0 tx=0).stop 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 msgr2=0x7f9e9819bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e9819bf90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9e88005ab0 tx=0x7f9e88003c10 comp rx=0 tx=0).stop 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 shutdown_connections 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f9e7c0b3d40 0x7f9e7c0b6250 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9810b860 0x7f9e981a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9e9810af10 0x7f9e9819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 --2- 172.21.15.44:0/2116006282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e981055d0 0x7f9e9819bf90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 >> 172.21.15.44:0/2116006282 conn(0x7f9e98101040 msgr2=0x7f9e98103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 shutdown_connections 2024-07-26T08:40:20.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:20.915+0000 7f9e97e00640 1 -- 172.21.15.44:0/2116006282 wait complete. 2024-07-26T08:40:21.857 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T08:40:22.346 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:22.500 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:22 smithi044 ceph-mon[32404]: pgmap v533: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 473 KiB/s wr, 17 op/s 2024-07-26T08:40:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:22 smithi116 ceph-mon[37097]: pgmap v533: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 473 KiB/s wr, 17 op/s 2024-07-26T08:40:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:22 smithi171 ceph-mon[37789]: pgmap v533: 129 pgs: 129 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 473 KiB/s wr, 17 op/s 2024-07-26T08:40:24.477 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:24 smithi044 ceph-mon[32404]: pgmap v534: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 569 KiB/s wr, 33 op/s 2024-07-26T08:40:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:24 smithi116 ceph-mon[37097]: pgmap v534: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 569 KiB/s wr, 33 op/s 2024-07-26T08:40:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:24 smithi171 ceph-mon[37789]: pgmap v534: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 569 KiB/s wr, 33 op/s 2024-07-26T08:40:24.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 -- 172.21.15.44:0/206842961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c08109670 msgr2=0x7f3c0810fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 --2- 172.21.15.44:0/206842961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c08109670 0x7f3c0810fea0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009900 tx=0x7f3c0402a410 comp rx=0 tx=0).stop 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 -- 172.21.15.44:0/206842961 shutdown_connections 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 --2- 172.21.15.44:0/206842961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c08108d20 0x7f3c08109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 --2- 172.21.15.44:0/206842961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c08109670 0x7f3c0810fea0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 --2- 172.21.15.44:0/206842961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c081033e0 0x7f3c081037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 -- 172.21.15.44:0/206842961 >> 172.21.15.44:0/206842961 conn(0x7f3c080fee70 msgr2=0x7f3c081012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 -- 172.21.15.44:0/206842961 shutdown_connections 2024-07-26T08:40:24.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.813+0000 7f3c0f200640 1 -- 172.21.15.44:0/206842961 wait complete. 2024-07-26T08:40:24.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.814+0000 7f3c0f200640 1 Processor -- start 2024-07-26T08:40:24.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.814+0000 7f3c0f200640 1 -- start start 2024-07-26T08:40:24.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 0x7f3c08075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0f200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c08108d20 0x7f3c08075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0f200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c08109670 0x7f3c08075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0f200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c08113370 con 0x7f3c08109670 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081131f0 con 0x7f3c081033e0 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0f200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081134f0 con 0x7f3c08108d20 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 0x7f3c08075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 0x7f3c08075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39364/0 (socket says 172.21.15.44:39364) 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0de00640 1 -- 172.21.15.44:0/1361776239 learned_addr learned my addr 172.21.15.44:0/1361776239 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c08108d20 0x7f3c08075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.815+0000 7f3c0e800640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c08109670 0x7f3c08075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0de00640 1 -- 172.21.15.44:0/1361776239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c08108d20 msgr2=0x7f3c08075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0de00640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c08108d20 0x7f3c08075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0de00640 1 -- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c08109670 msgr2=0x7f3c08075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0de00640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c08109670 0x7f3c08075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0de00640 1 -- 172.21.15.44:0/1361776239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c081a8040 con 0x7f3c081033e0 2024-07-26T08:40:24.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0de00640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 0x7f3c08075140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000ebc0 tx=0x7f3c0000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:24.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3bfea00640 1 -- 172.21.15.44:0/1361776239 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00010630 con 0x7f3c081033e0 2024-07-26T08:40:24.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3bfea00640 1 -- 172.21.15.44:0/1361776239 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c00010c80 con 0x7f3c081033e0 2024-07-26T08:40:24.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3bfea00640 1 -- 172.21.15.44:0/1361776239 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00018b60 con 0x7f3c081033e0 2024-07-26T08:40:24.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.816+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c081a8330 con 0x7f3c081033e0 2024-07-26T08:40:24.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.817+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c081a88c0 con 0x7f3c081033e0 2024-07-26T08:40:24.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.820+0000 7f3c174b7740 1 Processor -- start 2024-07-26T08:40:24.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.820+0000 7f3c174b7740 1 -- start start 2024-07-26T08:40:24.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.820+0000 7f3bfea00640 1 -- 172.21.15.44:0/1361776239 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f3c00018d00 con 0x7f3c081033e0 2024-07-26T08:40:24.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.820+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab74030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.820+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab74570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7ae390 0x56144ab74ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c174b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56144aaeabf0 con 0x56144a7ae390 2024-07-26T08:40:24.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c174b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56144aada0f0 con 0x56144a7cf070 2024-07-26T08:40:24.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c174b7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56144aade130 con 0x56144a7a61b0 2024-07-26T08:40:24.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab74030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab74030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39366/0 (socket says 172.21.15.44:39366) 2024-07-26T08:40:24.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 -- 172.21.15.44:0/3337109365 learned_addr learned my addr 172.21.15.44:0/3337109365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:24.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0de00640 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab74570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 -- 172.21.15.44:0/3337109365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 msgr2=0x56144ab74570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab74570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 -- 172.21.15.44:0/3337109365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7ae390 msgr2=0x56144ab74ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7ae390 0x56144ab74ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.821+0000 7f3c0e800640 1 -- 172.21.15.44:0/3337109365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56144a83a650 con 0x56144a7cf070 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.822+0000 7f3c0e800640 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab74030 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3c0402aa00 tx=0x7f3c040040d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54f34a8b8d7a4c8f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.822+0000 7f3bf3000640 1 -- 172.21.15.44:0/3337109365 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04035030 con 0x56144a7cf070 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.822+0000 7f3bf3000640 1 -- 172.21.15.44:0/3337109365 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c040044c0 con 0x56144a7cf070 2024-07-26T08:40:24.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.822+0000 7f3bf3000640 1 -- 172.21.15.44:0/3337109365 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c040035f0 con 0x56144a7cf070 2024-07-26T08:40:24.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3bfea00640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3bf40b3dc0 0x7f3bf40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3bfea00640 1 -- 172.21.15.44:0/1361776239 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f3c000e4a20 con 0x7f3c081033e0 2024-07-26T08:40:24.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c0de00640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3bf40b3dc0 0x7f3bf40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 -- 172.21.15.44:0/3337109365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 msgr2=0x56144ab74030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab74030 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3c0402aa00 tx=0x7f3c040040d0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 -- 172.21.15.44:0/3337109365 shutdown_connections 2024-07-26T08:40:24.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab74570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7ae390 0x56144ab74ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3337109365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab74030 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.823+0000 7f3c174b7740 1 -- 172.21.15.44:0/3337109365 >> 172.21.15.44:0/3337109365 conn(0x56144a7c3460 msgr2=0x56144ab71f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.824+0000 7f3c174b7740 1 -- 172.21.15.44:0/3337109365 shutdown_connections 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.824+0000 7f3c174b7740 1 -- 172.21.15.44:0/3337109365 wait complete. 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.824+0000 7f3c0de00640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3bf40b3dc0 0x7f3bf40b62d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x56144ab72cb0 tx=0x7f3c000ea530 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.824+0000 7f3c174b7740 1 -- start start 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 Processor -- start 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab7f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab7fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab80060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56144a93d700 con 0x56144a7a61b0 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56144aa766c0 con 0x56144a7cf070 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56144ab827b0 con 0x56144a7ae390 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.825+0000 7f3c174b7740 10 client.0 _ll_register_callbacks cb 0x56144aa86170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab7f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab7f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36024/0 (socket says 172.21.15.44:36024) 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab7fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0e800640 1 -- 172.21.15.44:0/1827943632 learned_addr learned my addr 172.21.15.44:0/1827943632 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:24.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0e800640 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab80060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0d400640 1 -- 172.21.15.44:0/1827943632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 msgr2=0x56144ab80060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab80060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0d400640 1 -- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 msgr2=0x56144ab7f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab7f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.826+0000 7f3c0d400640 1 -- 172.21.15.44:0/1827943632 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56144aa8d860 con 0x56144a7ae390 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.827+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab7fb20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005d00 tx=0x7f3bf80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ede20c9a6fce7a4e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.827+0000 7f3be7400640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.827+0000 7f3be7400640 1 -- 172.21.15.44:0/1827943632 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8004220 con 0x56144a7ae390 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.827+0000 7f3be7400640 1 -- 172.21.15.44:0/1827943632 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f3bf80043c0 con 0x56144a7ae390 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.827+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56144aaecb40 con 0x56144a7ae390 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3be7400640 1 -- 172.21.15.44:0/1827943632 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3bf8024050 con 0x56144a7ae390 2024-07-26T08:40:24.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3c174b7740 10 client.25741 fetch_fsmap learned FSMap version 31 2024-07-26T08:40:24.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56144ab8e490 con 0x56144a7ae390 2024-07-26T08:40:24.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3be7400640 1 -- 172.21.15.44:0/1827943632 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56144aaecb40 con 0x56144a7ae390 2024-07-26T08:40:24.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3be7400640 1 -- 172.21.15.44:0/1827943632 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3bf8003780 con 0x56144a7ae390 2024-07-26T08:40:24.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3c174b7740 10 client.25741 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T08:40:24.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3c174b7740 10 client.25741 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T08:40:24.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3c174b7740 20 client.25741 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.828+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a78d9f0 0x56144ab919a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.829+0000 7f3c174b7740 4 client.25741 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.829+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56144aae85e0 con 0x56144a78d9f0 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.829+0000 7f3c0e800640 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a78d9f0 0x56144ab919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.830+0000 7f3c0e800640 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a78d9f0 0x56144ab919a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d61d8dc3041f571 server_cookie=e23324590e65c6e3 in_seq=0 out_seq=0 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.830+0000 7f3be7400640 10 client.25741 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.830+0000 7f3be7400640 1 -- 172.21.15.44:0/1827943632 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56144aae85e0 con 0x56144a78d9f0 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.830+0000 7f3be7400640 10 client.25741 handle_command_reply: tid=0 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.830+0000 7f3c174b7740 1 client.25741 shutdown 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 msgr2=0x56144ab7fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab7fb20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005d00 tx=0x7f3bf80107a0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 shutdown_connections 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a78d9f0 0x56144ab919a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab7fb20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3be7400640 0 client.25741 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab80060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1827943632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab7f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 >> 172.21.15.44:0/1827943632 conn(0x56144a7c3460 msgr2=0x56144ab771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 shutdown_connections 2024-07-26T08:40:24.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 1 -- 172.21.15.44:0/1827943632 wait complete. 2024-07-26T08:40:24.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.831+0000 7f3c174b7740 20 client.25741 trim_cache size 0 max 16384 2024-07-26T08:40:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.846+0000 7f3c174b7740 1 Processor -- start 2024-07-26T08:40:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.846+0000 7f3c174b7740 1 -- start start 2024-07-26T08:40:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.846+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab75a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab70560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab81210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c174b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081b29c0 con 0x56144a7a61b0 2024-07-26T08:40:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c174b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081b4530 con 0x56144a7cf070 2024-07-26T08:40:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c174b7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081b3090 con 0x56144a7ae390 2024-07-26T08:40:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab70560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab75a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab70560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48894/0 (socket says 172.21.15.44:48894) 2024-07-26T08:40:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab75a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36028/0 (socket says 172.21.15.44:36028) 2024-07-26T08:40:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 -- 172.21.15.44:0/1599667633 learned_addr learned my addr 172.21.15.44:0/1599667633 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 -- 172.21.15.44:0/1599667633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 msgr2=0x56144ab81210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab81210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 -- 172.21.15.44:0/1599667633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 msgr2=0x56144ab75a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab75a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.847+0000 7f3c0d400640 1 -- 172.21.15.44:0/1599667633 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56144ab76320 con 0x56144a7ae390 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.848+0000 7f3c0d400640 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab70560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80058c0 tx=0x7f3bf800bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57304cf1eb5773fc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.848+0000 7f3bf2600640 1 -- 172.21.15.44:0/1599667633 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80035f0 con 0x56144a7ae390 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.848+0000 7f3bf2600640 1 -- 172.21.15.44:0/1599667633 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf802c070 con 0x56144a7ae390 2024-07-26T08:40:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.848+0000 7f3bf2600640 1 -- 172.21.15.44:0/1599667633 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8004140 con 0x56144a7ae390 2024-07-26T08:40:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 -- 172.21.15.44:0/1599667633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 msgr2=0x56144ab70560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab70560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80058c0 tx=0x7f3bf800bfd0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 -- 172.21.15.44:0/1599667633 shutdown_connections 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7ae390 0x56144ab70560 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7cf070 0x56144ab81210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 --2- 172.21.15.44:0/1599667633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7a61b0 0x56144ab75a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 -- 172.21.15.44:0/1599667633 >> 172.21.15.44:0/1599667633 conn(0x56144a7c3460 msgr2=0x56144ab80b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.849+0000 7f3c174b7740 1 -- 172.21.15.44:0/1599667633 shutdown_connections 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.850+0000 7f3c174b7740 1 -- 172.21.15.44:0/1599667633 wait complete. 2024-07-26T08:40:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.850+0000 7f3c174b7740 1 -- start start 2024-07-26T08:40:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.850+0000 7f3c174b7740 1 Processor -- start 2024-07-26T08:40:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 0x56144ab85e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7cf070 0x56144ab86360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081b3e30 con 0x56144a7cf070 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081b3760 con 0x56144a7ae390 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3c081b09a0 con 0x56144a7a61b0 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c174b7740 10 client.0 _ll_register_callbacks cb 0x7f3bf800b9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c0d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 0x56144ab85e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c0e800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7cf070 0x56144ab86360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c0d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 0x56144ab85e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39410/0 (socket says 172.21.15.44:39410) 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c0d400640 1 -- 172.21.15.44:0/3563656038 learned_addr learned my addr 172.21.15.44:0/3563656038 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.851+0000 7f3c0e800640 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3c0d400640 1 -- 172.21.15.44:0/3563656038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 msgr2=0x56144ab858e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3c0d400640 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab858e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3c0d400640 1 -- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7cf070 msgr2=0x56144ab86360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3c0d400640 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7cf070 0x56144ab86360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3c0d400640 1 -- 172.21.15.44:0/3563656038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56144abf9190 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3c0d400640 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 0x56144ab85e20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80039c0 tx=0x7f3bf8003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=101322bef0501a5a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.852+0000 7f3be7400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3be7400640 1 -- 172.21.15.44:0/3563656038 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80040d0 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bf802c070 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3be7400640 1 -- 172.21.15.44:0/3563656038 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f3bf8004270 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3be7400640 1 -- 172.21.15.44:0/3563656038 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3bf8043050 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3c174b7740 10 client.25739 fetch_fsmap learned FSMap version 31 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56144abfa3e0 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.853+0000 7f3be7400640 1 -- 172.21.15.44:0/3563656038 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf802c070 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3be7400640 1 -- 172.21.15.44:0/3563656038 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3bf8004860 con 0x56144a7ae390 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c174b7740 10 client.25739 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T08:40:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c174b7740 10 client.25739 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c174b7740 20 client.25739 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a762da0 0x56144abfcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c174b7740 4 client.25739 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 --> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56144aae5d20 con 0x56144a762da0 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.854+0000 7f3c0e800640 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a762da0 0x56144abfcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.855+0000 7f3c0e800640 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a762da0 0x56144abfcb00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1cef4d6051c51eba server_cookie=35918cad77102ce6 in_seq=0 out_seq=0 2024-07-26T08:40:24.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.855+0000 7f3be7400640 10 client.25739 ms_handle_connect on v2:172.21.15.44:6838/1144072288 2024-07-26T08:40:24.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.855+0000 7f3be7400640 1 -- 172.21.15.44:0/3563656038 <== mds.1 v2:172.21.15.44:6838/1144072288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56144aae5d20 con 0x56144a762da0 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.855+0000 7f3be7400640 10 client.25739 handle_command_reply: tid=0 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.855+0000 7f3c174b7740 1 client.25739 shutdown 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 msgr2=0x56144ab85e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 0x56144ab85e20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80039c0 tx=0x7f3bf8003820 comp rx=0 tx=0).stop 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 shutdown_connections 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] conn(0x56144a762da0 0x56144abfcb00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56144a7cf070 0x56144ab86360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56144a7ae390 0x56144ab85e20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 --2- 172.21.15.44:0/3563656038 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56144a7a61b0 0x56144ab858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3be7400640 0 client.25739 ms_handle_reset on v2:172.21.15.44:6838/1144072288 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 >> 172.21.15.44:0/3563656038 conn(0x56144a7c3460 msgr2=0x56144ab81350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:24.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 shutdown_connections 2024-07-26T08:40:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 1 -- 172.21.15.44:0/3563656038 wait complete. 2024-07-26T08:40:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.856+0000 7f3c174b7740 20 client.25739 trim_cache size 0 max 16384 2024-07-26T08:40:24.861 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:40:24.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3bf40b3dc0 msgr2=0x7f3bf40b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3bf40b3dc0 0x7f3bf40b62d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x56144ab72cb0 tx=0x7f3c000ea530 comp rx=0 tx=0).stop 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 msgr2=0x7f3c08075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 0x7f3c08075140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000ebc0 tx=0x7f3c0000c6a0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 shutdown_connections 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f3bf40b3dc0 0x7f3bf40b62d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3c08108d20 0x7f3c08075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c08109670 0x7f3c08075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 --2- 172.21.15.44:0/1361776239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c081033e0 0x7f3c08075140 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 >> 172.21.15.44:0/1361776239 conn(0x7f3c080fee70 msgr2=0x7f3c08107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 shutdown_connections 2024-07-26T08:40:24.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:24.857+0000 7f3c0f200640 1 -- 172.21.15.44:0/1361776239 wait complete. 2024-07-26T08:40:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:25 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:40:25.740 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:25 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:40:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:25 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:40:25.824 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T08:40:26.315 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:26 smithi171 ceph-mon[37789]: pgmap v535: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 501 KiB/s wr, 28 op/s 2024-07-26T08:40:26.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:26 smithi044 ceph-mon[32404]: pgmap v535: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 501 KiB/s wr, 28 op/s 2024-07-26T08:40:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:26 smithi116 ceph-mon[37097]: pgmap v535: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 501 KiB/s wr, 28 op/s 2024-07-26T08:40:26.951 INFO:tasks.workunit.client.0.smithi044.stderr:Updating files: 7% (322/4329) Updating files: 8% (347/4329) Updating files: 9% (390/4329) Updating files: 10% (433/4329) Updating files: 11% (477/4329) Updating files: 11% (491/4329) Updating files: 12% (520/4329) Updating files: 13% (563/4329) Updating files: 14% (607/4329) Updating files: 15% (650/4329) Updating files: 16% (693/4329) Updating files: 17% (736/4329) Updating files: 17% (761/4329) Updating files: 18% (780/4329) Updating files: 19% (823/4329) Updating files: 20% (866/4329) Updating files: 21% (910/4329) Updating files: 22% (953/4329) Updating files: 23% (996/4329) Updating files: 24% (1039/4329) Updating files: 25% (1083/4329) Updating files: 26% (1126/4329) Updating files: 27% (1169/4329) Updating files: 28% (1213/4329) Updating files: 29% (1256/4329) Updating files: 29% (1289/4329) Updating files: 30% (1299/4329) Updating files: 31% (1342/4329) Updating files: 32% (1386/4329) Updating files: 33% (1429/4329) Updating files: 33% (1439/4329) Updating files: 34% (1472/4329) Updating files: 35% (1516/4329) Updating files: 36% (1559/4329) Updating files: 37% (1602/4329) Updating files: 38% (1646/4329) Updating files: 39% (1689/4329) Updating files: 40% (1732/4329) Updating files: 41% (1775/4329) Updating files: 42% (1819/4329) Updating files: 43% (1862/4329) Updating files: 44% (1905/4329) Updating files: 45% (1949/4329) Updating files: 45% (1963/4329) Updating files: 46% (1992/4329) Updating files: 47% (2035/4329) Updating files: 48% (2078/4329) Updating files: 49% (2122/4329) Updating files: 50% (2165/4329) Updating files: 51% (2208/4329) Updating files: 52% (2252/4329) Updating files: 53% (2295/4329) Updating files: 54% (2338/4329) Updating files: 55% (2381/4329) Updating files: 55% (2414/4329) Updating files: 56% (2425/4329) Updating files: 57% (2468/4329) Updating files: 58% (2511/4329) Updating files: 59% (2555/4329) Updating files: 60% (2598/4329) Updating files: 61% (2641/4329) Updating files: 62% (2684/4329) Updating files: 63% (2728/4329) Updating files: 64% (2771/4329) Updating files: 64% (2791/4329) Updating files: 65% (2814/4329) Updating files: 66% (2858/4329) Updating files: 67% (2901/4329) Updating files: 68% (2944/4329) Updating files: 69% (2988/4329) Updating files: 69% (3022/4329) Updating files: 70% (3031/4329) Updating files: 71% (3074/4329) Updating files: 72% (3117/4329) Updating files: 73% (3161/4329) Updating files: 74% (3204/4329) Updating files: 75% (3247/4329) Updating files: 75% (3264/4329) Updating files: 76% (3291/4329) Updating files: 77% (3334/4329) Updating files: 78% (3377/4329) Updating files: 79% (3420/4329) Updating files: 80% (3464/4329) Updating files: 81% (3507/4329) Updating files: 82% (3550/4329) Updating files: 83% (3594/4329) Updating files: 84% (3637/4329) Updating files: 85% (3680/4329) Updating files: 86% (3723/4329) Updating files: 87% (3767/4329) Updating files: 88% (3810/4329) Updating files: 88% (3819/4329) Updating files: 89% (3853/4329) Updating files: 90% (3897/4329) Updating files: 91% (3940/4329) Updating files: 92% (3983/4329) Updating files: 93% (4026/4329) Updating files: 94% (4070/4329) Updating files: 95% (4113/4329) Updating files: 96% (4156/4329) Updating files: 97% (4200/4329) Updating files: 98% (4243/4329) Updating files: 98% (4258/4329) Updating files: 99% (4286/4329) Updating files: 100% (4329/4329) Updating files: 100% (4329/4329), done. 2024-07-26T08:40:26.981 INFO:tasks.workunit.client.0.smithi044.stderr:+ cd xfstests-dev 2024-07-26T08:40:26.981 INFO:tasks.workunit.client.0.smithi044.stderr:+ git checkout 0e5c12dfd008efc2848c98108c9237487e91ef35 2024-07-26T08:40:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:28 smithi171 ceph-mon[37789]: pgmap v536: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 742 KiB/s wr, 37 op/s 2024-07-26T08:40:28.621 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:28 smithi044 ceph-mon[32404]: pgmap v536: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 742 KiB/s wr, 37 op/s 2024-07-26T08:40:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 -- 172.21.15.44:0/2020335459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80641033e0 msgr2=0x7f80641037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 --2- 172.21.15.44:0/2020335459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80641033e0 0x7f80641037f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8054009900 tx=0x7f805402a440 comp rx=0 tx=0).stop 2024-07-26T08:40:28.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 -- 172.21.15.44:0/2020335459 shutdown_connections 2024-07-26T08:40:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 --2- 172.21.15.44:0/2020335459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8064109670 0x7f806410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 --2- 172.21.15.44:0/2020335459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8064108d20 0x7f8064109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 --2- 172.21.15.44:0/2020335459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80641033e0 0x7f80641037f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.749+0000 7f8069200640 1 -- 172.21.15.44:0/2020335459 >> 172.21.15.44:0/2020335459 conn(0x7f80640fee70 msgr2=0x7f80641012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.750+0000 7f8069200640 1 -- 172.21.15.44:0/2020335459 shutdown_connections 2024-07-26T08:40:28.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.750+0000 7f8069200640 1 -- 172.21.15.44:0/2020335459 wait complete. 2024-07-26T08:40:28.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.751+0000 7f8069200640 1 Processor -- start 2024-07-26T08:40:28.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.751+0000 7f8069200640 1 -- start start 2024-07-26T08:40:28.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8069200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 0x7f806419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8069200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8064108d20 0x7f806419c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8069200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8064109670 0x7f80641a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8069200640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8064113370 con 0x7f80641033e0 2024-07-26T08:40:28.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8069200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80641131f0 con 0x7f8064109670 2024-07-26T08:40:28.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8069200640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f80641134f0 con 0x7f8064108d20 2024-07-26T08:40:28.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8063400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 0x7f806419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8063400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 0x7f806419bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36052/0 (socket says 172.21.15.44:36052) 2024-07-26T08:40:28.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.752+0000 7f8063400640 1 -- 172.21.15.44:0/1897714388 learned_addr learned my addr 172.21.15.44:0/1897714388 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:28.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.753+0000 7f8062a00640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8064108d20 0x7f806419c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.753+0000 7f8063400640 1 -- 172.21.15.44:0/1897714388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8064108d20 msgr2=0x7f806419c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.753+0000 7f8063400640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8064108d20 0x7f806419c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.753+0000 7f8063400640 1 -- 172.21.15.44:0/1897714388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8064109670 msgr2=0x7f80641a1780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T08:40:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.753+0000 7f8063400640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8064109670 0x7f80641a1780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.753+0000 7f8063400640 1 -- 172.21.15.44:0/1897714388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80641a1eb0 con 0x7f80641033e0 2024-07-26T08:40:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.754+0000 7f8063400640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 0x7f806419bf40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8054005d90 tx=0x7f80540045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:28.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.754+0000 7f805be00640 1 -- 172.21.15.44:0/1897714388 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805403f070 con 0x7f80641033e0 2024-07-26T08:40:28.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.755+0000 7f805be00640 1 -- 172.21.15.44:0/1897714388 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8054005590 con 0x7f80641033e0 2024-07-26T08:40:28.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.755+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80641a2140 con 0x7f80641033e0 2024-07-26T08:40:28.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.755+0000 7f805be00640 1 -- 172.21.15.44:0/1897714388 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054035030 con 0x7f80641033e0 2024-07-26T08:40:28.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.755+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8064109c60 con 0x7f80641033e0 2024-07-26T08:40:28.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.759+0000 7f80714bc740 1 Processor -- start 2024-07-26T08:40:28.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.759+0000 7f805be00640 1 -- 172.21.15.44:0/1897714388 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f8054042070 con 0x7f80641033e0 2024-07-26T08:40:28.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.760+0000 7f80714bc740 1 -- start start 2024-07-26T08:40:28.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.760+0000 7f80714bc740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x5621284e1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f80714bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f80714bc740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284e1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f80714bc740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562128457bd0 con 0x56212813c070 2024-07-26T08:40:28.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f80714bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621284470d0 con 0x5621281131b0 2024-07-26T08:40:28.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f80714bc740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56212844b110 con 0x56212811b390 2024-07-26T08:40:28.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f8062a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284e1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f8062a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284e1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48940/0 (socket says 172.21.15.44:48940) 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f8062a00640 1 -- 172.21.15.44:0/2032941911 learned_addr learned my addr 172.21.15.44:0/2032941911 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.761+0000 7f8063e00640 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x5621284e1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.762+0000 7f8062a00640 1 -- 172.21.15.44:0/2032941911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 msgr2=0x5621284e1740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.762+0000 7f8062a00640 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e1740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.762+0000 7f8062a00640 1 -- 172.21.15.44:0/2032941911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 msgr2=0x5621284e1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.762+0000 7f8062a00640 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x5621284e1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.762+0000 7f8062a00640 1 -- 172.21.15.44:0/2032941911 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621281a7650 con 0x56212811b390 2024-07-26T08:40:28.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.762+0000 7f8062a00640 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284e1c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8050009c20 tx=0x7f8050011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae74c99630ecc77f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:28.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.763+0000 7f805be00640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f80480bc6a0 0x7f80480bebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.763+0000 7f8058c00640 1 -- 172.21.15.44:0/2032941911 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050004230 con 0x56212811b390 2024-07-26T08:40:28.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.763+0000 7f805be00640 1 -- 172.21.15.44:0/1897714388 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8054104960 con 0x7f80641033e0 2024-07-26T08:40:28.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.763+0000 7f8058c00640 1 -- 172.21.15.44:0/2032941911 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80500043d0 con 0x56212811b390 2024-07-26T08:40:28.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.763+0000 7f8058c00640 1 -- 172.21.15.44:0/2032941911 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050004d80 con 0x56212811b390 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 -- 172.21.15.44:0/2032941911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 msgr2=0x5621284e1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284e1c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8050009c20 tx=0x7f8050011810 comp rx=0 tx=0).stop 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 -- 172.21.15.44:0/2032941911 shutdown_connections 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284e1c80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 --2- 172.21.15.44:0/2032941911 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x5621284e1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 -- 172.21.15.44:0/2032941911 >> 172.21.15.44:0/2032941911 conn(0x562128130460 msgr2=0x5621284df0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:28.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 -- 172.21.15.44:0/2032941911 shutdown_connections 2024-07-26T08:40:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f80714bc740 1 -- 172.21.15.44:0/2032941911 wait complete. 2024-07-26T08:40:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.764+0000 7f8063e00640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f80480bc6a0 0x7f80480bebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.765+0000 7f80714bc740 1 -- start start 2024-07-26T08:40:28.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.765+0000 7f80714bc740 1 Processor -- start 2024-07-26T08:40:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.766+0000 7f8063e00640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f80480bc6a0 0x7f80480bebb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f805c005fd0 tx=0x7f805c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:28.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.766+0000 7f80714bc740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 0x5621284ec870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.766+0000 7f80714bc740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284ecdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.766+0000 7f80714bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212813c070 0x5621284ed310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.766+0000 7f8062a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284ecdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.766+0000 7f8063e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212813c070 0x5621284ed310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212813c070 0x5621284ed310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39444/0 (socket says 172.21.15.44:39444) 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8062a00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284ecdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:48944/0 (socket says 172.21.15.44:48944) 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f80714bc740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5621282aa700 con 0x5621281131b0 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f80714bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621283e36a0 con 0x56212813c070 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f80714bc740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5621284ef9e0 con 0x56212811b390 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 0x5621284ec870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 0x5621284ec870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36068/0 (socket says 172.21.15.44:36068) 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 -- 172.21.15.44:0/3320616526 learned_addr learned my addr 172.21.15.44:0/3320616526 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f80714bc740 10 client.0 _ll_register_callbacks cb 0x5621283f3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 -- 172.21.15.44:0/3320616526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 msgr2=0x5621284ecdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284ecdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 -- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212813c070 msgr2=0x5621284ed310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212813c070 0x5621284ed310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.767+0000 7f8063400640 1 -- 172.21.15.44:0/3320616526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621283fa840 con 0x5621281131b0 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.768+0000 7f8063400640 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 0x5621284ec870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8054101dd0 tx=0x7f80540cac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143ae2bcfb0c9809 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.768+0000 7f804d600640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.768+0000 7f804d600640 1 -- 172.21.15.44:0/3320616526 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054049040 con 0x5621281131b0 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.768+0000 7f804d600640 1 -- 172.21.15.44:0/3320616526 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f8054003c80 con 0x5621281131b0 2024-07-26T08:40:28.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.768+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80500043d0 con 0x5621281131b0 2024-07-26T08:40:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:28 smithi116 ceph-mon[37097]: pgmap v536: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 742 KiB/s wr, 37 op/s 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f804d600640 1 -- 172.21.15.44:0/3320616526 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f80540cacf0 con 0x5621281131b0 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 10 client.15621 fetch_fsmap learned FSMap version 32 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5621284fb660 con 0x5621281131b0 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f804d600640 1 -- 172.21.15.44:0/3320616526 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80500043d0 con 0x5621281131b0 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f804d600640 1 -- 172.21.15.44:0/3320616526 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f80540d3e00 con 0x5621281131b0 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 10 client.15621 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 20 client.15621 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280fa9f0 0x5621284feb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 4 client.15621 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:28.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.769+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562128459b20 con 0x5621280fa9f0 2024-07-26T08:40:28.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.770+0000 7f8062a00640 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280fa9f0 0x5621284feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.770+0000 7f8062a00640 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280fa9f0 0x5621284feb90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6b9880b4243050a server_cookie=3ef46e1446fd134d in_seq=0 out_seq=0 2024-07-26T08:40:28.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.770+0000 7f804d600640 10 client.15621 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f804d600640 1 -- 172.21.15.44:0/3320616526 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562128459b20 con 0x5621280fa9f0 2024-07-26T08:40:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f804d600640 10 client.15621 handle_command_reply: tid=0 2024-07-26T08:40:28.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 client.15621 shutdown 2024-07-26T08:40:28.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 msgr2=0x5621284ec870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 0x5621284ec870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8054101dd0 tx=0x7f80540cac60 comp rx=0 tx=0).stop 2024-07-26T08:40:28.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 shutdown_connections 2024-07-26T08:40:28.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280fa9f0 0x5621284feb90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212811b390 0x5621284ecdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212813c070 0x5621284ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 --2- 172.21.15.44:0/3320616526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5621281131b0 0x5621284ec870 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 >> 172.21.15.44:0/3320616526 conn(0x562128130460 msgr2=0x5621284e43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.779+0000 7f804d600640 0 client.15621 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.780+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 shutdown_connections 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.780+0000 7f80714bc740 1 -- 172.21.15.44:0/3320616526 wait complete. 2024-07-26T08:40:28.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.780+0000 7f80714bc740 20 client.15621 trim_cache size 0 max 16384 2024-07-26T08:40:28.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.795+0000 7f80714bc740 1 Processor -- start 2024-07-26T08:40:28.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 -- start start 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212811b390 0x562128504350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212813c070 0x562128504890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80641ae490 con 0x56212811b390 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80641b0090 con 0x5621281131b0 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.796+0000 7f80714bc740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f80641aeb60 con 0x56212813c070 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39448/0 (socket says 172.21.15.44:39448) 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063e00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212813c070 0x562128504890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063400640 1 -- 172.21.15.44:0/3752869339 learned_addr learned my addr 172.21.15.44:0/3752869339 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:28.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8062a00640 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212811b390 0x562128504350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063e00640 1 -- 172.21.15.44:0/3752869339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 msgr2=0x5621284e2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063e00640 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063e00640 1 -- 172.21.15.44:0/3752869339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212811b390 msgr2=0x562128504350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063e00640 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212811b390 0x562128504350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063e00640 1 -- 172.21.15.44:0/3752869339 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621284edb30 con 0x56212813c070 2024-07-26T08:40:28.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.797+0000 7f8063400640 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:28.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.798+0000 7f8063e00640 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212813c070 0x562128504890 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5621284ed850 tx=0x7f805c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=823c2e51f11b6930 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:28.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.798+0000 7f804fe00640 1 -- 172.21.15.44:0/3752869339 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c01f980 con 0x56212813c070 2024-07-26T08:40:28.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.798+0000 7f804fe00640 1 -- 172.21.15.44:0/3752869339 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f805c003970 con 0x56212813c070 2024-07-26T08:40:28.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.798+0000 7f804fe00640 1 -- 172.21.15.44:0/3752869339 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c0045f0 con 0x56212813c070 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 -- 172.21.15.44:0/3752869339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212813c070 msgr2=0x562128504890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212813c070 0x562128504890 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5621284ed850 tx=0x7f805c013000 comp rx=0 tx=0).stop 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 -- 172.21.15.44:0/3752869339 shutdown_connections 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56212813c070 0x562128504890 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212811b390 0x562128504350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 --2- 172.21.15.44:0/3752869339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621281131b0 0x5621284e2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 -- 172.21.15.44:0/3752869339 >> 172.21.15.44:0/3752869339 conn(0x562128130460 msgr2=0x5621284e2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:28.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 -- 172.21.15.44:0/3752869339 shutdown_connections 2024-07-26T08:40:28.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.799+0000 7f80714bc740 1 -- 172.21.15.44:0/3752869339 wait complete. 2024-07-26T08:40:28.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.800+0000 7f80714bc740 1 -- start start 2024-07-26T08:40:28.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.800+0000 7f80714bc740 1 Processor -- start 2024-07-26T08:40:28.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.800+0000 7f80714bc740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621281131b0 0x5621284f2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f80714bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 0x5621284f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f80714bc740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x56212855e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f80714bc740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80641af900 con 0x56212813c070 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f80714bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80641af230 con 0x56212811b390 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f80714bc740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f80641ac4b0 con 0x5621281131b0 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f80714bc740 10 client.0 _ll_register_callbacks cb 0x7f80540cb040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f8062a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 0x5621284f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f8062a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 0x5621284f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:39462/0 (socket says 172.21.15.44:39462) 2024-07-26T08:40:28.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.801+0000 7f8062a00640 1 -- 172.21.15.44:0/4259947238 learned_addr learned my addr 172.21.15.44:0/4259947238 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.802+0000 7f8063e00640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x56212855e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.802+0000 7f8063400640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621281131b0 0x5621284f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.802+0000 7f8062a00640 1 -- 172.21.15.44:0/4259947238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621281131b0 msgr2=0x5621284f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.802+0000 7f8062a00640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621281131b0 0x5621284f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.802+0000 7f8062a00640 1 -- 172.21.15.44:0/4259947238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 msgr2=0x56212855e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.802+0000 7f8062a00640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x56212855e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.803+0000 7f8062a00640 1 -- 172.21.15.44:0/4259947238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56212855e9c0 con 0x56212811b390 2024-07-26T08:40:28.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.803+0000 7f8063e00640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x56212855e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:28.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.803+0000 7f8062a00640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 0x5621284f2be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8050011d10 tx=0x7f8050004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7807763083a4bee2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.803+0000 7f804d600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.804+0000 7f804d600640 1 -- 172.21.15.44:0/4259947238 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050034060 con 0x56212811b390 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.804+0000 7f804d600640 1 -- 172.21.15.44:0/4259947238 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f80500037f0 con 0x56212811b390 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.804+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f805c003970 con 0x56212811b390 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.804+0000 7f804d600640 1 -- 172.21.15.44:0/4259947238 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f805003b050 con 0x56212811b390 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.804+0000 7f80714bc740 10 client.25751 fetch_fsmap learned FSMap version 32 2024-07-26T08:40:28.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.804+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562128566ec0 con 0x56212811b390 2024-07-26T08:40:28.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f804d600640 1 -- 172.21.15.44:0/4259947238 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f805c003970 con 0x56212811b390 2024-07-26T08:40:28.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f804d600640 1 -- 172.21.15.44:0/4259947238 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f805002e070 con 0x56212811b390 2024-07-26T08:40:28.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f80714bc740 10 client.25751 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T08:40:28.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f80714bc740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:28.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f80714bc740 20 client.25751 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:28.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f80714bc740 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280cfda0 0x562128569510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:28.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f80714bc740 4 client.25751 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T08:40:28.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.805+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621284555c0 con 0x5621280cfda0 2024-07-26T08:40:28.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.806+0000 7f8063400640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280cfda0 0x562128569510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:28.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.807+0000 7f8063400640 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280cfda0 0x562128569510 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81f579f7ae6138e4 server_cookie=817eb42144d1108 in_seq=0 out_seq=0 2024-07-26T08:40:28.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.807+0000 7f804d600640 10 client.25751 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:28.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f804d600640 1 -- 172.21.15.44:0/4259947238 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621284555c0 con 0x5621280cfda0 2024-07-26T08:40:28.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f804d600640 10 client.25751 handle_command_reply: tid=0 2024-07-26T08:40:28.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 client.25751 shutdown 2024-07-26T08:40:28.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 msgr2=0x5621284f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 0x5621284f2be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8050011d10 tx=0x7f8050004270 comp rx=0 tx=0).stop 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 shutdown_connections 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5621280cfda0 0x562128569510 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56212813c070 0x56212855e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56212811b390 0x5621284f2be0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.815+0000 7f80714bc740 1 --2- 172.21.15.44:0/4259947238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621281131b0 0x5621284f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.816+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 >> 172.21.15.44:0/4259947238 conn(0x562128130460 msgr2=0x5621284edd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.816+0000 7f804d600640 0 client.25751 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.816+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 shutdown_connections 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.816+0000 7f80714bc740 1 -- 172.21.15.44:0/4259947238 wait complete. 2024-07-26T08:40:28.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.816+0000 7f80714bc740 20 client.25751 trim_cache size 0 max 16384 2024-07-26T08:40:28.820 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-07-26T08:40:28.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f80480bc6a0 msgr2=0x7f80480bebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f80480bc6a0 0x7f80480bebb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f805c005fd0 tx=0x7f805c009500 comp rx=0 tx=0).stop 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 msgr2=0x7f806419bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 0x7f806419bf40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8054005d90 tx=0x7f80540045f0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 shutdown_connections 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f80480bc6a0 0x7f80480bebb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8064108d20 0x7f806419c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8064109670 0x7f80641a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 --2- 172.21.15.44:0/1897714388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80641033e0 0x7f806419bf40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:28.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 >> 172.21.15.44:0/1897714388 conn(0x7f80640fee70 msgr2=0x7f8064108450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:28.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.817+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 shutdown_connections 2024-07-26T08:40:28.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:28.818+0000 7f8069200640 1 -- 172.21.15.44:0/1897714388 wait complete. 2024-07-26T08:40:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:29 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:40:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:29 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:40:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:29 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:40:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:30 smithi171 ceph-mon[37789]: pgmap v537: 129 pgs: 129 active+clean; 23 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 767 KiB/s wr, 41 op/s 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr:Note: switching to '0e5c12dfd008efc2848c98108c9237487e91ef35'. 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.717 INFO:tasks.workunit.client.0.smithi044.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: git switch -c 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr:Or undo this operation with: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: git switch - 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-07-26T08:40:30.718 INFO:tasks.workunit.client.0.smithi044.stderr:HEAD is now at 0e5c12df xfs/522: use reflink instead of crc as test feature 2024-07-26T08:40:30.742 INFO:tasks.workunit.client.0.smithi044.stderr:+ make -j4 2024-07-26T08:40:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:30 smithi044 ceph-mon[32404]: pgmap v537: 129 pgs: 129 active+clean; 23 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 767 KiB/s wr, 41 op/s 2024-07-26T08:40:30.771 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize -cfi 2024-07-26T08:40:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:30 smithi116 ceph-mon[37097]: pgmap v537: 129 pgs: 129 active+clean; 23 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 767 KiB/s wr, 41 op/s 2024-07-26T08:40:30.839 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T08:40:30.840 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T08:40:31.330 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:31.442 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: putting auxiliary files in '.'. 2024-07-26T08:40:31.442 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file './config.guess' 2024-07-26T08:40:31.455 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file './config.sub' 2024-07-26T08:40:31.470 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file './install-sh' 2024-07-26T08:40:31.485 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file './ltmain.sh' 2024-07-26T08:40:31.585 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-07-26T08:40:31.585 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-07-26T08:40:31.618 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-07-26T08:40:31.644 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-07-26T08:40:31.670 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-07-26T08:40:31.694 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-07-26T08:40:31.714 INFO:tasks.workunit.client.0.smithi044.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-07-26T08:40:31.717 INFO:tasks.workunit.client.0.smithi044.stdout:cp include/install-sh . 2024-07-26T08:40:31.728 INFO:tasks.workunit.client.0.smithi044.stdout:aclocal -I m4 2024-07-26T08:40:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:32 smithi171 ceph-mon[37789]: pgmap v538: 129 pgs: 129 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-07-26T08:40:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:32 smithi044 ceph-mon[32404]: pgmap v538: 129 pgs: 129 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-07-26T08:40:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:32 smithi116 ceph-mon[37097]: pgmap v538: 129 pgs: 129 active+clean; 27 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 1.1 MiB/s wr, 50 op/s 2024-07-26T08:40:34.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:34 smithi116 ceph-mon[37097]: pgmap v539: 129 pgs: 129 active+clean; 30 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 65 op/s 2024-07-26T08:40:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:34 smithi171 ceph-mon[37789]: pgmap v539: 129 pgs: 129 active+clean; 30 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 65 op/s 2024-07-26T08:40:34.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:34 smithi044 ceph-mon[32404]: pgmap v539: 129 pgs: 129 active+clean; 30 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 65 op/s 2024-07-26T08:40:35.116 INFO:tasks.workunit.client.0.smithi044.stdout:autoheader 2024-07-26T08:40:35.886 INFO:tasks.workunit.client.0.smithi044.stdout:autoconf 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 -- 172.21.15.44:0/507575971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 msgr2=0x7fb11c105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 --2- 172.21.15.44:0/507575971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c105bc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb110009900 tx=0x7fb11002a3f0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 -- 172.21.15.44:0/507575971 shutdown_connections 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 --2- 172.21.15.44:0/507575971 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10b0f0 0x7fb11c10b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 --2- 172.21.15.44:0/507575971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10ba40 0x7fb11c112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 --2- 172.21.15.44:0/507575971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c105bc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 -- 172.21.15.44:0/507575971 >> 172.21.15.44:0/507575971 conn(0x7fb11c101240 msgr2=0x7fb11c103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 -- 172.21.15.44:0/507575971 shutdown_connections 2024-07-26T08:40:36.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.107+0000 7fb123800640 1 -- 172.21.15.44:0/507575971 wait complete. 2024-07-26T08:40:36.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.112+0000 7fb123800640 1 Processor -- start 2024-07-26T08:40:36.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.112+0000 7fb123800640 1 -- start start 2024-07-26T08:40:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c19c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10b0f0 0x7fb11c19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10ba40 0x7fb11c1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c1156f0 con 0x7fb11c10b0f0 2024-07-26T08:40:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c115570 con 0x7fb11c1057b0 2024-07-26T08:40:36.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c115870 con 0x7fb11c10ba40 2024-07-26T08:40:36.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb11be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10b0f0 0x7fb11c19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb11be00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10b0f0 0x7fb11c19c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42978/0 (socket says 172.21.15.44:42978) 2024-07-26T08:40:36.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb11be00640 1 -- 172.21.15.44:0/1836014254 learned_addr learned my addr 172.21.15.44:0/1836014254 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:36.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c19c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb121800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10ba40 0x7fb11c1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 -- 172.21.15.44:0/1836014254 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10ba40 msgr2=0x7fb11c1a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10ba40 0x7fb11c1a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 -- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10b0f0 msgr2=0x7fb11c19c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10b0f0 0x7fb11c19c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 -- 172.21.15.44:0/1836014254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c1a2010 con 0x7fb11c1057b0 2024-07-26T08:40:36.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb121800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10ba40 0x7fb11c1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:36.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb120e00640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c19c0a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb11002a910 tx=0x7fb110031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:36.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb119600640 1 -- 172.21.15.44:0/1836014254 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11003f030 con 0x7fb11c1057b0 2024-07-26T08:40:36.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb119600640 1 -- 172.21.15.44:0/1836014254 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb110004490 con 0x7fb11c1057b0 2024-07-26T08:40:36.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11c1a22a0 con 0x7fb11c1057b0 2024-07-26T08:40:36.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11c1a8a60 con 0x7fb11c1057b0 2024-07-26T08:40:36.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.113+0000 7fb119600640 1 -- 172.21.15.44:0/1836014254 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110035070 con 0x7fb11c1057b0 2024-07-26T08:40:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 Processor -- start 2024-07-26T08:40:36.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- start start 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb119600640 1 -- 172.21.15.44:0/1836014254 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fb110042030 con 0x7fb11c1057b0 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e4501670 0x5647e48c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5647e483dbf0 con 0x5647e4522350 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5647e482d0f0 con 0x5647e44f9490 2024-07-26T08:40:36.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5647e4831130 con 0x5647e4501670 2024-07-26T08:40:36.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42990/0 (socket says 172.21.15.44:42990) 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 -- 172.21.15.44:0/2472528872 learned_addr learned my addr 172.21.15.44:0/2472528872 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb120e00640 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 -- 172.21.15.44:0/2472528872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e4501670 msgr2=0x5647e48c7810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e4501670 0x5647e48c7810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 -- 172.21.15.44:0/2472528872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 msgr2=0x5647e48c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 -- 172.21.15.44:0/2472528872 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e458d930 con 0x5647e4522350 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb121800640 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48c6d90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb1140117e0 tx=0x7fb114011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f87644ea578304d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb10e000640 1 -- 172.21.15.44:0/2472528872 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114011310 con 0x5647e4522350 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb10e000640 1 -- 172.21.15.44:0/2472528872 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb114003e20 con 0x5647e4522350 2024-07-26T08:40:36.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb10e000640 1 -- 172.21.15.44:0/2472528872 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1140044f0 con 0x5647e4522350 2024-07-26T08:40:36.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb119600640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1040b3dc0 0x7fb1040b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb119600640 1 -- 172.21.15.44:0/1836014254 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb110104080 con 0x7fb11c1057b0 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb120e00640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1040b3dc0 0x7fb1040b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- 172.21.15.44:0/2472528872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 msgr2=0x5647e48c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48c6d90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb1140117e0 tx=0x7fb114011ca0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- 172.21.15.44:0/2472528872 shutdown_connections 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e4501670 0x5647e48c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/2472528872 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48c6d90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- 172.21.15.44:0/2472528872 >> 172.21.15.44:0/2472528872 conn(0x5647e4516740 msgr2=0x5647e48c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- 172.21.15.44:0/2472528872 shutdown_connections 2024-07-26T08:40:36.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb120e00640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1040b3dc0 0x7fb1040b62d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5647e48c5a10 tx=0x7fb110000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:36.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.119+0000 7fb12a3c3740 1 -- 172.21.15.44:0/2472528872 wait complete. 2024-07-26T08:40:36.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.124+0000 7fb12a3c3740 1 -- start start 2024-07-26T08:40:36.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.124+0000 7fb12a3c3740 1 Processor -- start 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.124+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48dab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.124+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48db5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb121800640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:58376/0 (socket says 172.21.15.44:58376) 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 -- 172.21.15.44:0/4169684778 learned_addr learned my addr 172.21.15.44:0/4169684778 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:36.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5647e4690890 con 0x5647e4522350 2024-07-26T08:40:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5647e47c9330 con 0x5647e4501670 2024-07-26T08:40:36.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5647e48ddca0 con 0x5647e44f9490 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 10 client.0 _ll_register_callbacks cb 0x5647e47f5010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb121800640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48db5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 -- 172.21.15.44:0/4169684778 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 msgr2=0x5647e48dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48dab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 -- 172.21.15.44:0/4169684778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 msgr2=0x5647e48db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48db5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e48e7260 con 0x5647e4501670 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb121800640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48db5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb11be00640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db0b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1080116f0 tx=0x7fb108011bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4da43dd77d4bd3a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 1 -- 172.21.15.44:0/4169684778 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10800fcd0 con 0x5647e4501670 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 1 -- 172.21.15.44:0/4169684778 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb108004020 con 0x5647e4501670 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb114003e20 con 0x5647e4501670 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 1 -- 172.21.15.44:0/4169684778 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb10800fe70 con 0x5647e4501670 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 10 client.25763 fetch_fsmap learned FSMap version 32 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5647e48e9bc0 con 0x5647e4501670 2024-07-26T08:40:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 1 -- 172.21.15.44:0/4169684778 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb114003e20 con 0x5647e4501670 2024-07-26T08:40:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 1 -- 172.21.15.44:0/4169684778 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fb108019430 con 0x5647e4501670 2024-07-26T08:40:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 10 client.25763 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T08:40:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 20 client.25763 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e44e0cd0 0x5647e48ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 4 client.25763 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647e483fb40 con 0x5647e44e0cd0 2024-07-26T08:40:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb121800640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e44e0cd0 0x5647e48ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb121800640 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e44e0cd0 0x5647e48ed240 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94fb835b44ee2f50 server_cookie=90fe26ebf764647d in_seq=0 out_seq=0 2024-07-26T08:40:36.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 10 client.25763 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 1 -- 172.21.15.44:0/4169684778 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5647e483fb40 con 0x5647e44e0cd0 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.125+0000 7fb102a00640 10 client.25763 handle_command_reply: tid=0 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 client.25763 shutdown 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 msgr2=0x5647e48db0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db0b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1080116f0 tx=0x7fb108011bb0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 shutdown_connections 2024-07-26T08:40:36.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e44e0cd0 0x5647e48ed240 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4522350 0x5647e48db5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db0b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb102a00640 0 client.25763 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/4169684778 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48dab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 >> 172.21.15.44:0/4169684778 conn(0x5647e4516740 msgr2=0x5647e48c3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 shutdown_connections 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 1 -- 172.21.15.44:0/4169684778 wait complete. 2024-07-26T08:40:36.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.131+0000 7fb12a3c3740 20 client.25763 trim_cache size 0 max 16384 2024-07-26T08:40:36.145 INFO:tasks.workunit.client.0.smithi044.stdout:./configure \ 2024-07-26T08:40:36.145 INFO:tasks.workunit.client.0.smithi044.stdout: --libexecdir=/usr/lib \ 2024-07-26T08:40:36.145 INFO:tasks.workunit.client.0.smithi044.stdout: --exec_prefix=/var/lib 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 Processor -- start 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 -- start start 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e44e0cd0 0x5647e48d6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48db070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc00ff00 con 0x5647e44e0cd0 2024-07-26T08:40:36.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc011b00 con 0x5647e4501670 2024-07-26T08:40:36.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc0105d0 con 0x5647e44f9490 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48db070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb121800640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e44e0cd0 0x5647e48d6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48db070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:39784/0 (socket says 172.21.15.44:39784) 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 -- 172.21.15.44:0/236210988 learned_addr learned my addr 172.21.15.44:0/236210988 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb121800640 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 -- 172.21.15.44:0/236210988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 msgr2=0x5647e48db5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 -- 172.21.15.44:0/236210988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e44e0cd0 msgr2=0x5647e48d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e44e0cd0 0x5647e48d6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 -- 172.21.15.44:0/236210988 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e48dbe30 con 0x5647e44f9490 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb121800640 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e44e0cd0 0x5647e48d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb11be00640 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48db070 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1080058c0 tx=0x7fb10800fd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91357d5a589dec27 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb10d600640 1 -- 172.21.15.44:0/236210988 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108030070 con 0x5647e44f9490 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb10d600640 1 -- 172.21.15.44:0/236210988 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb108005520 con 0x5647e44f9490 2024-07-26T08:40:36.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.149+0000 7fb10d600640 1 -- 172.21.15.44:0/236210988 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108004680 con 0x5647e44f9490 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/236210988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 msgr2=0x5647e48db070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48db070 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1080058c0 tx=0x7fb10800fd90 comp rx=0 tx=0).stop 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/236210988 shutdown_connections 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44f9490 0x5647e48db070 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e4501670 0x5647e48db5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/236210988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e44e0cd0 0x5647e48d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/236210988 >> 172.21.15.44:0/236210988 conn(0x5647e4516740 msgr2=0x5647e48c4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/236210988 shutdown_connections 2024-07-26T08:40:36.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/236210988 wait complete. 2024-07-26T08:40:36.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- start start 2024-07-26T08:40:36.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 Processor -- start 2024-07-26T08:40:36.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44e0cd0 0x5647e48e37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48dec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4501670 0x5647e48df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc011370 con 0x5647e4501670 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc010ca0 con 0x5647e44f9490 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc00dee0 con 0x5647e44e0cd0 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 10 client.0 _ll_register_callbacks cb 0x7fb108009450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48dec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48dec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:58402/0 (socket says 172.21.15.44:58402) 2024-07-26T08:40:36.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 -- 172.21.15.44:0/3289194400 learned_addr learned my addr 172.21.15.44:0/3289194400 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 -- 172.21.15.44:0/3289194400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44e0cd0 msgr2=0x5647e48e37b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T08:40:36.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44e0cd0 0x5647e48e37b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 -- 172.21.15.44:0/3289194400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4501670 msgr2=0x5647e48df2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4501670 0x5647e48df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 -- 172.21.15.44:0/3289194400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e48dfa40 con 0x5647e44f9490 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb11be00640 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48dec60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1080199c0 tx=0x7fb10800f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=902476835331ebe9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb102a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb102a00640 1 -- 172.21.15.44:0/3289194400 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108030040 con 0x5647e44f9490 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb102a00640 1 -- 172.21.15.44:0/3289194400 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7fb108004020 con 0x5647e44f9490 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb108005520 con 0x5647e44f9490 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb102a00640 1 -- 172.21.15.44:0/3289194400 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb108004220 con 0x5647e44f9490 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 10 client.25769 fetch_fsmap learned FSMap version 32 2024-07-26T08:40:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.155+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5647e4955a60 con 0x5647e44f9490 2024-07-26T08:40:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb102a00640 1 -- 172.21.15.44:0/3289194400 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb108005520 con 0x5647e44f9490 2024-07-26T08:40:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb102a00640 1 -- 172.21.15.44:0/3289194400 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7fb108033070 con 0x5647e44f9490 2024-07-26T08:40:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb12a3c3740 10 client.25769 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T08:40:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb12a3c3740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb12a3c3740 20 client.25769 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e4522350 0x5647e4958290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb12a3c3740 4 client.25769 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T08:40:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5647e483b5e0 con 0x5647e4522350 2024-07-26T08:40:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.160+0000 7fb121800640 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e4522350 0x5647e4958290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb121800640 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e4522350 0x5647e4958290 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab36875ee50b7b46 server_cookie=fc5b17570f4a5ac9 in_seq=0 out_seq=0 2024-07-26T08:40:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb102a00640 10 client.25769 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb102a00640 1 -- 172.21.15.44:0/3289194400 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5647e483b5e0 con 0x5647e4522350 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb102a00640 10 client.25769 handle_command_reply: tid=0 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 client.25769 shutdown 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 msgr2=0x5647e48dec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48dec60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1080199c0 tx=0x7fb10800f850 comp rx=0 tx=0).stop 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 shutdown_connections 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5647e4522350 0x5647e4958290 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5647e4501670 0x5647e48df2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647e44f9490 0x5647e48dec60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 --2- 172.21.15.44:0/3289194400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5647e44e0cd0 0x5647e48e37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 >> 172.21.15.44:0/3289194400 conn(0x5647e4516740 msgr2=0x5647e48c4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb102a00640 0 client.25769 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 shutdown_connections 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 1 -- 172.21.15.44:0/3289194400 wait complete. 2024-07-26T08:40:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.161+0000 7fb12a3c3740 20 client.25769 trim_cache size 0 max 16384 2024-07-26T08:40:36.168 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:40:36.168 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-07-26T08:40:36.168 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "12e9efd8-fa46-455e-80ad-7ec28b42808d", 2024-07-26T08:40:36.168 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-07-26T08:40:36.168 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:40:36.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1040b3dc0 msgr2=0x7fb1040b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1040b3dc0 0x7fb1040b62d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5647e48c5a10 tx=0x7fb110000bc0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 msgr2=0x7fb11c19c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c19c0a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb11002a910 tx=0x7fb110031a20 comp rx=0 tx=0).stop 2024-07-26T08:40:36.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 shutdown_connections 2024-07-26T08:40:36.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fb1040b3dc0 0x7fb1040b62d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb11c10ba40 0x7fb11c1a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb11c10b0f0 0x7fb11c19c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 --2- 172.21.15.44:0/1836014254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb11c1057b0 0x7fb11c19c0a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:36.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 >> 172.21.15.44:0/1836014254 conn(0x7fb11c101240 msgr2=0x7fb11c103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:36.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 shutdown_connections 2024-07-26T08:40:36.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:36.167+0000 7fb123800640 1 -- 172.21.15.44:0/1836014254 wait complete. 2024-07-26T08:40:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:36 smithi044 ceph-mon[32404]: pgmap v540: 129 pgs: 129 active+clean; 32 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.1 MiB/s wr, 64 op/s 2024-07-26T08:40:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:36 smithi044 ceph-mon[32404]: scrub queued for path: / 2024-07-26T08:40:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:36 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:36 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:36 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:36.585 INFO:tasks.workunit.client.0.smithi044.stdout:checking build system type... x86_64-pc-linux-gnu 2024-07-26T08:40:36.585 INFO:tasks.workunit.client.0.smithi044.stdout:checking host system type... x86_64-pc-linux-gnu 2024-07-26T08:40:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:36 smithi171 ceph-mon[37789]: pgmap v540: 129 pgs: 129 active+clean; 32 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.1 MiB/s wr, 64 op/s 2024-07-26T08:40:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:36 smithi171 ceph-mon[37789]: scrub queued for path: / 2024-07-26T08:40:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:36 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:36 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:36 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:36.588 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to print strings... printf 2024-07-26T08:40:36.589 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gcc... gcc 2024-07-26T08:40:36.733 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether the C compiler works... yes 2024-07-26T08:40:36.734 INFO:tasks.workunit.client.0.smithi044.stdout:checking for C compiler default output file name... a.out 2024-07-26T08:40:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:36 smithi116 ceph-mon[37097]: pgmap v540: 129 pgs: 129 active+clean; 32 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.1 MiB/s wr, 64 op/s 2024-07-26T08:40:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:36 smithi116 ceph-mon[37097]: scrub queued for path: / 2024-07-26T08:40:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:36 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:36 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:36 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:36.802 INFO:tasks.workunit.client.0.smithi044.stdout:checking for suffix of executables... 2024-07-26T08:40:36.881 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether we are cross compiling... no 2024-07-26T08:40:36.962 INFO:tasks.workunit.client.0.smithi044.stdout:checking for suffix of object files... o 2024-07-26T08:40:37.030 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether we are using the GNU C compiler... yes 2024-07-26T08:40:37.087 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether gcc accepts -g... yes 2024-07-26T08:40:37.243 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T08:40:37.258 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gcc option to accept ISO C89... none needed 2024-07-26T08:40:37.274 INFO:tasks.workunit.client.0.smithi044.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-07-26T08:40:37.289 INFO:tasks.workunit.client.0.smithi044.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-07-26T08:40:37.295 INFO:tasks.workunit.client.0.smithi044.stdout:checking for egrep... /usr/bin/grep -E 2024-07-26T08:40:37.302 INFO:tasks.workunit.client.0.smithi044.stdout:checking for fgrep... /usr/bin/grep -F 2024-07-26T08:40:37.319 INFO:tasks.workunit.client.0.smithi044.stdout:checking for ld used by gcc... /usr/bin/ld 2024-07-26T08:40:37.325 INFO:tasks.workunit.client.0.smithi044.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-07-26T08:40:37.345 INFO:tasks.workunit.client.0.smithi044.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-07-26T08:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: active paths [/] 2024-07-26T08:40:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:37 smithi044 ceph-mon[32404]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: active paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:37 smithi171 ceph-mon[37789]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.739 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:40:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: active paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:37 smithi116 ceph-mon[37097]: scrub summary: idle+waiting paths [/] 2024-07-26T08:40:38.493 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:38 smithi044 ceph-mon[32404]: pgmap v541: 129 pgs: 129 active+clean; 39 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 251 op/s 2024-07-26T08:40:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:38 smithi171 ceph-mon[37789]: pgmap v541: 129 pgs: 129 active+clean; 39 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 251 op/s 2024-07-26T08:40:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:38 smithi116 ceph-mon[37097]: pgmap v541: 129 pgs: 129 active+clean; 39 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 251 op/s 2024-07-26T08:40:40.067 INFO:tasks.workunit.client.0.smithi044.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-07-26T08:40:40.067 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether ln -s works... yes 2024-07-26T08:40:40.137 INFO:tasks.workunit.client.0.smithi044.stdout:checking the maximum length of command line arguments... 1572864 2024-07-26T08:40:40.147 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-07-26T08:40:40.147 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-07-26T08:40:40.148 INFO:tasks.workunit.client.0.smithi044.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-07-26T08:40:40.148 INFO:tasks.workunit.client.0.smithi044.stdout:checking for objdump... objdump 2024-07-26T08:40:40.148 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to recognize dependent libraries... pass_all 2024-07-26T08:40:40.149 INFO:tasks.workunit.client.0.smithi044.stdout:checking for dlltool... no 2024-07-26T08:40:40.149 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-07-26T08:40:40.150 INFO:tasks.workunit.client.0.smithi044.stdout:checking for ar... ar 2024-07-26T08:40:40.360 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:40 smithi044 ceph-mon[32404]: pgmap v542: 129 pgs: 129 active+clean; 44 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 364 op/s 2024-07-26T08:40:40.365 INFO:tasks.workunit.client.0.smithi044.stdout:checking for archiver @FILE support... @ 2024-07-26T08:40:40.366 INFO:tasks.workunit.client.0.smithi044.stdout:checking for strip... strip 2024-07-26T08:40:40.367 INFO:tasks.workunit.client.0.smithi044.stdout:checking for ranlib... ranlib 2024-07-26T08:40:40.367 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gawk... gawk 2024-07-26T08:40:40.553 INFO:tasks.workunit.client.0.smithi044.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-07-26T08:40:40.557 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sysroot... no 2024-07-26T08:40:40.573 INFO:tasks.workunit.client.0.smithi044.stdout:checking for a working dd... /usr/bin/dd 2024-07-26T08:40:40.593 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-07-26T08:40:40.646 INFO:tasks.workunit.client.0.smithi044.stdout:checking for mt... no 2024-07-26T08:40:40.667 INFO:tasks.workunit.client.0.smithi044.stdout:checking if : is a manifest tool... no 2024-07-26T08:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 -- 172.21.15.44:0/4102168889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 msgr2=0x7f4c5c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 --2- 172.21.15.44:0/4102168889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 0x7f4c5c10fea0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009be0 tx=0x7f4c5402a6d0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 -- 172.21.15.44:0/4102168889 shutdown_connections 2024-07-26T08:40:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 --2- 172.21.15.44:0/4102168889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 0x7f4c5c10fea0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 --2- 172.21.15.44:0/4102168889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 --2- 172.21.15.44:0/4102168889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 0x7f4c5c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 -- 172.21.15.44:0/4102168889 >> 172.21.15.44:0/4102168889 conn(0x7f4c5c0fee30 msgr2=0x7f4c5c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:40.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 -- 172.21.15.44:0/4102168889 shutdown_connections 2024-07-26T08:40:40.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.722+0000 7f4c63000640 1 -- 172.21.15.44:0/4102168889 wait complete. 2024-07-26T08:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.723+0000 7f4c63000640 1 Processor -- start 2024-07-26T08:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 -- start start 2024-07-26T08:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 0x7f4c5c19be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 0x7f4c5c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c113320 con 0x7f4c5c108d20 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1131a0 con 0x7f4c5c109670 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c63000640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1134a0 con 0x7f4c5c1033e0 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c61200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c61200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43028/0 (socket says 172.21.15.44:43028) 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.724+0000 7f4c61200640 1 -- 172.21.15.44:0/3730622906 learned_addr learned my addr 172.21.15.44:0/3730622906 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 0x7f4c5c19be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c62600640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 0x7f4c5c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61c00640 1 -- 172.21.15.44:0/3730622906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 msgr2=0x7f4c5c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 0x7f4c5c1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61c00640 1 -- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 msgr2=0x7f4c5c19c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c19c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61c00640 1 -- 172.21.15.44:0/3730622906 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c1a1e00 con 0x7f4c5c1033e0 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.725+0000 7f4c61200640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c19c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:40.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.726+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 0x7f4c5c19be90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00e9f0 tx=0x7f4c4c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:40.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.726+0000 7f4c5a000640 1 -- 172.21.15.44:0/3730622906 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00cd80 con 0x7f4c5c1033e0 2024-07-26T08:40:40.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.726+0000 7f4c5a000640 1 -- 172.21.15.44:0/3730622906 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c0035f0 con 0x7f4c5c1033e0 2024-07-26T08:40:40.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.726+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5c1a20f0 con 0x7f4c5c1033e0 2024-07-26T08:40:40.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.726+0000 7f4c5a000640 1 -- 172.21.15.44:0/3730622906 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c010670 con 0x7f4c5c1033e0 2024-07-26T08:40:40.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.726+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5c1a8850 con 0x7f4c5c1033e0 2024-07-26T08:40:40.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.729+0000 7f4c6b3da740 1 Processor -- start 2024-07-26T08:40:40.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 -- start start 2024-07-26T08:40:40.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9bb070 0x5604aad60000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad60540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad60a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5604aacd6bc0 con 0x5604aa99a390 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5604aacc60c0 con 0x5604aa9921b0 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5604aacca100 con 0x5604aa9bb070 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c5a000640 1 -- 172.21.15.44:0/3730622906 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f4c4c0037a0 con 0x7f4c5c1033e0 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c61200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad60a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c61200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad60a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43038/0 (socket says 172.21.15.44:43038) 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c61200640 1 -- 172.21.15.44:0/4088073687 learned_addr learned my addr 172.21.15.44:0/4088073687 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c62600640 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9bb070 0x5604aad60000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.730+0000 7f4c61c00640 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad60540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c61200640 1 -- 172.21.15.44:0/4088073687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9bb070 msgr2=0x5604aad60000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c61200640 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9bb070 0x5604aad60000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c61200640 1 -- 172.21.15.44:0/4088073687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 msgr2=0x5604aad60540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c61200640 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad60540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c61200640 1 -- 172.21.15.44:0/4088073687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604aaa26650 con 0x5604aa99a390 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c61200640 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad60a80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c50011510 tx=0x7f4c500119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8b58618284d509 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.731+0000 7f4c4aa00640 1 -- 172.21.15.44:0/4088073687 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5001e070 con 0x5604aa99a390 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c4aa00640 1 -- 172.21.15.44:0/4088073687 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c5000fa70 con 0x5604aa99a390 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c4aa00640 1 -- 172.21.15.44:0/4088073687 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50019be0 con 0x5604aa99a390 2024-07-26T08:40:40.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c5a000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4c440b3d40 0x7f4c440b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c5a000640 1 -- 172.21.15.44:0/3730622906 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4c4c0e4650 con 0x7f4c5c1033e0 2024-07-26T08:40:40.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c62600640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4c440b3d40 0x7f4c440b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c6b3da740 1 -- 172.21.15.44:0/4088073687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 msgr2=0x5604aad60a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.732+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad60a80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c50011510 tx=0x7f4c500119d0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 -- 172.21.15.44:0/4088073687 shutdown_connections 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad60a80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad60540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/4088073687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9bb070 0x5604aad60000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 -- 172.21.15.44:0/4088073687 >> 172.21.15.44:0/4088073687 conn(0x5604aa9af460 msgr2=0x5604aad5def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 -- 172.21.15.44:0/4088073687 shutdown_connections 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c6b3da740 1 -- 172.21.15.44:0/4088073687 wait complete. 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.733+0000 7f4c62600640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4c440b3d40 0x7f4c440b6250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009370 tx=0x7f4c54004a40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.734+0000 7f4c6b3da740 1 -- start start 2024-07-26T08:40:40.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.734+0000 7f4c6b3da740 1 Processor -- start 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.734+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9921b0 0x5604aad6b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.734+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad6bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.734+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad6c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5604aab29700 con 0x5604aa99a390 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5604aac62690 con 0x5604aa9bb070 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5604aad6e7e0 con 0x5604aa9921b0 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c6b3da740 10 client.0 _ll_register_callbacks cb 0x5604aac72140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad6bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad6bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43040/0 (socket says 172.21.15.44:43040) 2024-07-26T08:40:40.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 -- 172.21.15.44:0/3007016405 learned_addr learned my addr 172.21.15.44:0/3007016405 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c62600640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad6c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9921b0 0x5604aad6b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 -- 172.21.15.44:0/3007016405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9921b0 msgr2=0x5604aad6b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9921b0 0x5604aad6b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 -- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 msgr2=0x5604aad6c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad6c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c61200640 1 -- 172.21.15.44:0/3007016405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604aac79830 con 0x5604aa99a390 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.735+0000 7f4c62600640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad6c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.736+0000 7f4c61200640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad6bb40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c5001c040 tx=0x7f4c50009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc4e3f78793533e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.736+0000 7f4c3f400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.736+0000 7f4c3f400640 1 -- 172.21.15.44:0/3007016405 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5001e040 con 0x5604aa99a390 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.736+0000 7f4c3f400640 1 -- 172.21.15.44:0/3007016405 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4c5000f5c0 con 0x5604aa99a390 2024-07-26T08:40:40.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.736+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c5000fa70 con 0x5604aa99a390 2024-07-26T08:40:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.736+0000 7f4c3f400640 1 -- 172.21.15.44:0/3007016405 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4c5000f7c0 con 0x5604aa99a390 2024-07-26T08:40:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c6b3da740 10 client.15651 fetch_fsmap learned FSMap version 32 2024-07-26T08:40:40.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5604aad7a430 con 0x5604aa99a390 2024-07-26T08:40:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c3f400640 1 -- 172.21.15.44:0/3007016405 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c5000fa70 con 0x5604aa99a390 2024-07-26T08:40:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c3f400640 1 -- 172.21.15.44:0/3007016405 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f4c50021030 con 0x5604aa99a390 2024-07-26T08:40:40.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c6b3da740 10 client.15651 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c6b3da740 10 client.15651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c6b3da740 20 client.15651 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.737+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa9799f0 0x5604aad7d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.738+0000 7f4c6b3da740 4 client.15651 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.738+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604aacd8b10 con 0x5604aa9799f0 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.738+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa9799f0 0x5604aad7d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.738+0000 7f4c61c00640 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa9799f0 0x5604aad7d790 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c028636436ac8c4 server_cookie=ef12ba56b0c7e7de in_seq=0 out_seq=0 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.738+0000 7f4c3f400640 10 client.15651 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c3f400640 1 -- 172.21.15.44:0/3007016405 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604aacd8b10 con 0x5604aa9799f0 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c3f400640 10 client.15651 handle_command_reply: tid=0 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 client.15651 shutdown 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 msgr2=0x5604aad6bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad6bb40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c5001c040 tx=0x7f4c50009810 comp rx=0 tx=0).stop 2024-07-26T08:40:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 shutdown_connections 2024-07-26T08:40:40.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa9799f0 0x5604aad7d790 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad6c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa99a390 0x5604aad6bb40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.739+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/3007016405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa9921b0 0x5604aad6b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.740+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 >> 172.21.15.44:0/3007016405 conn(0x5604aa9af460 msgr2=0x5604aad63190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.740+0000 7f4c3f400640 0 client.15651 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.740+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 shutdown_connections 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.740+0000 7f4c6b3da740 1 -- 172.21.15.44:0/3007016405 wait complete. 2024-07-26T08:40:40.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.740+0000 7f4c6b3da740 20 client.15651 trim_cache size 0 max 16384 2024-07-26T08:40:40.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 Processor -- start 2024-07-26T08:40:40.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 -- start start 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad5cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad6d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9bb070 0x5604aad82e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1b2990 con 0x5604aa9bb070 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1b4590 con 0x5604aa9921b0 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.759+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1b3060 con 0x5604aa99a390 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad5cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c62600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9bb070 0x5604aad82e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad5cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.44:58440/0 (socket says 172.21.15.44:58440) 2024-07-26T08:40:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 -- 172.21.15.44:0/2746544742 learned_addr learned my addr 172.21.15.44:0/2746544742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:40.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61200640 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad6d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 -- 172.21.15.44:0/2746544742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 msgr2=0x5604aad6d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad6d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 -- 172.21.15.44:0/2746544742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9bb070 msgr2=0x5604aad82e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9bb070 0x5604aad82e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 -- 172.21.15.44:0/2746544742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604aad6a560 con 0x5604aa9921b0 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c62600640 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9bb070 0x5604aad82e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T08:40:40.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.760+0000 7f4c61c00640 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad5cef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00cfd0 tx=0x7f4c4c0e6aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4ec2e5d2813b9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.761+0000 7f4c4a000640 1 -- 172.21.15.44:0/2746544742 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0aa980 con 0x5604aa9921b0 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.761+0000 7f4c4a000640 1 -- 172.21.15.44:0/2746544742 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c0ab570 con 0x5604aa9921b0 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.761+0000 7f4c4a000640 1 -- 172.21.15.44:0/2746544742 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0a7100 con 0x5604aa9921b0 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 -- 172.21.15.44:0/2746544742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 msgr2=0x5604aad5cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad5cef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00cfd0 tx=0x7f4c4c0e6aa0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 -- 172.21.15.44:0/2746544742 shutdown_connections 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad6d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9bb070 0x5604aad82e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/2746544742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9921b0 0x5604aad5cef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 -- 172.21.15.44:0/2746544742 >> 172.21.15.44:0/2746544742 conn(0x5604aa9af460 msgr2=0x5604aad6cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 -- 172.21.15.44:0/2746544742 shutdown_connections 2024-07-26T08:40:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 -- 172.21.15.44:0/2746544742 wait complete. 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.762+0000 7f4c6b3da740 1 -- start start 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 Processor -- start 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9921b0 0x5604aad6ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad70520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad70a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1b3e00 con 0x5604aa9921b0 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1b3730 con 0x5604aa9bb070 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c1b0970 con 0x5604aa99a390 2024-07-26T08:40:40.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c6b3da740 10 client.0 _ll_register_callbacks cb 0x7f4c50005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c61200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad70520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c61c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9921b0 0x5604aad6ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c61200640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad70520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:39854/0 (socket says 172.21.15.44:39854) 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.763+0000 7f4c61200640 1 -- 172.21.15.44:0/1411201350 learned_addr learned my addr 172.21.15.44:0/1411201350 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c62600640 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad70a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c61c00640 1 -- 172.21.15.44:0/1411201350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 msgr2=0x5604aad70520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c61c00640 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad70520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c61c00640 1 -- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 msgr2=0x5604aad70a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c61c00640 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad70a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c61c00640 1 -- 172.21.15.44:0/1411201350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604aade3fa0 con 0x5604aa9921b0 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c61c00640 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9921b0 0x5604aad6ffe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0ab030 tx=0x7f4c4c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26d618f6c8fe65 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:40.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.764+0000 7f4c3f400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:40:40.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c3f400640 1 -- 172.21.15.44:0/1411201350 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c003bc0 con 0x5604aa9921b0 2024-07-26T08:40:40.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c3f400640 1 -- 172.21.15.44:0/1411201350 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9431+0+0 (secure 0 0 0) 0x7f4c4c0e1c30 con 0x5604aa9921b0 2024-07-26T08:40:40.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c4c0ab570 con 0x5604aa9921b0 2024-07-26T08:40:40.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c3f400640 1 -- 172.21.15.44:0/1411201350 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4c4c0aae70 con 0x5604aa9921b0 2024-07-26T08:40:40.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c6b3da740 10 client.15657 fetch_fsmap learned FSMap version 32 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5604aade5230 con 0x5604aa9921b0 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.765+0000 7f4c3f400640 1 -- 172.21.15.44:0/1411201350 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c4c0ab570 con 0x5604aa9921b0 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c3f400640 1 -- 172.21.15.44:0/1411201350 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 32) ==== 8386+0+0 (secure 0 0 0) 0x7f4c4c0b3b20 con 0x5604aa9921b0 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c6b3da740 10 client.15657 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c6b3da740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c6b3da740 20 client.15657 populate_metadata read hostname 'smithi044' 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa94eda0 0x5604aade7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c6b3da740 4 client.15657 mds_command: new command op to 24701 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 --> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5604aad5e350 con 0x5604aa94eda0 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c61200640 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa94eda0 0x5604aade7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c61200640 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa94eda0 0x5604aade7860 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5370519a983ccdf server_cookie=a5b5359fa0d4d5c5 in_seq=0 out_seq=0 2024-07-26T08:40:40.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.766+0000 7f4c3f400640 10 client.15657 ms_handle_connect on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c3f400640 1 -- 172.21.15.44:0/1411201350 <== mds.0 v2:172.21.15.116:6838/3153223603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5604aad5e350 con 0x5604aa94eda0 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c3f400640 10 client.15657 handle_command_reply: tid=0 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 client.15657 shutdown 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9921b0 msgr2=0x5604aad6ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9921b0 0x5604aad6ffe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0ab030 tx=0x7f4c4c009540 comp rx=0 tx=0).stop 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 shutdown_connections 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:6838/3153223603,v1:172.21.15.116:6839/3153223603] conn(0x5604aa94eda0 0x5604aade7860 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5604aa99a390 0x5604aad70520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604aa9bb070 0x5604aad70a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 --2- 172.21.15.44:0/1411201350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5604aa9921b0 0x5604aad6ffe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 >> 172.21.15.44:0/1411201350 conn(0x5604aa9af460 msgr2=0x5604aad5ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c3f400640 0 client.15657 ms_handle_reset on v2:172.21.15.116:6838/3153223603 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 shutdown_connections 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 1 -- 172.21.15.44:0/1411201350 wait complete. 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.767+0000 7f4c6b3da740 20 client.15657 trim_cache size 0 max 16384 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout: "12e9efd8-fa46-455e-80ad-7ec28b42808d": { 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "12e9efd8-fa46-455e-80ad-7ec28b42808d", 2024-07-26T08:40:40.772 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-07-26T08:40:40.773 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-07-26T08:40:40.773 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-07-26T08:40:40.773 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-07-26T08:40:40.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4c440b3d40 msgr2=0x7f4c440b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4c440b3d40 0x7f4c440b6250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4c54009370 tx=0x7f4c54004a40 comp rx=0 tx=0).stop 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 msgr2=0x7f4c5c19be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 0x7f4c5c19be90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00e9f0 tx=0x7f4c4c00eeb0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 shutdown_connections 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f4c440b3d40 0x7f4c440b6250 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c5c109670 0x7f4c5c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c5c108d20 0x7f4c5c19c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 --2- 172.21.15.44:0/3730622906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c5c1033e0 0x7f4c5c19be90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 >> 172.21.15.44:0/3730622906 conn(0x7f4c5c0fee30 msgr2=0x7f4c5c107320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.768+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 shutdown_connections 2024-07-26T08:40:40.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:40:40.769+0000 7f4c63000640 1 -- 172.21.15.44:0/3730622906 wait complete. 2024-07-26T08:40:40.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:40 smithi116 ceph-mon[37097]: pgmap v542: 129 pgs: 129 active+clean; 44 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 364 op/s 2024-07-26T08:40:40.798 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to run the C preprocessor... gcc -E 2024-07-26T08:40:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:40 smithi171 ceph-mon[37789]: pgmap v542: 129 pgs: 129 active+clean; 44 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 364 op/s 2024-07-26T08:40:41.194 INFO:tasks.workunit.client.0.smithi044.stdout:checking for ANSI C header files... yes 2024-07-26T08:40:41.281 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/types.h... yes 2024-07-26T08:40:41.371 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/stat.h... yes 2024-07-26T08:40:41.466 INFO:tasks.workunit.client.0.smithi044.stdout:checking for stdlib.h... yes 2024-07-26T08:40:41.560 INFO:tasks.workunit.client.0.smithi044.stdout:checking for string.h... yes 2024-07-26T08:40:41.691 INFO:tasks.workunit.client.0.smithi044.stdout:checking for memory.h... yes 2024-07-26T08:40:41.783 INFO:tasks.cephfs.filesystem:scrub status for tag:12e9efd8-fa46-455e-80ad-7ec28b42808d - {'path': '/', 'tag': '12e9efd8-fa46-455e-80ad-7ec28b42808d', 'options': 'recursive,force'} 2024-07-26T08:40:41.789 INFO:tasks.workunit.client.0.smithi044.stdout:checking for strings.h... yes 2024-07-26T08:40:41.882 INFO:tasks.workunit.client.0.smithi044.stdout:checking for inttypes.h... yes 2024-07-26T08:40:41.978 INFO:tasks.workunit.client.0.smithi044.stdout:checking for stdint.h... yes 2024-07-26T08:40:42.078 INFO:tasks.workunit.client.0.smithi044.stdout:checking for unistd.h... yes 2024-07-26T08:40:42.187 INFO:tasks.workunit.client.0.smithi044.stdout:checking for dlfcn.h... yes 2024-07-26T08:40:42.206 INFO:tasks.workunit.client.0.smithi044.stdout:checking for objdir... .libs 2024-07-26T08:40:42.462 INFO:tasks.workunit.client.0.smithi044.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-07-26T08:40:42.462 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-07-26T08:40:42.552 INFO:tasks.workunit.client.0.smithi044.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-07-26T08:40:42.642 INFO:tasks.workunit.client.0.smithi044.stdout:checking if gcc static flag -static works... no 2024-07-26T08:40:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:42 smithi044 ceph-mon[32404]: pgmap v543: 129 pgs: 129 active+clean; 45 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 384 op/s 2024-07-26T08:40:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:42 smithi116 ceph-mon[37097]: pgmap v543: 129 pgs: 129 active+clean; 45 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 384 op/s 2024-07-26T08:40:42.777 INFO:tasks.workunit.client.0.smithi044.stdout:checking if gcc supports -c -o file.o... yes 2024-07-26T08:40:42.777 INFO:tasks.workunit.client.0.smithi044.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-07-26T08:40:42.801 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-07-26T08:40:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:42 smithi171 ceph-mon[37789]: pgmap v543: 129 pgs: 129 active+clean; 45 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 384 op/s 2024-07-26T08:40:42.920 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether -lc should be explicitly linked in... no 2024-07-26T08:40:43.092 INFO:tasks.workunit.client.0.smithi044.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-07-26T08:40:43.092 INFO:tasks.workunit.client.0.smithi044.stdout:checking how to hardcode library paths into programs... immediate 2024-07-26T08:40:43.097 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether stripping libraries is possible... yes 2024-07-26T08:40:43.097 INFO:tasks.workunit.client.0.smithi044.stdout:checking if libtool supports shared libraries... yes 2024-07-26T08:40:43.097 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether to build shared libraries... yes 2024-07-26T08:40:43.097 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether to build static libraries... yes 2024-07-26T08:40:43.114 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gcc... (cached) gcc 2024-07-26T08:40:43.196 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-07-26T08:40:43.196 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether gcc accepts -g... (cached) yes 2024-07-26T08:40:43.196 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-07-26T08:40:43.197 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gmake... /usr/bin/gmake 2024-07-26T08:40:43.197 INFO:tasks.workunit.client.0.smithi044.stderr:./configure: line 12082: test: $(SHELL): binary operator expected 2024-07-26T08:40:43.198 INFO:tasks.workunit.client.0.smithi044.stdout:checking for tar... /bin/tar 2024-07-26T08:40:43.199 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gzip... /bin/gzip 2024-07-26T08:40:43.235 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether gcc -MM is supported... yes 2024-07-26T08:40:43.237 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sort... /bin/sort 2024-07-26T08:40:43.237 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether ln -s works... yes 2024-07-26T08:40:43.237 INFO:tasks.workunit.client.0.smithi044.stdout:checking for rpm... /bin/rpm 2024-07-26T08:40:43.248 INFO:tasks.workunit.client.0.smithi044.stdout:checking for rpmbuild... no 2024-07-26T08:40:43.521 INFO:tasks.workunit.client.0.smithi044.stdout:checking assert.h usability... yes 2024-07-26T08:40:43.640 INFO:tasks.workunit.client.0.smithi044.stdout:checking assert.h presence... yes 2024-07-26T08:40:43.640 INFO:tasks.workunit.client.0.smithi044.stdout:checking for assert.h... yes 2024-07-26T08:40:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:43 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:40:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:43 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:40:43.769 INFO:tasks.workunit.client.0.smithi044.stdout:checking bstring.h usability... no 2024-07-26T08:40:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:43 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:40:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:43 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:40:43.834 INFO:tasks.workunit.client.0.smithi044.stdout:checking bstring.h presence... no 2024-07-26T08:40:43.834 INFO:tasks.workunit.client.0.smithi044.stdout:checking for bstring.h... no 2024-07-26T08:40:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:43 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:40:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:43 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:40:43.929 INFO:tasks.workunit.client.0.smithi044.stdout:checking libgen.h usability... yes 2024-07-26T08:40:43.967 INFO:tasks.workunit.client.0.smithi044.stdout:checking libgen.h presence... yes 2024-07-26T08:40:43.967 INFO:tasks.workunit.client.0.smithi044.stdout:checking for libgen.h... yes 2024-07-26T08:40:44.088 INFO:tasks.workunit.client.0.smithi044.stdout:checking dirent.h usability... yes 2024-07-26T08:40:44.129 INFO:tasks.workunit.client.0.smithi044.stdout:checking dirent.h presence... yes 2024-07-26T08:40:44.129 INFO:tasks.workunit.client.0.smithi044.stdout:checking for dirent.h... yes 2024-07-26T08:40:44.243 INFO:tasks.workunit.client.0.smithi044.stdout:checking errno.h usability... yes 2024-07-26T08:40:44.287 INFO:tasks.workunit.client.0.smithi044.stdout:checking errno.h presence... yes 2024-07-26T08:40:44.287 INFO:tasks.workunit.client.0.smithi044.stdout:checking for errno.h... yes 2024-07-26T08:40:44.398 INFO:tasks.workunit.client.0.smithi044.stdout:checking malloc.h usability... yes 2024-07-26T08:40:44.443 INFO:tasks.workunit.client.0.smithi044.stdout:checking malloc.h presence... yes 2024-07-26T08:40:44.443 INFO:tasks.workunit.client.0.smithi044.stdout:checking for malloc.h... yes 2024-07-26T08:40:44.562 INFO:tasks.workunit.client.0.smithi044.stdout:checking uuid.h usability... no 2024-07-26T08:40:44.633 INFO:tasks.workunit.client.0.smithi044.stdout:checking uuid.h presence... no 2024-07-26T08:40:44.633 INFO:tasks.workunit.client.0.smithi044.stdout:checking for uuid.h... no 2024-07-26T08:40:44.744 INFO:tasks.workunit.client.0.smithi044.stdout:checking uuid/uuid.h usability... yes 2024-07-26T08:40:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:44 smithi044 ceph-mon[32404]: pgmap v544: 129 pgs: 129 active+clean; 50 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 471 op/s 2024-07-26T08:40:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:44 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:40:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:44 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:40:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:44 smithi044 ceph-mon[32404]: scrub summary: idle 2024-07-26T08:40:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:44 smithi116 ceph-mon[37097]: pgmap v544: 129 pgs: 129 active+clean; 50 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 471 op/s 2024-07-26T08:40:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:44 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:40:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:44 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:40:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:44 smithi116 ceph-mon[37097]: scrub summary: idle 2024-07-26T08:40:44.793 INFO:tasks.workunit.client.0.smithi044.stdout:checking uuid/uuid.h presence... yes 2024-07-26T08:40:44.793 INFO:tasks.workunit.client.0.smithi044.stdout:checking for uuid/uuid.h... yes 2024-07-26T08:40:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:44 smithi171 ceph-mon[37789]: pgmap v544: 129 pgs: 129 active+clean; 50 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 471 op/s 2024-07-26T08:40:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:44 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:40:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:44 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:40:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:44 smithi171 ceph-mon[37789]: scrub summary: idle 2024-07-26T08:40:44.908 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/uuid.h usability... no 2024-07-26T08:40:44.974 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/uuid.h presence... no 2024-07-26T08:40:44.975 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/uuid.h... no 2024-07-26T08:40:45.087 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/file.h usability... yes 2024-07-26T08:40:45.127 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/file.h presence... yes 2024-07-26T08:40:45.127 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/file.h... yes 2024-07-26T08:40:45.264 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/fcntl.h usability... yes 2024-07-26T08:40:45.302 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/fcntl.h presence... yes 2024-07-26T08:40:45.303 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/fcntl.h... yes 2024-07-26T08:40:45.417 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/syssgi.h usability... no 2024-07-26T08:40:45.483 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/syssgi.h presence... no 2024-07-26T08:40:45.483 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/syssgi.h... no 2024-07-26T08:40:45.608 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/param.h usability... yes 2024-07-26T08:40:45.655 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/param.h presence... yes 2024-07-26T08:40:45.655 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/param.h... yes 2024-07-26T08:40:45.691 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/stat.h... (cached) yes 2024-07-26T08:40:45.785 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/statvfs.h usability... yes 2024-07-26T08:40:45.825 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/statvfs.h presence... yes 2024-07-26T08:40:45.826 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/statvfs.h... yes 2024-07-26T08:40:45.937 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/time.h usability... yes 2024-07-26T08:40:45.973 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/time.h presence... yes 2024-07-26T08:40:45.973 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/time.h... yes 2024-07-26T08:40:46.073 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/ioctl.h usability... yes 2024-07-26T08:40:46.112 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/ioctl.h presence... yes 2024-07-26T08:40:46.112 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/ioctl.h... yes 2024-07-26T08:40:46.233 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/wait.h usability... yes 2024-07-26T08:40:46.276 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/wait.h presence... yes 2024-07-26T08:40:46.277 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/wait.h... yes 2024-07-26T08:40:46.312 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/types.h... (cached) yes 2024-07-26T08:40:46.333 INFO:tasks.workunit.client.0.smithi044.stdout:checking for strings.h... (cached) yes 2024-07-26T08:40:46.425 INFO:tasks.workunit.client.0.smithi044.stdout:checking err.h usability... yes 2024-07-26T08:40:46.465 INFO:tasks.workunit.client.0.smithi044.stdout:checking err.h presence... yes 2024-07-26T08:40:46.465 INFO:tasks.workunit.client.0.smithi044.stdout:checking for err.h... yes 2024-07-26T08:40:46.578 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/falloc.h usability... yes 2024-07-26T08:40:46.614 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/falloc.h presence... yes 2024-07-26T08:40:46.615 INFO:tasks.workunit.client.0.smithi044.stdout:checking for linux/falloc.h... yes 2024-07-26T08:40:46.725 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-07-26T08:40:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:46 smithi044 ceph-mon[32404]: pgmap v545: 129 pgs: 129 active+clean; 51 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 2.3 MiB/s wr, 484 op/s 2024-07-26T08:40:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:46 smithi116 ceph-mon[37097]: pgmap v545: 129 pgs: 129 active+clean; 51 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 2.3 MiB/s wr, 484 op/s 2024-07-26T08:40:46.787 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-07-26T08:40:46.788 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/fs/xfs_fsops.h... no 2024-07-26T08:40:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:46 smithi171 ceph-mon[37789]: pgmap v545: 129 pgs: 129 active+clean; 51 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 2.3 MiB/s wr, 484 op/s 2024-07-26T08:40:46.885 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/fs/xfs_itable.h usability... no 2024-07-26T08:40:46.950 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/fs/xfs_itable.h presence... no 2024-07-26T08:40:46.950 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/fs/xfs_itable.h... no 2024-07-26T08:40:47.051 INFO:tasks.workunit.client.0.smithi044.stdout:checking xfs/platform_defs.h usability... no 2024-07-26T08:40:47.117 INFO:tasks.workunit.client.0.smithi044.stdout:checking xfs/platform_defs.h presence... no 2024-07-26T08:40:47.117 INFO:tasks.workunit.client.0.smithi044.stdout:checking for xfs/platform_defs.h... no 2024-07-26T08:40:47.223 INFO:tasks.workunit.client.0.smithi044.stdout:checking btrfs/ioctl.h usability... no 2024-07-26T08:40:47.293 INFO:tasks.workunit.client.0.smithi044.stdout:checking btrfs/ioctl.h presence... no 2024-07-26T08:40:47.293 INFO:tasks.workunit.client.0.smithi044.stdout:checking for btrfs/ioctl.h... no 2024-07-26T08:40:47.401 INFO:tasks.workunit.client.0.smithi044.stdout:checking cifs/ioctl.h usability... no 2024-07-26T08:40:47.467 INFO:tasks.workunit.client.0.smithi044.stdout:checking cifs/ioctl.h presence... no 2024-07-26T08:40:47.468 INFO:tasks.workunit.client.0.smithi044.stdout:checking for cifs/ioctl.h... no 2024-07-26T08:40:47.574 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/mman.h usability... yes 2024-07-26T08:40:47.614 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/mman.h presence... yes 2024-07-26T08:40:47.614 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/mman.h... yes 2024-07-26T08:40:47.732 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/ext4.h usability... no 2024-07-26T08:40:47.796 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/ext4.h presence... no 2024-07-26T08:40:47.797 INFO:tasks.workunit.client.0.smithi044.stdout:checking for linux/ext4.h... no 2024-07-26T08:40:47.867 INFO:tasks.workunit.client.0.smithi044.stdout:checking for xfs/xfs_log_format.h... no 2024-07-26T08:40:47.997 INFO:tasks.workunit.client.0.smithi044.stdout:checking for uuid_compare... no 2024-07-26T08:40:48.088 INFO:tasks.workunit.client.0.smithi044.stdout:checking for uuid_compare in -luuid... yes 2024-07-26T08:40:48.223 INFO:tasks.workunit.client.0.smithi044.stdout:checking for xfs/xfs.h... yes 2024-07-26T08:40:48.295 INFO:tasks.workunit.client.0.smithi044.stdout:checking for xfs/libxfs.h... no 2024-07-26T08:40:48.373 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether xlog_assign_lsn is declared... no 2024-07-26T08:40:48.474 INFO:tasks.workunit.client.0.smithi044.stdout:checking for xfs/xqm.h... yes 2024-07-26T08:40:48.577 INFO:tasks.workunit.client.0.smithi044.stdout:checking xfsctl from xfs/xfs.h... ok 2024-07-26T08:40:48.656 INFO:tasks.workunit.client.0.smithi044.stdout:checking xfs/handle.h usability... yes 2024-07-26T08:40:48.694 INFO:tasks.workunit.client.0.smithi044.stdout:checking xfs/handle.h presence... yes 2024-07-26T08:40:48.694 INFO:tasks.workunit.client.0.smithi044.stdout:checking for xfs/handle.h... yes 2024-07-26T08:40:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:48 smithi044 ceph-mon[32404]: pgmap v546: 129 pgs: 129 active+clean; 52 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.2 MiB/s wr, 478 op/s 2024-07-26T08:40:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:48 smithi116 ceph-mon[37097]: pgmap v546: 129 pgs: 129 active+clean; 52 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.2 MiB/s wr, 478 op/s 2024-07-26T08:40:48.798 INFO:tasks.workunit.client.0.smithi044.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-07-26T08:40:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:48 smithi171 ceph-mon[37789]: pgmap v546: 129 pgs: 129 active+clean; 52 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 2.2 MiB/s wr, 478 op/s 2024-07-26T08:40:48.883 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/acl.h usability... yes 2024-07-26T08:40:48.922 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/acl.h presence... yes 2024-07-26T08:40:48.922 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/acl.h... yes 2024-07-26T08:40:49.014 INFO:tasks.workunit.client.0.smithi044.stdout:checking acl/libacl.h usability... yes 2024-07-26T08:40:49.056 INFO:tasks.workunit.client.0.smithi044.stdout:checking acl/libacl.h presence... yes 2024-07-26T08:40:49.057 INFO:tasks.workunit.client.0.smithi044.stdout:checking for acl/libacl.h... yes 2024-07-26T08:40:49.157 INFO:tasks.workunit.client.0.smithi044.stdout:checking for acl_init in -lacl... yes 2024-07-26T08:40:49.275 INFO:tasks.workunit.client.0.smithi044.stdout:checking gdbm-ndbm.h usability... no 2024-07-26T08:40:49.344 INFO:tasks.workunit.client.0.smithi044.stdout:checking gdbm-ndbm.h presence... no 2024-07-26T08:40:49.345 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gdbm-ndbm.h... no 2024-07-26T08:40:49.431 INFO:tasks.workunit.client.0.smithi044.stdout:checking gdbm/ndbm.h usability... no 2024-07-26T08:40:49.495 INFO:tasks.workunit.client.0.smithi044.stdout:checking gdbm/ndbm.h presence... no 2024-07-26T08:40:49.495 INFO:tasks.workunit.client.0.smithi044.stdout:checking for gdbm/ndbm.h... no 2024-07-26T08:40:49.581 INFO:tasks.workunit.client.0.smithi044.stdout:checking ndbm.h usability... no 2024-07-26T08:40:49.646 INFO:tasks.workunit.client.0.smithi044.stdout:checking ndbm.h presence... no 2024-07-26T08:40:49.646 INFO:tasks.workunit.client.0.smithi044.stdout:checking for ndbm.h... no 2024-07-26T08:40:49.723 INFO:tasks.workunit.client.0.smithi044.stdout:checking libaio.h usability... yes 2024-07-26T08:40:49.767 INFO:tasks.workunit.client.0.smithi044.stdout:checking libaio.h presence... yes 2024-07-26T08:40:49.767 INFO:tasks.workunit.client.0.smithi044.stdout:checking for libaio.h... yes 2024-07-26T08:40:49.866 INFO:tasks.workunit.client.0.smithi044.stdout:checking liburing.h usability... no 2024-07-26T08:40:49.928 INFO:tasks.workunit.client.0.smithi044.stdout:checking liburing.h presence... no 2024-07-26T08:40:49.928 INFO:tasks.workunit.client.0.smithi044.stdout:checking for liburing.h... no 2024-07-26T08:40:50.014 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/capability.h usability... no 2024-07-26T08:40:50.077 INFO:tasks.workunit.client.0.smithi044.stdout:checking sys/capability.h presence... no 2024-07-26T08:40:50.077 INFO:tasks.workunit.client.0.smithi044.stdout:checking for sys/capability.h... no 2024-07-26T08:40:50.157 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/fiemap.h usability... yes 2024-07-26T08:40:50.194 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/fiemap.h presence... yes 2024-07-26T08:40:50.194 INFO:tasks.workunit.client.0.smithi044.stdout:checking for linux/fiemap.h... yes 2024-07-26T08:40:50.294 INFO:tasks.workunit.client.0.smithi044.stdout:checking for fallocate... yes 2024-07-26T08:40:50.406 INFO:tasks.workunit.client.0.smithi044.stdout:checking for open_by_handle_at... yes 2024-07-26T08:40:50.493 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/fs.h usability... yes 2024-07-26T08:40:50.534 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/fs.h presence... yes 2024-07-26T08:40:50.534 INFO:tasks.workunit.client.0.smithi044.stdout:checking for linux/fs.h... yes 2024-07-26T08:40:50.633 INFO:tasks.workunit.client.0.smithi044.stdout:checking btrfsutil.h usability... no 2024-07-26T08:40:50.703 INFO:tasks.workunit.client.0.smithi044.stdout:checking btrfsutil.h presence... no 2024-07-26T08:40:50.704 INFO:tasks.workunit.client.0.smithi044.stdout:checking for btrfsutil.h... no 2024-07-26T08:40:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:50 smithi044 ceph-mon[32404]: pgmap v547: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.7 MiB/s wr, 307 op/s 2024-07-26T08:40:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:50 smithi116 ceph-mon[37097]: pgmap v547: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.7 MiB/s wr, 307 op/s 2024-07-26T08:40:50.804 INFO:tasks.workunit.client.0.smithi044.stdout:checking for copy_file_range... yes 2024-07-26T08:40:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:50 smithi171 ceph-mon[37789]: pgmap v547: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.7 MiB/s wr, 307 op/s 2024-07-26T08:40:50.908 INFO:tasks.workunit.client.0.smithi044.stdout:checking for SEEK_DATA... yes 2024-07-26T08:40:51.035 INFO:tasks.workunit.client.0.smithi044.stdout:checking for BMV_OF_SHARED... yes 2024-07-26T08:40:51.144 INFO:tasks.workunit.client.0.smithi044.stdout:checking for nftw... yes 2024-07-26T08:40:51.258 INFO:tasks.workunit.client.0.smithi044.stdout:checking for RLIMIT_NOFILE... yes 2024-07-26T08:40:51.358 INFO:tasks.workunit.client.0.smithi044.stdout:checking for XFS_IOC_EXCHANGE_RANGE... no 2024-07-26T08:40:51.485 INFO:tasks.workunit.client.0.smithi044.stdout:checking for FICLONE... yes 2024-07-26T08:40:51.587 INFO:tasks.workunit.client.0.smithi044.stdout:checking for renameat2... yes 2024-07-26T08:40:51.688 INFO:tasks.workunit.client.0.smithi044.stdout:checking for reallocarray... yes 2024-07-26T08:40:51.834 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct mount_attr... yes 2024-07-26T08:40:51.984 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_qgroup_limit... yes 2024-07-26T08:40:52.123 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_qgroup_inherit... yes 2024-07-26T08:40:52.264 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_ioctl_vol_args... yes 2024-07-26T08:40:52.407 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2024-07-26T08:40:52.551 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2024-07-26T08:40:52.692 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2024-07-26T08:40:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:52 smithi044 ceph-mon[32404]: pgmap v548: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 1.2 MiB/s wr, 184 op/s 2024-07-26T08:40:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:52 smithi116 ceph-mon[37097]: pgmap v548: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 1.2 MiB/s wr, 184 op/s 2024-07-26T08:40:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:52 smithi171 ceph-mon[37789]: pgmap v548: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 1.2 MiB/s wr, 184 op/s 2024-07-26T08:40:52.838 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2024-07-26T08:40:52.936 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/btrfs.h usability... yes 2024-07-26T08:40:52.976 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/btrfs.h presence... yes 2024-07-26T08:40:52.976 INFO:tasks.workunit.client.0.smithi044.stdout:checking for linux/btrfs.h... yes 2024-07-26T08:40:53.086 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/btrfs_tree.h usability... yes 2024-07-26T08:40:53.126 INFO:tasks.workunit.client.0.smithi044.stdout:checking linux/btrfs_tree.h presence... yes 2024-07-26T08:40:53.126 INFO:tasks.workunit.client.0.smithi044.stdout:checking for linux/btrfs_tree.h... yes 2024-07-26T08:40:53.216 INFO:tasks.workunit.client.0.smithi044.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... yes 2024-07-26T08:40:53.299 INFO:tasks.workunit.client.0.smithi044.stdout:checking whether BTRFS_IOC_SNAP_DESTROY_V2 is declared... yes 2024-07-26T08:40:53.341 INFO:tasks.workunit.client.0.smithi044.stdout:configure: creating ./config.status 2024-07-26T08:40:54.609 INFO:tasks.workunit.client.0.smithi044.stdout:config.status: creating include/builddefs 2024-07-26T08:40:54.675 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-mon[37097]: pgmap v549: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 167 op/s 2024-07-26T08:40:54.716 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:54 smithi044 ceph-mon[32404]: pgmap v549: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 167 op/s 2024-07-26T08:40:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:54 smithi171 ceph-mon[37789]: pgmap v549: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 167 op/s 2024-07-26T08:40:55.023 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:40:55.023 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: *** Caught signal (Aborted) ** 2024-07-26T08:40:55.023 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: in thread 7f91ad3f8640 thread_name:ms_dispatch 2024-07-26T08:40:55.023 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:40:55.023 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f91b27cf6f0] 2024-07-26T08:40:55.023 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f91b281c94c] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 3: raise() 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 4: abort() 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 5: /lib64/libc.so.6(+0x2871b) [0x7f91b27b971b] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 6: /lib64/libc.so.6(+0x37386) [0x7f91b27c8386] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b08cb1d6b9] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b08ccdb80c] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b08cc29cde] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b08cbeb763] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b08cbeca1b] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b08cbd0752] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b08cb1ea56] 2024-07-26T08:40:55.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b08cb22637] 2024-07-26T08:40:55.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b08cb22f5e] 2024-07-26T08:40:55.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b08cb0603a] 2024-07-26T08:40:55.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f91b309c407] 2024-07-26T08:40:55.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 18: (DispatchQueue::entry()+0x60f) [0x7f91b309ccf9] 2024-07-26T08:40:55.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f91b31795a1] 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 20: (Thread::entry_wrapper()+0x43) [0x7f91b2f26cdd] 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 21: (Thread::_entry_func(void*)+0xd) [0x7f91b2f26cf9] 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 22: /lib64/libc.so.6(+0x89c02) [0x7f91b281ac02] 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f91b289fc40] 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:54.674+0000 7f91ad3f8640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: in thread 7f91ad3f8640 thread_name:ms_dispatch 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:40:55.026 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f91b27cf6f0] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f91b281c94c] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 3: raise() 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 4: abort() 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 5: /lib64/libc.so.6(+0x2871b) [0x7f91b27b971b] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 6: /lib64/libc.so.6(+0x37386) [0x7f91b27c8386] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b08cb1d6b9] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b08ccdb80c] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b08cc29cde] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b08cbeb763] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b08cbeca1b] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b08cbd0752] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b08cb1ea56] 2024-07-26T08:40:55.027 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b08cb22637] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b08cb22f5e] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b08cb0603a] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f91b309c407] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 18: (DispatchQueue::entry()+0x60f) [0x7f91b309ccf9] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f91b31795a1] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 20: (Thread::entry_wrapper()+0x43) [0x7f91b2f26cdd] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 21: (Thread::_entry_func(void*)+0xd) [0x7f91b2f26cf9] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 22: /lib64/libc.so.6(+0x89c02) [0x7f91b281ac02] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f91b289fc40] 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:55.028 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 0> 2024-07-26T08:40:54.674+0000 7f91ad3f8640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: in thread 7f91ad3f8640 thread_name:ms_dispatch 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f91b27cf6f0] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f91b281c94c] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 3: raise() 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 4: abort() 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 5: /lib64/libc.so.6(+0x2871b) [0x7f91b27b971b] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 6: /lib64/libc.so.6(+0x37386) [0x7f91b27c8386] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b08cb1d6b9] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b08ccdb80c] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b08cc29cde] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b08cbeb763] 2024-07-26T08:40:55.029 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b08cbeca1b] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b08cbd0752] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b08cb1ea56] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b08cb22637] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b08cb22f5e] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b08cb0603a] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f91b309c407] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 18: (DispatchQueue::entry()+0x60f) [0x7f91b309ccf9] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f91b31795a1] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 20: (Thread::entry_wrapper()+0x43) [0x7f91b2f26cdd] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 21: (Thread::_entry_func(void*)+0xd) [0x7f91b2f26cf9] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 22: /lib64/libc.so.6(+0x89c02) [0x7f91b281ac02] 2024-07-26T08:40:55.030 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f91b289fc40] 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 0> 2024-07-26T08:40:54.674+0000 7f91ad3f8640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: in thread 7f91ad3f8640 thread_name:ms_dispatch 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f91b27cf6f0] 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f91b281c94c] 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 3: raise() 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 4: abort() 2024-07-26T08:40:55.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 5: /lib64/libc.so.6(+0x2871b) [0x7f91b27b971b] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 6: /lib64/libc.so.6(+0x37386) [0x7f91b27c8386] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b08cb1d6b9] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b08ccdb80c] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b08cc29cde] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b08cbeb763] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b08cbeca1b] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b08cbd0752] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b08cb1ea56] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b08cb22637] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b08cb22f5e] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b08cb0603a] 2024-07-26T08:40:55.032 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f91b309c407] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 18: (DispatchQueue::entry()+0x60f) [0x7f91b309ccf9] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f91b31795a1] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 20: (Thread::entry_wrapper()+0x43) [0x7f91b2f26cdd] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 21: (Thread::_entry_func(void*)+0xd) [0x7f91b2f26cf9] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 22: /lib64/libc.so.6(+0x89c02) [0x7f91b281ac02] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f91b289fc40] 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[68230]: 2024-07-26T08:40:55.033 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:54 smithi116 podman[70814]: 2024-07-26 08:40:54.734525675 +0000 UTC m=+0.019825211 container died bad20a32fc44c967ac13339e65d4d1c81ca77772ec2afa5550a4802bb4949513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:40:55.659 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:55 smithi116 podman[70814]: 2024-07-26 08:40:55.408337978 +0000 UTC m=+0.693637550 container remove bad20a32fc44c967ac13339e65d4d1c81ca77772ec2afa5550a4802bb4949513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0) 2024-07-26T08:40:55.660 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:55 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:40:56.022 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:55 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T08:40:56.022 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:40:55 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 24.219s CPU time. 2024-07-26T08:40:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:56 smithi044 ceph-mon[32404]: pgmap v550: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 714 KiB/s wr, 68 op/s 2024-07-26T08:40:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:56 smithi116 ceph-mon[37097]: pgmap v550: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 714 KiB/s wr, 68 op/s 2024-07-26T08:40:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:56 smithi171 ceph-mon[37789]: pgmap v550: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 714 KiB/s wr, 68 op/s 2024-07-26T08:40:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:40:58 smithi044 ceph-mon[32404]: pgmap v551: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 498 KiB/s wr, 41 op/s 2024-07-26T08:40:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:40:58 smithi116 ceph-mon[37097]: pgmap v551: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 498 KiB/s wr, 41 op/s 2024-07-26T08:40:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:40:58 smithi171 ceph-mon[37789]: pgmap v551: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 498 KiB/s wr, 41 op/s 2024-07-26T08:41:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:00 smithi044 ceph-mon[32404]: pgmap v552: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 450 KiB/s wr, 36 op/s 2024-07-26T08:41:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:00 smithi116 ceph-mon[37097]: pgmap v552: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 450 KiB/s wr, 36 op/s 2024-07-26T08:41:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:00 smithi171 ceph-mon[37789]: pgmap v552: 129 pgs: 129 active+clean; 57 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 450 KiB/s wr, 36 op/s 2024-07-26T08:41:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:02 smithi044 ceph-mon[32404]: pgmap v553: 129 pgs: 129 active+clean; 58 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 202 KiB/s wr, 24 op/s 2024-07-26T08:41:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:02 smithi116 ceph-mon[37097]: pgmap v553: 129 pgs: 129 active+clean; 58 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 202 KiB/s wr, 24 op/s 2024-07-26T08:41:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:02 smithi171 ceph-mon[37789]: pgmap v553: 129 pgs: 129 active+clean; 58 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 202 KiB/s wr, 24 op/s 2024-07-26T08:41:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:04 smithi044 ceph-mon[32404]: pgmap v554: 129 pgs: 129 active+clean; 59 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 311 KiB/s wr, 49 op/s 2024-07-26T08:41:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:04 smithi116 ceph-mon[37097]: pgmap v554: 129 pgs: 129 active+clean; 59 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 311 KiB/s wr, 49 op/s 2024-07-26T08:41:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:04 smithi171 ceph-mon[37789]: pgmap v554: 129 pgs: 129 active+clean; 59 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 311 KiB/s wr, 49 op/s 2024-07-26T08:41:06.140 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:05 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-07-26T08:41:06.140 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:05 smithi116 systemd[1]: Stopped Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:06.141 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:05 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 24.219s CPU time. 2024-07-26T08:41:06.141 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:05 smithi116 systemd[1]: Starting Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:41:06.141 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 podman[70909]: 2024-07-26 08:41:05.907135263 +0000 UTC m=+0.017394394 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:41:06.141 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 podman[70909]: 2024-07-26 08:41:06.0178234 +0000 UTC m=+0.128082556 container create 3fea386de2789185f0327fb4cec250f4a92ab04ac0cb1ac381f7655d938b9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:41:06.407 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 podman[70909]: 2024-07-26 08:41:06.264370097 +0000 UTC m=+0.374629228 container init 3fea386de2789185f0327fb4cec250f4a92ab04ac0cb1ac381f7655d938b9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:41:06.408 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 podman[70909]: 2024-07-26 08:41:06.267053036 +0000 UTC m=+0.377312153 container start 3fea386de2789185f0327fb4cec250f4a92ab04ac0cb1ac381f7655d938b9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:41:06.408 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mds[70928]: -- 172.21.15.116:0/2315390932 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5620dc93d860 con 0x5620e0adfa80 2024-07-26T08:41:06.408 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: starting mds.b at 2024-07-26T08:41:06.408 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 bash[70909]: 3fea386de2789185f0327fb4cec250f4a92ab04ac0cb1ac381f7655d938b9c1f 2024-07-26T08:41:06.408 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:06 smithi116 systemd[1]: Started Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: pgmap v555: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 242 KiB/s wr, 48 op/s 2024-07-26T08:41:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: daemon mds.b restarted 2024-07-26T08:41:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T08:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:boot 2024-07-26T08:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:06 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: pgmap v555: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 242 KiB/s wr, 48 op/s 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: daemon mds.b restarted 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:boot 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:06 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: pgmap v555: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 242 KiB/s wr, 48 op/s 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: daemon mds.b restarted 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:boot 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:06 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:08 smithi044 ceph-mon[32404]: pgmap v557: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 289 KiB/s wr, 56 op/s 2024-07-26T08:41:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:08 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:resolve 2024-07-26T08:41:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:08 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:08 smithi116 ceph-mon[37097]: pgmap v557: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 289 KiB/s wr, 56 op/s 2024-07-26T08:41:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:08 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:resolve 2024-07-26T08:41:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:08 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:08 smithi171 ceph-mon[37789]: pgmap v557: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 289 KiB/s wr, 56 op/s 2024-07-26T08:41:08.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:08 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:resolve 2024-07-26T08:41:08.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:08 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:08.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:reconnect 2024-07-26T08:41:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0099998 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0779985 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:41:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:41:09.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:41:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:reconnect 2024-07-26T08:41:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0099998 2024-07-26T08:41:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0779985 2024-07-26T08:41:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:41:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:reconnect 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0099998 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0779985 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:41:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:41:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:41:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: *** Caught signal (Aborted) ** 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: in thread 7fca74088640 thread_name:ms_dispatch 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca7945f6f0] 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fca794ac94c] 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 3: raise() 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 4: abort() 2024-07-26T08:41:10.338 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 5: /lib64/libc.so.6(+0x2871b) [0x7fca7944971b] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 6: /lib64/libc.so.6(+0x37386) [0x7fca79458386] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5574240816b9] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55742423f80c] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55742418dcde] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55742414f763] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557424150a1b] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557424134752] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557424082a56] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557424086637] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557424086f5e] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55742406a03a] 2024-07-26T08:41:10.339 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fca79d2c407] 2024-07-26T08:41:10.340 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 18: (DispatchQueue::entry()+0x60f) [0x7fca79d2ccf9] 2024-07-26T08:41:10.340 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fca79e095a1] 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 20: (Thread::entry_wrapper()+0x43) [0x7fca79bb6cdd] 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 21: (Thread::_entry_func(void*)+0xd) [0x7fca79bb6cf9] 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 22: /lib64/libc.so.6(+0x89c02) [0x7fca794aac02] 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fca7952fc40] 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.018+0000 7fca74088640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: in thread 7fca74088640 thread_name:ms_dispatch 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca7945f6f0] 2024-07-26T08:41:10.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fca794ac94c] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 3: raise() 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 4: abort() 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 5: /lib64/libc.so.6(+0x2871b) [0x7fca7944971b] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 6: /lib64/libc.so.6(+0x37386) [0x7fca79458386] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5574240816b9] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55742423f80c] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55742418dcde] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55742414f763] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557424150a1b] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557424134752] 2024-07-26T08:41:10.342 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557424082a56] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557424086637] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557424086f5e] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55742406a03a] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fca79d2c407] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 18: (DispatchQueue::entry()+0x60f) [0x7fca79d2ccf9] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fca79e095a1] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 20: (Thread::entry_wrapper()+0x43) [0x7fca79bb6cdd] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 21: (Thread::_entry_func(void*)+0xd) [0x7fca79bb6cf9] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 22: /lib64/libc.so.6(+0x89c02) [0x7fca794aac02] 2024-07-26T08:41:10.343 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fca7952fc40] 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 0> 2024-07-26T08:41:10.018+0000 7fca74088640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: in thread 7fca74088640 thread_name:ms_dispatch 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca7945f6f0] 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fca794ac94c] 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 3: raise() 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 4: abort() 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 5: /lib64/libc.so.6(+0x2871b) [0x7fca7944971b] 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 6: /lib64/libc.so.6(+0x37386) [0x7fca79458386] 2024-07-26T08:41:10.344 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5574240816b9] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55742423f80c] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55742418dcde] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55742414f763] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557424150a1b] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557424134752] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557424082a56] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557424086637] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557424086f5e] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55742406a03a] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fca79d2c407] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 18: (DispatchQueue::entry()+0x60f) [0x7fca79d2ccf9] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fca79e095a1] 2024-07-26T08:41:10.345 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 20: (Thread::entry_wrapper()+0x43) [0x7fca79bb6cdd] 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 21: (Thread::_entry_func(void*)+0xd) [0x7fca79bb6cf9] 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 22: /lib64/libc.so.6(+0x89c02) [0x7fca794aac02] 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fca7952fc40] 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: -2> 2024-07-26T08:41:10.018+0000 7fca74088640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: in thread 7fca74088640 thread_name:ms_dispatch 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:10.346 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fca7945f6f0] 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fca794ac94c] 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 3: raise() 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 4: abort() 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 5: /lib64/libc.so.6(+0x2871b) [0x7fca7944971b] 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 6: /lib64/libc.so.6(+0x37386) [0x7fca79458386] 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5574240816b9] 2024-07-26T08:41:10.347 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55742423f80c] 2024-07-26T08:41:10.348 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55742418dcde] 2024-07-26T08:41:10.348 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55742414f763] 2024-07-26T08:41:10.348 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557424150a1b] 2024-07-26T08:41:10.348 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557424134752] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557424082a56] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557424086637] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557424086f5e] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55742406a03a] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fca79d2c407] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 18: (DispatchQueue::entry()+0x60f) [0x7fca79d2ccf9] 2024-07-26T08:41:10.349 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fca79e095a1] 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 20: (Thread::entry_wrapper()+0x43) [0x7fca79bb6cdd] 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 21: (Thread::_entry_func(void*)+0xd) [0x7fca79bb6cf9] 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 22: /lib64/libc.so.6(+0x89c02) [0x7fca794aac02] 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fca7952fc40] 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[69265]: 2024-07-26T08:41:10.350 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 podman[72619]: 2024-07-26 08:41:10.078837225 +0000 UTC m=+0.020477599 container died 22bb25534d60ea3db68a759b7cc8f9f0fc45366c95a7993c488501ac7d89bbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T08:41:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: pgmap v558: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 246 KiB/s wr, 53 op/s 2024-07-26T08:41:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:rejoin 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:active 2024-07-26T08:41:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:10 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: pgmap v558: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 246 KiB/s wr, 53 op/s 2024-07-26T08:41:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T08:41:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T08:41:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:rejoin 2024-07-26T08:41:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:41:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:41:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:active 2024-07-26T08:41:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:10 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: pgmap v558: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 246 KiB/s wr, 53 op/s 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:rejoin 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] up:active 2024-07-26T08:41:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:10 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:11.193 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 podman[72619]: 2024-07-26 08:41:10.941934338 +0000 UTC m=+0.883574712 container remove 22bb25534d60ea3db68a759b7cc8f9f0fc45366c95a7993c488501ac7d89bbf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:41:11.193 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:10 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:41:11.587 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:11 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:41:11.587 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:11 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 3.439s CPU time. 2024-07-26T08:41:11.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 10471e4e-4b28-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T08:41:12.268 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/10471e4e-4b28-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T08:41:12.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:12 smithi044 ceph-mon[32404]: pgmap v559: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 212 KiB/s wr, 49 op/s 2024-07-26T08:41:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:12 smithi116 ceph-mon[37097]: pgmap v559: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 212 KiB/s wr, 49 op/s 2024-07-26T08:41:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:12 smithi171 ceph-mon[37789]: pgmap v559: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 212 KiB/s wr, 49 op/s 2024-07-26T08:41:14.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 -- 172.21.15.44:0/947265644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10b0f0 msgr2=0x7fcf6c10b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:41:14.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 --2- 172.21.15.44:0/947265644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10b0f0 0x7fcf6c10b500 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcf58009900 tx=0x7fcf5802a3f0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 -- 172.21.15.44:0/947265644 shutdown_connections 2024-07-26T08:41:14.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 --2- 172.21.15.44:0/947265644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf6c10ba40 0x7fcf6c112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 --2- 172.21.15.44:0/947265644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10b0f0 0x7fcf6c10b500 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 --2- 172.21.15.44:0/947265644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1057b0 0x7fcf6c105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 -- 172.21.15.44:0/947265644 >> 172.21.15.44:0/947265644 conn(0x7fcf6c1011c0 msgr2=0x7fcf6c103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:41:14.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 -- 172.21.15.44:0/947265644 shutdown_connections 2024-07-26T08:41:14.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.663+0000 7fcf74600640 1 -- 172.21.15.44:0/947265644 wait complete. 2024-07-26T08:41:14.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 Processor -- start 2024-07-26T08:41:14.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 -- start start 2024-07-26T08:41:14.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf6c1057b0 0x7fcf6c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10b0f0 0x7fcf6c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10ba40 0x7fcf6c1a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c1156b0 con 0x7fcf6c10ba40 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c115530 con 0x7fcf6c10b0f0 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.665+0000 7fcf74600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c115830 con 0x7fcf6c1057b0 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf6c1057b0 0x7fcf6c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10b0f0 0x7fcf6c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf6c1057b0 0x7fcf6c19c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:50456/0 (socket says 172.21.15.44:50456) 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 -- 172.21.15.44:0/1872798322 learned_addr learned my addr 172.21.15.44:0/1872798322 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:41:14.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf72600640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10ba40 0x7fcf6c1a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10b0f0 msgr2=0x7fcf6c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:41:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10b0f0 0x7fcf6c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10ba40 msgr2=0x7fcf6c1a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:41:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf6c10ba40 0x7fcf6c1a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6c1a2150 con 0x7fcf6c1057b0 2024-07-26T08:41:14.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.666+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10b0f0 0x7fcf6c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:41:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.667+0000 7fcf71c00640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf6c1057b0 0x7fcf6c19c1e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000ea60 tx=0x7fcf6000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:41:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.667+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000cde0 con 0x7fcf6c1057b0 2024-07-26T08:41:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.667+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf600035f0 con 0x7fcf6c1057b0 2024-07-26T08:41:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.667+0000 7fcf74600640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6c1a2440 con 0x7fcf6c1057b0 2024-07-26T08:41:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.667+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60010690 con 0x7fcf6c1057b0 2024-07-26T08:41:14.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.667+0000 7fcf74600640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6c1a8ba0 con 0x7fcf6c1057b0 2024-07-26T08:41:14.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 Processor -- start 2024-07-26T08:41:14.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 -- start start 2024-07-26T08:41:14.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fcf600037a0 con 0x7fcf6c1057b0 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c94889a070 0x55c948c3edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c9488711b0 0x55c948c3f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c3f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c948bb5b90 con 0x55c9488711b0 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c948ba5090 con 0x55c94889a070 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.671+0000 7fcf7b1d4740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c948ba90d0 con 0x55c948879390 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c9488711b0 0x55c948c3f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf72600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c94889a070 0x55c948c3edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71c00640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c9488711b0 0x55c948c3f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38770/0 (socket says 172.21.15.44:38770) 2024-07-26T08:41:14.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71c00640 1 -- 172.21.15.44:0/409170725 learned_addr learned my addr 172.21.15.44:0/409170725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:41:14.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71200640 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c3f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71200640 1 -- 172.21.15.44:0/409170725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c94889a070 msgr2=0x55c948c3edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71200640 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c94889a070 0x55c948c3edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71200640 1 -- 172.21.15.44:0/409170725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c9488711b0 msgr2=0x55c948c3f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71200640 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c9488711b0 0x55c948c3f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.672+0000 7fcf71200640 1 -- 172.21.15.44:0/409170725 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c948905650 con 0x55c948879390 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.673+0000 7fcf71200640 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c3f870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802a910 tx=0x7fcf58000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f09dac833cdeafd0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.673+0000 7fcf5ea00640 1 -- 172.21.15.44:0/409170725 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58040070 con 0x55c948879390 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.673+0000 7fcf5ea00640 1 -- 172.21.15.44:0/409170725 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf58031720 con 0x55c948879390 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.673+0000 7fcf5ea00640 1 -- 172.21.15.44:0/409170725 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58035040 con 0x55c948879390 2024-07-26T08:41:14.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.673+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9517+0+0 (secure 0 0 0) 0x7fcf600e4760 con 0x7fcf6c1057b0 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf72600640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/409170725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 msgr2=0x55c948c3f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c3f870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802a910 tx=0x7fcf58000bc0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/409170725 shutdown_connections 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c3f870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c9488711b0 0x55c948c3f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 --2- 172.21.15.44:0/409170725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c94889a070 0x55c948c3edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.674+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/409170725 >> 172.21.15.44:0/409170725 conn(0x55c94888e460 msgr2=0x55c948c3cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.675+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/409170725 shutdown_connections 2024-07-26T08:41:14.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.675+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/409170725 wait complete. 2024-07-26T08:41:14.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.675+0000 7fcf72600640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b6250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcf64005fd0 tx=0x7fcf64009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:41:14.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.675+0000 7fcf7b1d4740 1 -- start start 2024-07-26T08:41:14.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 Processor -- start 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9488711b0 0x55c948c4a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c4a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c94889a070 0x55c948c4ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c948a08700 con 0x55c94889a070 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c948b41660 con 0x55c9488711b0 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c948c4d570 con 0x55c948879390 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.676+0000 7fcf7b1d4740 10 client.0 _ll_register_callbacks cb 0x55c948b51110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T08:41:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c94889a070 0x55c948c4ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c94889a070 0x55c948c4ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38772/0 (socket says 172.21.15.44:38772) 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 -- 172.21.15.44:0/2544213633 learned_addr learned my addr 172.21.15.44:0/2544213633 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9488711b0 0x55c948c4a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf71200640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c4a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 msgr2=0x55c948c4a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c948879390 0x55c948c4a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9488711b0 msgr2=0x55c948c4a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9488711b0 0x55c948c4a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c948b58800 con 0x55c94889a070 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9488711b0 0x55c948c4a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.677+0000 7fcf72600640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c94889a070 0x55c948c4ae10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcf64012010 tx=0x7fcf64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc66d3d114203d81 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf53400640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-07-26T08:41:14.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6400f7d0 con 0x55c94889a070 2024-07-26T08:41:14.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9517+0+0 (secure 0 0 0) 0x7fcf6400f970 con 0x55c94889a070 2024-07-26T08:41:14.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf58031720 con 0x55c94889a070 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcf6400fb70 con 0x55c94889a070 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf7b1d4740 10 client.15669 fetch_fsmap learned FSMap version 38 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.678+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c948c59160 con 0x55c94889a070 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf58031720 con 0x55c94889a070 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7fcf64026070 con 0x55c94889a070 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf7b1d4740 10 client.15669 fetch_fsmap finished waiting for FSMap version 38 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf7b1d4740 10 client.15669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.f 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf7b1d4740 20 client.15669 populate_metadata read hostname 'smithi044' 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf7b1d4740 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf7b1d4740 4 client.15669 mds_command: new command op to 24655 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf7b1d4740 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c948bb7ae0 con 0x55c9488589f0 2024-07-26T08:41:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:41:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:41:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.679+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T08:41:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:14 smithi044 ceph-mon[32404]: pgmap v560: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 88 KiB/s wr, 27 op/s 2024-07-26T08:41:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:14 smithi116 ceph-mon[37097]: pgmap v560: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 88 KiB/s wr, 27 op/s 2024-07-26T08:41:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:14 smithi171 ceph-mon[37789]: pgmap v560: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 88 KiB/s wr, 27 op/s 2024-07-26T08:41:14.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.880+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:41:14.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.880+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T08:41:14.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:14.880+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:41:15.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:15.281+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:41:15.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:15.281+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T08:41:15.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:15.281+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:41:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:16.082+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:41:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:16.082+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T08:41:16.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:16.082+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:41:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:16 smithi044 ceph-mon[32404]: pgmap v561: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.6 KiB/s wr, 17 op/s 2024-07-26T08:41:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:16 smithi116 ceph-mon[37097]: pgmap v561: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.6 KiB/s wr, 17 op/s 2024-07-26T08:41:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:16 smithi171 ceph-mon[37789]: pgmap v561: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.6 KiB/s wr, 17 op/s 2024-07-26T08:41:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:17.684+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:41:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:17.684+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T08:41:17.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:17.684+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:41:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:18 smithi044 ceph-mon[32404]: pgmap v562: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-07-26T08:41:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:18 smithi116 ceph-mon[37097]: pgmap v562: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-07-26T08:41:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:18 smithi171 ceph-mon[37789]: pgmap v562: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-07-26T08:41:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:20 smithi044 ceph-mon[32404]: pgmap v563: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-07-26T08:41:20.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:20 smithi116 ceph-mon[37097]: pgmap v563: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-07-26T08:41:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:20 smithi171 ceph-mon[37789]: pgmap v563: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 KiB/s wr, 17 op/s 2024-07-26T08:41:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:20.888+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:41:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:20.888+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:41:20.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:20.888+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T08:41:21.619 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-07-26T08:41:21.619 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 systemd[1]: Stopped Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:21.619 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 3.439s CPU time. 2024-07-26T08:41:21.619 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 systemd[1]: Starting Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:41:21.619 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 podman[72712]: 2024-07-26 08:41:21.472001931 +0000 UTC m=+0.018331492 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:41:21.619 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 podman[72712]: 2024-07-26 08:41:21.618657821 +0000 UTC m=+0.164987373 container create 977e33e438a91f627285b56e7f038b5bd32677041feeba83970bd7db0a34f530 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T08:41:22.247 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 podman[72712]: 2024-07-26 08:41:21.872433185 +0000 UTC m=+0.418762737 container init 977e33e438a91f627285b56e7f038b5bd32677041feeba83970bd7db0a34f530 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:41:22.248 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 podman[72712]: 2024-07-26 08:41:21.875128094 +0000 UTC m=+0.421457645 container start 977e33e438a91f627285b56e7f038b5bd32677041feeba83970bd7db0a34f530 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True) 2024-07-26T08:41:22.248 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 ceph-mds[72730]: -- 172.21.15.171:0/170681589 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5643fe02b860 con 0x5644021cda80 2024-07-26T08:41:22.248 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: starting mds.f at 2024-07-26T08:41:22.248 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 bash[72712]: 977e33e438a91f627285b56e7f038b5bd32677041feeba83970bd7db0a34f530 2024-07-26T08:41:22.248 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:21 smithi171 systemd[1]: Started Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: pgmap v564: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-07-26T08:41:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: daemon mds.f restarted 2024-07-26T08:41:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:41:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T08:41:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:boot 2024-07-26T08:41:22.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:41:22.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:22.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:22 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: pgmap v564: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-07-26T08:41:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: daemon mds.f restarted 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:boot 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:22 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: pgmap v564: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-07-26T08:41:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: daemon mds.f restarted 2024-07-26T08:41:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:41:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T08:41:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:boot 2024-07-26T08:41:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:41:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:22 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:24 smithi044 ceph-mon[32404]: pgmap v566: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:24 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:resolve 2024-07-26T08:41:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:24 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:24 smithi116 ceph-mon[37097]: pgmap v566: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:24 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:resolve 2024-07-26T08:41:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:24 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:24 smithi171 ceph-mon[37789]: pgmap v566: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:24 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:resolve 2024-07-26T08:41:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:24 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:25 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:41:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:25 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:reconnect 2024-07-26T08:41:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:25 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:25 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999982 2024-07-26T08:41:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:25 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:41:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:25 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:reconnect 2024-07-26T08:41:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:25 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:25 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999982 2024-07-26T08:41:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:25 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:41:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:25 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:reconnect 2024-07-26T08:41:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:25 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:25 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999982 2024-07-26T08:41:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:26 smithi171 ceph-mon[37789]: pgmap v567: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:26 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:rejoin 2024-07-26T08:41:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:26 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:26 smithi171 ceph-mon[37789]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:41:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-mon[37097]: pgmap v567: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:rejoin 2024-07-26T08:41:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-mon[37097]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: *** Caught signal (Aborted) ** 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: in thread 7f455ec45640 thread_name:ms_dispatch 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f456401c6f0] 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f456406994c] 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 3: raise() 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 4: abort() 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 5: /lib64/libc.so.6(+0x2871b) [0x7f456400671b] 2024-07-26T08:41:26.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 6: /lib64/libc.so.6(+0x37386) [0x7f4564015386] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5620db4696b9] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5620db62780c] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5620db575cde] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5620db537763] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5620db538a1b] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5620db51c752] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5620db46aa56] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5620db46e637] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5620db46ef5e] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5620db45203a] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f45648e9407] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 18: (DispatchQueue::entry()+0x60f) [0x7f45648e9cf9] 2024-07-26T08:41:26.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f45649c65a1] 2024-07-26T08:41:26.930 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 20: (Thread::entry_wrapper()+0x43) [0x7f4564773cdd] 2024-07-26T08:41:26.930 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4564773cf9] 2024-07-26T08:41:26.930 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4564067c02] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f45640ecc40] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.553+0000 7f455ec45640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: in thread 7f455ec45640 thread_name:ms_dispatch 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f456401c6f0] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f456406994c] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 3: raise() 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 4: abort() 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 5: /lib64/libc.so.6(+0x2871b) [0x7f456400671b] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 6: /lib64/libc.so.6(+0x37386) [0x7f4564015386] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5620db4696b9] 2024-07-26T08:41:26.931 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5620db62780c] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5620db575cde] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5620db537763] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5620db538a1b] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5620db51c752] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5620db46aa56] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5620db46e637] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5620db46ef5e] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5620db45203a] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f45648e9407] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 18: (DispatchQueue::entry()+0x60f) [0x7f45648e9cf9] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f45649c65a1] 2024-07-26T08:41:26.932 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 20: (Thread::entry_wrapper()+0x43) [0x7f4564773cdd] 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4564773cf9] 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4564067c02] 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f45640ecc40] 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 0> 2024-07-26T08:41:26.553+0000 7f455ec45640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: in thread 7f455ec45640 thread_name:ms_dispatch 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f456401c6f0] 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f456406994c] 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 3: raise() 2024-07-26T08:41:26.933 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 4: abort() 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 5: /lib64/libc.so.6(+0x2871b) [0x7f456400671b] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 6: /lib64/libc.so.6(+0x37386) [0x7f4564015386] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5620db4696b9] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5620db62780c] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5620db575cde] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5620db537763] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5620db538a1b] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5620db51c752] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5620db46aa56] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5620db46e637] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5620db46ef5e] 2024-07-26T08:41:26.934 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5620db45203a] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f45648e9407] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 18: (DispatchQueue::entry()+0x60f) [0x7f45648e9cf9] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f45649c65a1] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 20: (Thread::entry_wrapper()+0x43) [0x7f4564773cdd] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4564773cf9] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4564067c02] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f45640ecc40] 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: -1> 2024-07-26T08:41:26.553+0000 7f455ec45640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: in thread 7f455ec45640 thread_name:ms_dispatch 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.935 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f456401c6f0] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f456406994c] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 3: raise() 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 4: abort() 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 5: /lib64/libc.so.6(+0x2871b) [0x7f456400671b] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 6: /lib64/libc.so.6(+0x37386) [0x7f4564015386] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5620db4696b9] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5620db62780c] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5620db575cde] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5620db537763] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5620db538a1b] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5620db51c752] 2024-07-26T08:41:26.936 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5620db46aa56] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5620db46e637] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5620db46ef5e] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5620db45203a] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f45648e9407] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 18: (DispatchQueue::entry()+0x60f) [0x7f45648e9cf9] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f45649c65a1] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 20: (Thread::entry_wrapper()+0x43) [0x7f4564773cdd] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4564773cf9] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4564067c02] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f45640ecc40] 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[70924]: 2024-07-26T08:41:26.937 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:26 smithi116 podman[71095]: 2024-07-26 08:41:26.615559877 +0000 UTC m=+0.018736997 container died 3fea386de2789185f0327fb4cec250f4a92ab04ac0cb1ac381f7655d938b9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-07-26T08:41:26.956 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:26 smithi044 ceph-mon[32404]: pgmap v567: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:26.956 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:26 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:rejoin 2024-07-26T08:41:26.956 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:26 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:26.956 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:26 smithi044 ceph-mon[32404]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:41:27.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:27.294+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:27.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:27.295+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:41:27.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:27.295+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:27.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:27.295+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:27.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:27.295+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T08:41:27.552 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:27 smithi116 podman[71095]: 2024-07-26 08:41:27.279369938 +0000 UTC m=+0.682547094 container remove 3fea386de2789185f0327fb4cec250f4a92ab04ac0cb1ac381f7655d938b9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T08:41:27.552 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:27 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:41:27.552 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:27 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T08:41:27.552 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:27 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.417s CPU time. 2024-07-26T08:41:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:27 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:27 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:41:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:27 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:active 2024-07-26T08:41:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:27 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:27 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:27 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:27 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:41:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:27 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:active 2024-07-26T08:41:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:27 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:27 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:27 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:27 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:41:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:27 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/4015063702,v1:172.21.15.116:6839/4015063702] up:active 2024-07-26T08:41:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:27 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:27 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:28 smithi171 ceph-mon[37789]: pgmap v568: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:41:28.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:28 smithi044 ceph-mon[32404]: pgmap v568: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:41:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:28 smithi116 ceph-mon[37097]: pgmap v568: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:41:30.597 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:30 smithi044 ceph-mon[32404]: pgmap v569: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:41:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:30 smithi171 ceph-mon[37789]: pgmap v569: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:41:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:30 smithi116 ceph-mon[37097]: pgmap v569: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:41:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:32 smithi171 ceph-mon[37789]: pgmap v570: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:41:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:32 smithi044 ceph-mon[32404]: pgmap v570: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:41:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:32 smithi116 ceph-mon[37097]: pgmap v570: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:41:34.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:34 smithi116 ceph-mon[37097]: pgmap v571: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 184 B/s wr, 16 op/s 2024-07-26T08:41:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:34 smithi171 ceph-mon[37789]: pgmap v571: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 184 B/s wr, 16 op/s 2024-07-26T08:41:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:34 smithi044 ceph-mon[32404]: pgmap v571: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 184 B/s wr, 16 op/s 2024-07-26T08:41:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:36 smithi044 ceph-mon[32404]: pgmap v572: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:41:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:36 smithi116 ceph-mon[37097]: pgmap v572: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:41:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:36 smithi171 ceph-mon[37789]: pgmap v572: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:41:38.024 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:37 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:41:38.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:37 smithi116 systemd[1]: Stopped Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:38.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:37 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.417s CPU time. 2024-07-26T08:41:38.025 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:37 smithi116 systemd[1]: Starting Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:41:38.309 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 podman[71199]: 2024-07-26 08:41:37.926296936 +0000 UTC m=+0.017877962 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:41:38.309 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 podman[71199]: 2024-07-26 08:41:38.064666568 +0000 UTC m=+0.156247596 container create c20b6012506f301445462584851ab67b042461a655ec6444cc872887d26c6b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-07-26T08:41:38.596 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 podman[71199]: 2024-07-26 08:41:38.308861944 +0000 UTC m=+0.400442973 container init c20b6012506f301445462584851ab67b042461a655ec6444cc872887d26c6b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-07-26T08:41:38.596 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 podman[71199]: 2024-07-26 08:41:38.311485967 +0000 UTC m=+0.403066987 container start c20b6012506f301445462584851ab67b042461a655ec6444cc872887d26c6b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-07-26T08:41:38.596 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mds[71218]: -- 172.21.15.116:0/4210749766 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564268b21860 con 0x56426ccc3a80 2024-07-26T08:41:38.597 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: starting mds.b at 2024-07-26T08:41:38.597 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 bash[71199]: c20b6012506f301445462584851ab67b042461a655ec6444cc872887d26c6b3e 2024-07-26T08:41:38.597 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:38 smithi116 systemd[1]: Started Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:38.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: pgmap v573: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:41:38.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: daemon mds.b restarted 2024-07-26T08:41:38.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:38.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:41:38.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T08:41:38.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:boot 2024-07-26T08:41:38.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:41:38.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:38.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:38 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: pgmap v573: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:41:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: daemon mds.b restarted 2024-07-26T08:41:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:41:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T08:41:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:boot 2024-07-26T08:41:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:41:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:38 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: pgmap v573: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:41:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: daemon mds.b restarted 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:boot 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:38 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:40.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:40.097+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:40.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:40.098+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:41:40.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:40.098+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:40.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:40.098+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:40.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:40.099+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:40 smithi044 ceph-mon[32404]: pgmap v575: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 op/s 2024-07-26T08:41:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:40 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:resolve 2024-07-26T08:41:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:40 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:40 smithi116 ceph-mon[37097]: pgmap v575: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 op/s 2024-07-26T08:41:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:40 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:resolve 2024-07-26T08:41:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:40 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:40 smithi171 ceph-mon[37789]: pgmap v575: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 op/s 2024-07-26T08:41:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:40 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:resolve 2024-07-26T08:41:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:40 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:41.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:41 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399992 2024-07-26T08:41:41.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:41 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:reconnect 2024-07-26T08:41:41.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:41 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:41.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:41 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:41:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:41 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399992 2024-07-26T08:41:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:41 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:reconnect 2024-07-26T08:41:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:41 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:41 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:41:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:41 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399992 2024-07-26T08:41:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:41 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:reconnect 2024-07-26T08:41:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:41 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:41 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:41:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:42 smithi044 ceph-mon[32404]: pgmap v576: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 11 op/s 2024-07-26T08:41:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:42 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:rejoin 2024-07-26T08:41:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:42 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:42 smithi044 ceph-mon[32404]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:41:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:42 smithi116 ceph-mon[37097]: pgmap v576: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 11 op/s 2024-07-26T08:41:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:42 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:rejoin 2024-07-26T08:41:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:42 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:42 smithi116 ceph-mon[37097]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:41:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-mon[37789]: pgmap v576: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 11 op/s 2024-07-26T08:41:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:rejoin 2024-07-26T08:41:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-mon[37789]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: *** Caught signal (Aborted) ** 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: in thread 7f4bf7c0a640 thread_name:ms_dispatch 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4bfcfe16f0] 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4bfd02e94c] 2024-07-26T08:41:42.838 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 3: raise() 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 4: abort() 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4bfcfcb71b] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 6: /lib64/libc.so.6(+0x37386) [0x7f4bfcfda386] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643fbe1f6b9] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643fbfdd80c] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5643fbf2bcde] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5643fbeed763] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643fbeeea1b] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643fbed2752] 2024-07-26T08:41:42.839 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643fbe20a56] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643fbe24637] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643fbe24f5e] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643fbe0803a] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4bfd8ae407] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 18: (DispatchQueue::entry()+0x60f) [0x7f4bfd8aecf9] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4bfd98b5a1] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 20: (Thread::entry_wrapper()+0x43) [0x7f4bfd738cdd] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4bfd738cf9] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4bfd02cc02] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4bfd0b1c40] 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.451+0000 7f4bf7c0a640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: in thread 7f4bf7c0a640 thread_name:ms_dispatch 2024-07-26T08:41:42.840 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4bfcfe16f0] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4bfd02e94c] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 3: raise() 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 4: abort() 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4bfcfcb71b] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 6: /lib64/libc.so.6(+0x37386) [0x7f4bfcfda386] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643fbe1f6b9] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643fbfdd80c] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5643fbf2bcde] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5643fbeed763] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643fbeeea1b] 2024-07-26T08:41:42.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643fbed2752] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643fbe20a56] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643fbe24637] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643fbe24f5e] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643fbe0803a] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4bfd8ae407] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 18: (DispatchQueue::entry()+0x60f) [0x7f4bfd8aecf9] 2024-07-26T08:41:42.842 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4bfd98b5a1] 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 20: (Thread::entry_wrapper()+0x43) [0x7f4bfd738cdd] 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4bfd738cf9] 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4bfd02cc02] 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4bfd0b1c40] 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 0> 2024-07-26T08:41:42.451+0000 7f4bf7c0a640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: in thread 7f4bf7c0a640 thread_name:ms_dispatch 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:42.843 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4bfcfe16f0] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4bfd02e94c] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 3: raise() 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 4: abort() 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4bfcfcb71b] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 6: /lib64/libc.so.6(+0x37386) [0x7f4bfcfda386] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643fbe1f6b9] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643fbfdd80c] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5643fbf2bcde] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5643fbeed763] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643fbeeea1b] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643fbed2752] 2024-07-26T08:41:42.844 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643fbe20a56] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643fbe24637] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643fbe24f5e] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643fbe0803a] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4bfd8ae407] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 18: (DispatchQueue::entry()+0x60f) [0x7f4bfd8aecf9] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4bfd98b5a1] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 20: (Thread::entry_wrapper()+0x43) [0x7f4bfd738cdd] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4bfd738cf9] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4bfd02cc02] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4bfd0b1c40] 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:42.845 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 0> 2024-07-26T08:41:42.451+0000 7f4bf7c0a640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: in thread 7f4bf7c0a640 thread_name:ms_dispatch 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4bfcfe16f0] 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4bfd02e94c] 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 3: raise() 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 4: abort() 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4bfcfcb71b] 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 6: /lib64/libc.so.6(+0x37386) [0x7f4bfcfda386] 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643fbe1f6b9] 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643fbfdd80c] 2024-07-26T08:41:42.846 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5643fbf2bcde] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5643fbeed763] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643fbeeea1b] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643fbed2752] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643fbe20a56] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643fbe24637] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643fbe24f5e] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643fbe0803a] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4bfd8ae407] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 18: (DispatchQueue::entry()+0x60f) [0x7f4bfd8aecf9] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4bfd98b5a1] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 20: (Thread::entry_wrapper()+0x43) [0x7f4bfd738cdd] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4bfd738cf9] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4bfd02cc02] 2024-07-26T08:41:42.847 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4bfd0b1c40] 2024-07-26T08:41:42.848 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:42.848 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72726]: 2024-07-26T08:41:42.848 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:42 smithi171 podman[72779]: 2024-07-26 08:41:42.509268103 +0000 UTC m=+0.018558373 container died 977e33e438a91f627285b56e7f038b5bd32677041feeba83970bd7db0a34f530 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T08:41:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:43 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:43 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:41:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:43 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:active 2024-07-26T08:41:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:43 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:43 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:43 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:43 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:41:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:43 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:active 2024-07-26T08:41:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:43 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:43 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:43.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:43 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:43.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:43 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:41:43.787 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:43 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/2633321876,v1:172.21.15.171:6835/2633321876] up:active 2024-07-26T08:41:43.787 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:43 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:43.787 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:43 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:43.787 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:43 smithi171 podman[72779]: 2024-07-26 08:41:43.546801316 +0000 UTC m=+1.056091577 container remove 977e33e438a91f627285b56e7f038b5bd32677041feeba83970bd7db0a34f530 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:41:43.787 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:43 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:41:44.086 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:43 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:41:44.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:43 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.470s CPU time. 2024-07-26T08:41:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:44 smithi044 ceph-mon[32404]: pgmap v577: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:44 smithi116 ceph-mon[37097]: pgmap v577: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:44 smithi171 ceph-mon[37789]: pgmap v577: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:41:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:46 smithi044 ceph-mon[32404]: pgmap v578: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:41:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:46 smithi116 ceph-mon[37097]: pgmap v578: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:41:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:46 smithi171 ceph-mon[37789]: pgmap v578: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:41:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:48 smithi044 ceph-mon[32404]: pgmap v579: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:41:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:48 smithi116 ceph-mon[37097]: pgmap v579: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:41:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:48 smithi171 ceph-mon[37789]: pgmap v579: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:41:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:50 smithi044 ceph-mon[32404]: pgmap v580: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 192 B/s wr, 15 op/s 2024-07-26T08:41:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:50 smithi116 ceph-mon[37097]: pgmap v580: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 192 B/s wr, 15 op/s 2024-07-26T08:41:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:50 smithi171 ceph-mon[37789]: pgmap v580: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 192 B/s wr, 15 op/s 2024-07-26T08:41:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:52 smithi044 ceph-mon[32404]: pgmap v581: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:41:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:52 smithi116 ceph-mon[37097]: pgmap v581: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:41:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:52 smithi171 ceph-mon[37789]: pgmap v581: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:41:54.107 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:53 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:41:54.107 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:53 smithi171 systemd[1]: Stopped Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:54.107 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:53 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.470s CPU time. 2024-07-26T08:41:54.107 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:53 smithi171 systemd[1]: Starting Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:41:54.107 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 podman[72875]: 2024-07-26 08:41:53.973385084 +0000 UTC m=+0.017790804 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:41:54.378 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 podman[72875]: 2024-07-26 08:41:54.106479636 +0000 UTC m=+0.150885348 container create 4b2af47f36c714d8bc6f2cf3f4d6ae97931ceb8f3c15be06624c75244077191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0) 2024-07-26T08:41:54.378 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 podman[72875]: 2024-07-26 08:41:54.343403353 +0000 UTC m=+0.387809080 container init 4b2af47f36c714d8bc6f2cf3f4d6ae97931ceb8f3c15be06624c75244077191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:41:54.379 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 podman[72875]: 2024-07-26 08:41:54.346005317 +0000 UTC m=+0.390411026 container start 4b2af47f36c714d8bc6f2cf3f4d6ae97931ceb8f3c15be06624c75244077191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T08:41:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: pgmap v582: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T08:41:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: daemon mds.f restarted 2024-07-26T08:41:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:41:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T08:41:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:boot 2024-07-26T08:41:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:41:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:54 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: pgmap v582: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: daemon mds.f restarted 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:boot 2024-07-26T08:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:41:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:54 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mds[72894]: -- 172.21.15.171:0/4039349955 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563abf22b860 con 0x563ac33cd600 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: starting mds.f at 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 bash[72875]: 4b2af47f36c714d8bc6f2cf3f4d6ae97931ceb8f3c15be06624c75244077191d 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:41:54 smithi171 systemd[1]: Started Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: pgmap v582: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: daemon mds.f restarted 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:41:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:41:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T08:41:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:boot 2024-07-26T08:41:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:41:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:41:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:54 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:41:55.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:55.108+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:41:55.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:55.109+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:41:55.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:55.109+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:41:55.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:55.109+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:41:55.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:41:55.109+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:56 smithi044 ceph-mon[32404]: pgmap v584: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 op/s 2024-07-26T08:41:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:56 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:resolve 2024-07-26T08:41:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:56 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:56 smithi116 ceph-mon[37097]: pgmap v584: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 op/s 2024-07-26T08:41:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:56 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:resolve 2024-07-26T08:41:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:56 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:56 smithi171 ceph-mon[37789]: pgmap v584: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 13 op/s 2024-07-26T08:41:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:56 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:resolve 2024-07-26T08:41:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:56 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:41:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:57 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:reconnect 2024-07-26T08:41:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:57 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:57 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00999982 2024-07-26T08:41:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:57 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:41:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:57 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:reconnect 2024-07-26T08:41:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:57 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:57 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00999982 2024-07-26T08:41:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:57 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:41:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:57 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:reconnect 2024-07-26T08:41:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:57 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:41:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:57 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00999982 2024-07-26T08:41:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:57 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:41:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:58 smithi171 ceph-mon[37789]: pgmap v585: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 10 op/s 2024-07-26T08:41:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:58 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:rejoin 2024-07-26T08:41:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:58 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:58 smithi171 ceph-mon[37789]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:41:58.913 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-mon[37097]: pgmap v585: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 10 op/s 2024-07-26T08:41:58.914 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:rejoin 2024-07-26T08:41:58.914 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:58.914 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-mon[37097]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:41:58.914 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: *** Caught signal (Aborted) ** 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: in thread 7ff5fe721640 thread_name:ms_dispatch 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff603af86f0] 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff603b4594c] 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 3: raise() 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 4: abort() 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff603ae271b] 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 6: /lib64/libc.so.6(+0x37386) [0x7ff603af1386] 2024-07-26T08:41:58.915 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5642663e96b9] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5642665a780c] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5642664f5cde] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5642664b7763] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5642664b8a1b] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56426649c752] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5642663eaa56] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5642663ee637] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5642663eef5e] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5642663d203a] 2024-07-26T08:41:58.916 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff6043c5407] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 18: (DispatchQueue::entry()+0x60f) [0x7ff6043c5cf9] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6044a25a1] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 20: (Thread::entry_wrapper()+0x43) [0x7ff60424fcdd] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff60424fcf9] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff603b43c02] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff603bc8c40] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.532+0000 7ff5fe721640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: in thread 7ff5fe721640 thread_name:ms_dispatch 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff603af86f0] 2024-07-26T08:41:58.917 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff603b4594c] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 3: raise() 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 4: abort() 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff603ae271b] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 6: /lib64/libc.so.6(+0x37386) [0x7ff603af1386] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5642663e96b9] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5642665a780c] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5642664f5cde] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5642664b7763] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5642664b8a1b] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56426649c752] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5642663eaa56] 2024-07-26T08:41:58.918 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5642663ee637] 2024-07-26T08:41:58.919 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5642663eef5e] 2024-07-26T08:41:58.919 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5642663d203a] 2024-07-26T08:41:58.919 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff6043c5407] 2024-07-26T08:41:58.919 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 18: (DispatchQueue::entry()+0x60f) [0x7ff6043c5cf9] 2024-07-26T08:41:58.919 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6044a25a1] 2024-07-26T08:41:58.919 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 20: (Thread::entry_wrapper()+0x43) [0x7ff60424fcdd] 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff60424fcf9] 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff603b43c02] 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff603bc8c40] 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 0> 2024-07-26T08:41:58.532+0000 7ff5fe721640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: in thread 7ff5fe721640 thread_name:ms_dispatch 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff603af86f0] 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff603b4594c] 2024-07-26T08:41:58.920 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 3: raise() 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 4: abort() 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff603ae271b] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 6: /lib64/libc.so.6(+0x37386) [0x7ff603af1386] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5642663e96b9] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5642665a780c] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5642664f5cde] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5642664b7763] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5642664b8a1b] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56426649c752] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5642663eaa56] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5642663ee637] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5642663eef5e] 2024-07-26T08:41:58.921 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5642663d203a] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff6043c5407] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 18: (DispatchQueue::entry()+0x60f) [0x7ff6043c5cf9] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6044a25a1] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 20: (Thread::entry_wrapper()+0x43) [0x7ff60424fcdd] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff60424fcf9] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff603b43c02] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff603bc8c40] 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.922 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: -1> 2024-07-26T08:41:58.532+0000 7ff5fe721640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: in thread 7ff5fe721640 thread_name:ms_dispatch 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff603af86f0] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff603b4594c] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 3: raise() 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 4: abort() 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff603ae271b] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 6: /lib64/libc.so.6(+0x37386) [0x7ff603af1386] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5642663e96b9] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5642665a780c] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5642664f5cde] 2024-07-26T08:41:58.923 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5642664b7763] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5642664b8a1b] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56426649c752] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5642663eaa56] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5642663ee637] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5642663eef5e] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5642663d203a] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff6043c5407] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 18: (DispatchQueue::entry()+0x60f) [0x7ff6043c5cf9] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff6044a25a1] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 20: (Thread::entry_wrapper()+0x43) [0x7ff60424fcdd] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff60424fcf9] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff603b43c02] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff603bc8c40] 2024-07-26T08:41:58.924 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:41:58.925 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71214]: 2024-07-26T08:41:58.925 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:58 smithi116 podman[71253]: 2024-07-26 08:41:58.591755317 +0000 UTC m=+0.018391793 container died c20b6012506f301445462584851ab67b042461a655ec6444cc872887d26c6b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-26T08:41:58.949 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:58 smithi044 ceph-mon[32404]: pgmap v585: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 10 op/s 2024-07-26T08:41:58.949 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:58 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:rejoin 2024-07-26T08:41:58.949 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:58 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:41:58.949 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:58 smithi044 ceph-mon[32404]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:41:59.561 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:59 smithi116 podman[71253]: 2024-07-26 08:41:59.305741843 +0000 UTC m=+0.732378306 container remove c20b6012506f301445462584851ab67b042461a655ec6444cc872887d26c6b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:41:59.562 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:59 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:41:59.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:59 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:59.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:59 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:41:59.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:59 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:active 2024-07-26T08:41:59.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:59 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:59.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:41:59 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:41:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:59 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:41:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:59 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:41:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:59 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:active 2024-07-26T08:41:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:59 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:41:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:41:59 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:59 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:59 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:59 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/1433351459,v1:172.21.15.116:6839/1433351459] up:active 2024-07-26T08:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:59 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:41:59 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:00.022 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:59 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T08:42:00.022 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:41:59 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.395s CPU time. 2024-07-26T08:42:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:00 smithi171 ceph-mon[37789]: pgmap v586: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:42:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:00 smithi044 ceph-mon[32404]: pgmap v586: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:42:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:00 smithi116 ceph-mon[37097]: pgmap v586: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:42:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:02 smithi171 ceph-mon[37789]: pgmap v587: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 102 B/s wr, 12 op/s 2024-07-26T08:42:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:02 smithi044 ceph-mon[32404]: pgmap v587: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 102 B/s wr, 12 op/s 2024-07-26T08:42:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:02 smithi116 ceph-mon[37097]: pgmap v587: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 102 B/s wr, 12 op/s 2024-07-26T08:42:04.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:04 smithi116 ceph-mon[37097]: pgmap v588: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-07-26T08:42:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:04 smithi171 ceph-mon[37789]: pgmap v588: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-07-26T08:42:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:04 smithi044 ceph-mon[32404]: pgmap v588: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-07-26T08:42:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:06 smithi171 ceph-mon[37789]: pgmap v589: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-07-26T08:42:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:06 smithi044 ceph-mon[32404]: pgmap v589: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-07-26T08:42:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:06 smithi116 ceph-mon[37097]: pgmap v589: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-07-26T08:42:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:08 smithi171 ceph-mon[37789]: pgmap v590: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-07-26T08:42:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:08 smithi044 ceph-mon[32404]: pgmap v590: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-07-26T08:42:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:08 smithi116 ceph-mon[37097]: pgmap v590: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-07-26T08:42:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:42:10.002 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:42:10.003 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:09 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:42:10.003 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:09 smithi116 systemd[1]: Stopped Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:10.003 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:09 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.395s CPU time. 2024-07-26T08:42:10.003 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:09 smithi116 systemd[1]: Starting Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:42:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:42:10.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:10.111+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:42:10.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:10.113+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:42:10.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:10.113+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:42:10.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:10.113+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:42:10.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:10.113+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:10.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 podman[71440]: 2024-07-26 08:42:09.904437028 +0000 UTC m=+0.017761710 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:42:10.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 podman[71440]: 2024-07-26 08:42:10.106658118 +0000 UTC m=+0.219982799 container create 577ae053058a83a25989eb93ca58f3143e316703171e0343ea7c0bcc4453559c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724) 2024-07-26T08:42:10.536 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 podman[71440]: 2024-07-26 08:42:10.368566373 +0000 UTC m=+0.481891068 container init 577ae053058a83a25989eb93ca58f3143e316703171e0343ea7c0bcc4453559c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux ) 2024-07-26T08:42:10.536 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 podman[71440]: 2024-07-26 08:42:10.371376887 +0000 UTC m=+0.484701572 container start 577ae053058a83a25989eb93ca58f3143e316703171e0343ea7c0bcc4453559c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:42:10.536 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mds[71462]: -- 172.21.15.116:0/2443476382 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582d2b0b860 con 0x5582d6cad180 2024-07-26T08:42:10.536 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: starting mds.b at 2024-07-26T08:42:10.536 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 bash[71440]: 577ae053058a83a25989eb93ca58f3143e316703171e0343ea7c0bcc4453559c 2024-07-26T08:42:10.536 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:10 smithi116 systemd[1]: Started Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: pgmap v591: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-07-26T08:42:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: daemon mds.b restarted 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:boot 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:10.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:10 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: pgmap v591: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-07-26T08:42:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: daemon mds.b restarted 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:boot 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:11.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:10 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: pgmap v591: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: daemon mds.b restarted 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:boot 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:10 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T08:42:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:resolve 2024-07-26T08:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:11 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:resolve 2024-07-26T08:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:11 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T08:42:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T08:42:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:42:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T08:42:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:42:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:resolve 2024-07-26T08:42:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:11 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:12 smithi044 ceph-mon[32404]: pgmap v593: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-07-26T08:42:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:12 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:reconnect 2024-07-26T08:42:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:12 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:12 smithi116 ceph-mon[37097]: pgmap v593: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-07-26T08:42:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:12 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:reconnect 2024-07-26T08:42:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:12 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:12 smithi171 ceph-mon[37789]: pgmap v593: 129 pgs: 129 active+clean; 60 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-07-26T08:42:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:12 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:reconnect 2024-07-26T08:42:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:12 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:13 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0099998 2024-07-26T08:42:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:13 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:42:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:13 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:rejoin 2024-07-26T08:42:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:13 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:13 smithi044 ceph-mon[32404]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:42:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:13 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0099998 2024-07-26T08:42:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:13 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:42:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:13 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:rejoin 2024-07-26T08:42:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:13 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:13 smithi116 ceph-mon[37097]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:42:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:13 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0099998 2024-07-26T08:42:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:13 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:42:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:13 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:rejoin 2024-07-26T08:42:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:13 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:13 smithi171 ceph-mon[37789]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:42:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:14 smithi044 ceph-mon[32404]: pgmap v594: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:42:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:14 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:14 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:42:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:14 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:active 2024-07-26T08:42:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:14 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:42:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:14 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-mon[37789]: pgmap v594: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:active 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:14.808 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: *** Caught signal (Aborted) ** 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: in thread 7f44f0f2d640 thread_name:ms_dispatch 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44f63046f0] 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44f635194c] 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 3: raise() 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 4: abort() 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44f62ee71b] 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7f44f62fd386] 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563abcf546b9] 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563abd11280c] 2024-07-26T08:42:14.809 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563abd060cde] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563abd022763] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563abd023a1b] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563abd007752] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563abcf55a56] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563abcf59637] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563abcf59f5e] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563abcf3d03a] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44f6bd1407] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7f44f6bd1cf9] 2024-07-26T08:42:14.810 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44f6cae5a1] 2024-07-26T08:42:14.811 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7f44f6a5bcdd] 2024-07-26T08:42:14.811 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44f6a5bcf9] 2024-07-26T08:42:14.811 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44f634fc02] 2024-07-26T08:42:14.811 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44f63d4c40] 2024-07-26T08:42:14.811 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.461+0000 7f44f0f2d640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: in thread 7f44f0f2d640 thread_name:ms_dispatch 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44f63046f0] 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44f635194c] 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 3: raise() 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 4: abort() 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44f62ee71b] 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7f44f62fd386] 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563abcf546b9] 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563abd11280c] 2024-07-26T08:42:14.812 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563abd060cde] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563abd022763] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563abd023a1b] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563abd007752] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563abcf55a56] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563abcf59637] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563abcf59f5e] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563abcf3d03a] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44f6bd1407] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7f44f6bd1cf9] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44f6cae5a1] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7f44f6a5bcdd] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44f6a5bcf9] 2024-07-26T08:42:14.813 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44f634fc02] 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44f63d4c40] 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 0> 2024-07-26T08:42:14.461+0000 7f44f0f2d640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: in thread 7f44f0f2d640 thread_name:ms_dispatch 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44f63046f0] 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44f635194c] 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 3: raise() 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 4: abort() 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44f62ee71b] 2024-07-26T08:42:14.814 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7f44f62fd386] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563abcf546b9] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563abd11280c] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563abd060cde] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563abd022763] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563abd023a1b] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563abd007752] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563abcf55a56] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563abcf59637] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563abcf59f5e] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563abcf3d03a] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44f6bd1407] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7f44f6bd1cf9] 2024-07-26T08:42:14.815 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44f6cae5a1] 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7f44f6a5bcdd] 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44f6a5bcf9] 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44f634fc02] 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44f63d4c40] 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 0> 2024-07-26T08:42:14.461+0000 7f44f0f2d640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: in thread 7f44f0f2d640 thread_name:ms_dispatch 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44f63046f0] 2024-07-26T08:42:14.816 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44f635194c] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 3: raise() 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 4: abort() 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44f62ee71b] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7f44f62fd386] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563abcf546b9] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563abd11280c] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563abd060cde] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563abd022763] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563abd023a1b] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563abd007752] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563abcf55a56] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563abcf59637] 2024-07-26T08:42:14.817 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563abcf59f5e] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563abcf3d03a] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44f6bd1407] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7f44f6bd1cf9] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44f6cae5a1] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7f44f6a5bcdd] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44f6a5bcf9] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44f634fc02] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44f63d4c40] 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[72890]: 2024-07-26T08:42:14.818 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:14 smithi171 podman[73058]: 2024-07-26 08:42:14.52271137 +0000 UTC m=+0.019696530 container died 4b2af47f36c714d8bc6f2cf3f4d6ae97931ceb8f3c15be06624c75244077191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T08:42:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:14 smithi116 ceph-mon[37097]: pgmap v594: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:42:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:14 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:14 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:42:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:14 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1871502001,v1:172.21.15.171:6835/1871502001] up:active 2024-07-26T08:42:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:14 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T08:42:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:14 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:15.481 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:15 smithi171 podman[73058]: 2024-07-26 08:42:15.226762111 +0000 UTC m=+0.723747309 container remove 4b2af47f36c714d8bc6f2cf3f4d6ae97931ceb8f3c15be06624c75244077191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:42:15.481 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:15 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:42:15.836 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:15 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:42:15.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:15 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.386s CPU time. 2024-07-26T08:42:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:16 smithi044 ceph-mon[32404]: pgmap v595: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:42:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:16 smithi116 ceph-mon[37097]: pgmap v595: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:42:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:16 smithi171 ceph-mon[37789]: pgmap v595: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:42:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:18 smithi044 ceph-mon[32404]: pgmap v596: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:42:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:18 smithi116 ceph-mon[37097]: pgmap v596: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:42:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:18 smithi171 ceph-mon[37789]: pgmap v596: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:42:20.968 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:20 smithi116 ceph-mon[37097]: pgmap v597: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:20 smithi044 ceph-mon[32404]: pgmap v597: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:20 smithi171 ceph-mon[37789]: pgmap v597: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:23.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:22 smithi044 ceph-mon[32404]: pgmap v598: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 192 B/s wr, 14 op/s 2024-07-26T08:42:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:22 smithi116 ceph-mon[37097]: pgmap v598: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 192 B/s wr, 14 op/s 2024-07-26T08:42:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:22 smithi171 ceph-mon[37789]: pgmap v598: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 192 B/s wr, 14 op/s 2024-07-26T08:42:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:24 smithi044 ceph-mon[32404]: pgmap v599: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:42:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:24 smithi116 ceph-mon[37097]: pgmap v599: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:42:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:24 smithi171 ceph-mon[37789]: pgmap v599: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:42:25.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:25.122+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:42:25.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:25.122+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:42:25.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:25.122+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:25.936 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:25 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:42:25.937 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:25 smithi171 systemd[1]: Stopped Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:25.937 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:25 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.386s CPU time. 2024-07-26T08:42:25.937 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:25 smithi171 systemd[1]: Starting Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:42:25.937 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:25 smithi171 podman[73149]: 2024-07-26 08:42:25.722092687 +0000 UTC m=+0.017559805 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:42:26.215 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:25 smithi171 podman[73149]: 2024-07-26 08:42:25.936085462 +0000 UTC m=+0.231552621 container create 851b919e78cd9d885da340b5973ed94100ab4404f43303c5fde4ac5078004c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:42:26.216 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:26 smithi171 podman[73149]: 2024-07-26 08:42:26.181046226 +0000 UTC m=+0.476513353 container init 851b919e78cd9d885da340b5973ed94100ab4404f43303c5fde4ac5078004c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:42:26.216 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:26 smithi171 podman[73149]: 2024-07-26 08:42:26.183802605 +0000 UTC m=+0.479269725 container start 851b919e78cd9d885da340b5973ed94100ab4404f43303c5fde4ac5078004c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T08:42:26.486 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mds[73168]: -- 172.21.15.171:0/501067809 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc13f8d860 con 0x55bc1812fa80 2024-07-26T08:42:26.486 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: starting mds.f at 2024-07-26T08:42:26.486 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:26 smithi171 bash[73149]: 851b919e78cd9d885da340b5973ed94100ab4404f43303c5fde4ac5078004c30 2024-07-26T08:42:26.486 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:26 smithi171 systemd[1]: Started Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: pgmap v600: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: daemon mds.f restarted 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:boot 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:26 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: pgmap v600: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: daemon mds.f restarted 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:boot 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:26 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: pgmap v600: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: daemon mds.f restarted 2024-07-26T08:42:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:42:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T08:42:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:boot 2024-07-26T08:42:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:42:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:26 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:28 smithi044 ceph-mon[32404]: pgmap v602: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:42:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:28 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:resolve 2024-07-26T08:42:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:28 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:28 smithi116 ceph-mon[37097]: pgmap v602: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:42:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:28 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:resolve 2024-07-26T08:42:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:28 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:28 smithi171 ceph-mon[37789]: pgmap v602: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:42:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:28 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:resolve 2024-07-26T08:42:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:28 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:29 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00199996 2024-07-26T08:42:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:29 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:reconnect 2024-07-26T08:42:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:29 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:29 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999982 2024-07-26T08:42:29.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:29 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00199996 2024-07-26T08:42:29.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:29 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:reconnect 2024-07-26T08:42:29.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:29 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:29.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:29 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999982 2024-07-26T08:42:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:29 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00199996 2024-07-26T08:42:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:29 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:reconnect 2024-07-26T08:42:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:29 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:29 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999982 2024-07-26T08:42:30.464 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: pgmap v603: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:42:30.464 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:rejoin 2024-07-26T08:42:30.464 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:active 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: *** Caught signal (Aborted) ** 2024-07-26T08:42:30.465 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: in thread 7f9596401640 thread_name:ms_dispatch 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f959b7d86f0] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f959b82594c] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 3: raise() 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 4: abort() 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 5: /lib64/libc.so.6(+0x2871b) [0x7f959b7c271b] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 6: /lib64/libc.so.6(+0x37386) [0x7f959b7d1386] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5582d1b0c6b9] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5582d1cca80c] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5582d1c18cde] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5582d1bda763] 2024-07-26T08:42:30.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5582d1bdba1b] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5582d1bbf752] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5582d1b0da56] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5582d1b11637] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5582d1b11f5e] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5582d1af503a] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f959c0a5407] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 18: (DispatchQueue::entry()+0x60f) [0x7f959c0a5cf9] 2024-07-26T08:42:30.467 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f959c1825a1] 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 20: (Thread::entry_wrapper()+0x43) [0x7f959bf2fcdd] 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 21: (Thread::_entry_func(void*)+0xd) [0x7f959bf2fcf9] 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 22: /lib64/libc.so.6(+0x89c02) [0x7f959b823c02] 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f959b8a8c40] 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.071+0000 7f9596401640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: in thread 7f9596401640 thread_name:ms_dispatch 2024-07-26T08:42:30.468 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f959b7d86f0] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f959b82594c] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 3: raise() 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 4: abort() 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 5: /lib64/libc.so.6(+0x2871b) [0x7f959b7c271b] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 6: /lib64/libc.so.6(+0x37386) [0x7f959b7d1386] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5582d1b0c6b9] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5582d1cca80c] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5582d1c18cde] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5582d1bda763] 2024-07-26T08:42:30.469 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5582d1bdba1b] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5582d1bbf752] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5582d1b0da56] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5582d1b11637] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5582d1b11f5e] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5582d1af503a] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f959c0a5407] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 18: (DispatchQueue::entry()+0x60f) [0x7f959c0a5cf9] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f959c1825a1] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 20: (Thread::entry_wrapper()+0x43) [0x7f959bf2fcdd] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 21: (Thread::_entry_func(void*)+0xd) [0x7f959bf2fcf9] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 22: /lib64/libc.so.6(+0x89c02) [0x7f959b823c02] 2024-07-26T08:42:30.470 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f959b8a8c40] 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 0> 2024-07-26T08:42:30.071+0000 7f9596401640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: in thread 7f9596401640 thread_name:ms_dispatch 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f959b7d86f0] 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f959b82594c] 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 3: raise() 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 4: abort() 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 5: /lib64/libc.so.6(+0x2871b) [0x7f959b7c271b] 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 6: /lib64/libc.so.6(+0x37386) [0x7f959b7d1386] 2024-07-26T08:42:30.471 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5582d1b0c6b9] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5582d1cca80c] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5582d1c18cde] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5582d1bda763] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5582d1bdba1b] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5582d1bbf752] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5582d1b0da56] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5582d1b11637] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5582d1b11f5e] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5582d1af503a] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f959c0a5407] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 18: (DispatchQueue::entry()+0x60f) [0x7f959c0a5cf9] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f959c1825a1] 2024-07-26T08:42:30.472 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 20: (Thread::entry_wrapper()+0x43) [0x7f959bf2fcdd] 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 21: (Thread::_entry_func(void*)+0xd) [0x7f959bf2fcf9] 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 22: /lib64/libc.so.6(+0x89c02) [0x7f959b823c02] 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f959b8a8c40] 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 0> 2024-07-26T08:42:30.071+0000 7f9596401640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: in thread 7f9596401640 thread_name:ms_dispatch 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f959b7d86f0] 2024-07-26T08:42:30.473 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f959b82594c] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 3: raise() 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 4: abort() 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 5: /lib64/libc.so.6(+0x2871b) [0x7f959b7c271b] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 6: /lib64/libc.so.6(+0x37386) [0x7f959b7d1386] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5582d1b0c6b9] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5582d1cca80c] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5582d1c18cde] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5582d1bda763] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5582d1bdba1b] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5582d1bbf752] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5582d1b0da56] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5582d1b11637] 2024-07-26T08:42:30.474 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5582d1b11f5e] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5582d1af503a] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f959c0a5407] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 18: (DispatchQueue::entry()+0x60f) [0x7f959c0a5cf9] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f959c1825a1] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 20: (Thread::entry_wrapper()+0x43) [0x7f959bf2fcdd] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 21: (Thread::_entry_func(void*)+0xd) [0x7f959bf2fcf9] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 22: /lib64/libc.so.6(+0x89c02) [0x7f959b823c02] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f959b8a8c40] 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71457]: 2024-07-26T08:42:30.475 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 podman[71541]: 2024-07-26 08:42:30.127891344 +0000 UTC m=+0.018203948 container died 577ae053058a83a25989eb93ca58f3143e316703171e0343ea7c0bcc4453559c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:42:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: pgmap v603: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:42:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:rejoin 2024-07-26T08:42:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:42:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:42:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:active 2024-07-26T08:42:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:30 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: pgmap v603: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:rejoin 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2831649562,v1:172.21.15.116:6839/2831649562] up:active 2024-07-26T08:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:30 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:31.273 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 podman[71541]: 2024-07-26 08:42:30.806390582 +0000 UTC m=+0.696703186 container remove 577ae053058a83a25989eb93ca58f3143e316703171e0343ea7c0bcc4453559c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-26T08:42:31.273 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:30 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:42:31.273 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:31 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T08:42:31.273 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:31 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.359s CPU time. 2024-07-26T08:42:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:32 smithi171 ceph-mon[37789]: pgmap v604: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:42:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:32 smithi044 ceph-mon[32404]: pgmap v604: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:42:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:32 smithi116 ceph-mon[37097]: pgmap v604: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:42:34.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:34 smithi116 ceph-mon[37097]: pgmap v605: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:34 smithi171 ceph-mon[37789]: pgmap v605: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:34 smithi044 ceph-mon[32404]: pgmap v605: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:36 smithi171 ceph-mon[37789]: pgmap v606: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:36 smithi044 ceph-mon[32404]: pgmap v606: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:36 smithi116 ceph-mon[37097]: pgmap v606: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:38 smithi171 ceph-mon[37789]: pgmap v607: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 189 B/s wr, 15 op/s 2024-07-26T08:42:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:38 smithi044 ceph-mon[32404]: pgmap v607: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 189 B/s wr, 15 op/s 2024-07-26T08:42:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:38 smithi116 ceph-mon[37097]: pgmap v607: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 189 B/s wr, 15 op/s 2024-07-26T08:42:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:40.122+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:42:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:40.124+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:42:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:40.124+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:42:40.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:40.124+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:42:40.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:40.124+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:40 smithi044 ceph-mon[32404]: pgmap v608: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:42:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:40 smithi171 ceph-mon[37789]: pgmap v608: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:42:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:40 smithi116 ceph-mon[37097]: pgmap v608: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:42:41.525 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:42:41.525 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 systemd[1]: Stopped Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:41.525 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.359s CPU time. 2024-07-26T08:42:41.525 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 systemd[1]: Starting Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:42:41.525 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 podman[71633]: 2024-07-26 08:42:41.4097912 +0000 UTC m=+0.017810854 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:42:41.525 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 podman[71633]: 2024-07-26 08:42:41.523947121 +0000 UTC m=+0.131966771 container create 6c62847eaf4ac527fd6e55638b990ab2092680b0aed3e0285df20bbcc05bdaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, ceph=True) 2024-07-26T08:42:41.786 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 podman[71633]: 2024-07-26 08:42:41.751528062 +0000 UTC m=+0.359547716 container init 6c62847eaf4ac527fd6e55638b990ab2092680b0aed3e0285df20bbcc05bdaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-07-26T08:42:41.786 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 podman[71633]: 2024-07-26 08:42:41.754024108 +0000 UTC m=+0.362043759 container start 6c62847eaf4ac527fd6e55638b990ab2092680b0aed3e0285df20bbcc05bdaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-07-26T08:42:42.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 ceph-mds[71652]: -- 172.21.15.116:0/606669396 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0b3e13860 con 0x55b0b7fb5a80 2024-07-26T08:42:42.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: starting mds.b at 2024-07-26T08:42:42.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 bash[71633]: 6c62847eaf4ac527fd6e55638b990ab2092680b0aed3e0285df20bbcc05bdaba 2024-07-26T08:42:42.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:42:41 smithi116 systemd[1]: Started Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: pgmap v609: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: daemon mds.b restarted 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:boot 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:42 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: pgmap v609: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: daemon mds.b restarted 2024-07-26T08:42:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:42:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T08:42:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:boot 2024-07-26T08:42:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:42:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:42 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: pgmap v609: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: daemon mds.b restarted 2024-07-26T08:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T08:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T08:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:boot 2024-07-26T08:42:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:42:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:42 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:43 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:resolve 2024-07-26T08:42:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:43 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:43 smithi171 ceph-mon[37789]: pgmap v611: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:42:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:43 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:resolve 2024-07-26T08:42:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:43 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:43 smithi044 ceph-mon[32404]: pgmap v611: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:42:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:43 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:resolve 2024-07-26T08:42:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:43 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:43 smithi116 ceph-mon[37097]: pgmap v611: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:42:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:44 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:42:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:44 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:reconnect 2024-07-26T08:42:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:44 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:44 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:42:45.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:42:45.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: *** Caught signal (Aborted) ** 2024-07-26T08:42:45.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: in thread 7ff101f19640 thread_name:ms_dispatch 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff1072f06f0] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff10733d94c] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 3: raise() 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 4: abort() 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff1072da71b] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 6: /lib64/libc.so.6(+0x37386) [0x7ff1072e9386] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bc12fbc6b9] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bc1317a80c] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bc130c8cde] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bc1308a763] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bc1308ba1b] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bc1306f752] 2024-07-26T08:42:45.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bc12fbda56] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bc12fc1637] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bc12fc1f5e] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bc12fa503a] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff107bbd407] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 18: (DispatchQueue::entry()+0x60f) [0x7ff107bbdcf9] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff107c9a5a1] 2024-07-26T08:42:45.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 20: (Thread::entry_wrapper()+0x43) [0x7ff107a47cdd] 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff107a47cf9] 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff10733bc02] 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff1073c0c40] 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.071+0000 7ff101f19640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: in thread 7ff101f19640 thread_name:ms_dispatch 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff1072f06f0] 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff10733d94c] 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 3: raise() 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 4: abort() 2024-07-26T08:42:45.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff1072da71b] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 6: /lib64/libc.so.6(+0x37386) [0x7ff1072e9386] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bc12fbc6b9] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bc1317a80c] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bc130c8cde] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bc1308a763] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bc1308ba1b] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bc1306f752] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bc12fbda56] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bc12fc1637] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bc12fc1f5e] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bc12fa503a] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff107bbd407] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 18: (DispatchQueue::entry()+0x60f) [0x7ff107bbdcf9] 2024-07-26T08:42:45.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff107c9a5a1] 2024-07-26T08:42:45.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 20: (Thread::entry_wrapper()+0x43) [0x7ff107a47cdd] 2024-07-26T08:42:45.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff107a47cf9] 2024-07-26T08:42:45.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff10733bc02] 2024-07-26T08:42:45.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff1073c0c40] 2024-07-26T08:42:45.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:45.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:44 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:42:45.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:44 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:reconnect 2024-07-26T08:42:45.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:44 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:45.113 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:44 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:42:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:44 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:42:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:44 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:reconnect 2024-07-26T08:42:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:44 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:42:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:44 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 0> 2024-07-26T08:42:45.071+0000 7ff101f19640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: in thread 7ff101f19640 thread_name:ms_dispatch 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff1072f06f0] 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff10733d94c] 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 3: raise() 2024-07-26T08:42:45.438 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 4: abort() 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff1072da71b] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 6: /lib64/libc.so.6(+0x37386) [0x7ff1072e9386] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bc12fbc6b9] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bc1317a80c] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bc130c8cde] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bc1308a763] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bc1308ba1b] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bc1306f752] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bc12fbda56] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bc12fc1637] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bc12fc1f5e] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bc12fa503a] 2024-07-26T08:42:45.439 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff107bbd407] 2024-07-26T08:42:45.440 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 18: (DispatchQueue::entry()+0x60f) [0x7ff107bbdcf9] 2024-07-26T08:42:45.440 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff107c9a5a1] 2024-07-26T08:42:45.440 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 20: (Thread::entry_wrapper()+0x43) [0x7ff107a47cdd] 2024-07-26T08:42:45.440 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff107a47cf9] 2024-07-26T08:42:45.440 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff10733bc02] 2024-07-26T08:42:45.440 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff1073c0c40] 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 0> 2024-07-26T08:42:45.071+0000 7ff101f19640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: in thread 7ff101f19640 thread_name:ms_dispatch 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff1072f06f0] 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff10733d94c] 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 3: raise() 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 4: abort() 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff1072da71b] 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 6: /lib64/libc.so.6(+0x37386) [0x7ff1072e9386] 2024-07-26T08:42:45.441 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bc12fbc6b9] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bc1317a80c] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bc130c8cde] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bc1308a763] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bc1308ba1b] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bc1306f752] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bc12fbda56] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bc12fc1637] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bc12fc1f5e] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bc12fa503a] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff107bbd407] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 18: (DispatchQueue::entry()+0x60f) [0x7ff107bbdcf9] 2024-07-26T08:42:45.442 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff107c9a5a1] 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 20: (Thread::entry_wrapper()+0x43) [0x7ff107a47cdd] 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff107a47cf9] 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff10733bc02] 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff1073c0c40] 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73164]: 2024-07-26T08:42:45.443 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 podman[73216]: 2024-07-26 08:42:45.133905561 +0000 UTC m=+0.019227504 container died 851b919e78cd9d885da340b5973ed94100ab4404f43303c5fde4ac5078004c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T08:42:46.040 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 podman[73216]: 2024-07-26 08:42:45.779466739 +0000 UTC m=+0.664788669 container remove 851b919e78cd9d885da340b5973ed94100ab4404f43303c5fde4ac5078004c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T08:42:46.040 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:45 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:rejoin 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: pgmap v612: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:active 2024-07-26T08:42:46.041 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:45 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:rejoin 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: pgmap v612: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:active 2024-07-26T08:42:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:45 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:rejoin 2024-07-26T08:42:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:42:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:42:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:42:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:42:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: pgmap v612: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:42:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1085336191,v1:172.21.15.171:6835/1085336191] up:active 2024-07-26T08:42:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:45 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:42:46.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:46 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:42:46.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:46 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.405s CPU time. 2024-07-26T08:42:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:48 smithi044 ceph-mon[32404]: pgmap v613: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:42:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:48 smithi116 ceph-mon[37097]: pgmap v613: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:42:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:48 smithi171 ceph-mon[37789]: pgmap v613: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:42:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:50 smithi044 ceph-mon[32404]: pgmap v614: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:50 smithi116 ceph-mon[37097]: pgmap v614: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:50 smithi171 ceph-mon[37789]: pgmap v614: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:52 smithi044 ceph-mon[32404]: pgmap v615: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:52 smithi116 ceph-mon[37097]: pgmap v615: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:52 smithi171 ceph-mon[37789]: pgmap v615: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:42:54.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:54 smithi044 ceph-mon[32404]: pgmap v616: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 181 B/s wr, 15 op/s 2024-07-26T08:42:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:54 smithi116 ceph-mon[37097]: pgmap v616: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 181 B/s wr, 15 op/s 2024-07-26T08:42:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:54 smithi171 ceph-mon[37789]: pgmap v616: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 181 B/s wr, 15 op/s 2024-07-26T08:42:55.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:55.131+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:42:55.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:55.131+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:55.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:42:55.131+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:42:56.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:56 smithi171 ceph-mon[37789]: pgmap v617: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:56.340 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:42:56.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 systemd[1]: Stopped Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:56.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.405s CPU time. 2024-07-26T08:42:56.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 systemd[1]: Starting Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:42:56.341 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 podman[73308]: 2024-07-26 08:42:56.22336907 +0000 UTC m=+0.017613868 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:42:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:56 smithi044 ceph-mon[32404]: pgmap v617: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:56 smithi116 ceph-mon[37097]: pgmap v617: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:42:56.610 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 podman[73308]: 2024-07-26 08:42:56.339941834 +0000 UTC m=+0.134186621 container create 8b35a22b669df59caeaf5658b05b22ad0353aa94a4f873ea6a05d8e5e6f5a80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:42:57.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 podman[73308]: 2024-07-26 08:42:56.610206204 +0000 UTC m=+0.404450992 container init 8b35a22b669df59caeaf5658b05b22ad0353aa94a4f873ea6a05d8e5e6f5a80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T08:42:57.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 podman[73308]: 2024-07-26 08:42:56.612986917 +0000 UTC m=+0.407231713 container start 8b35a22b669df59caeaf5658b05b22ad0353aa94a4f873ea6a05d8e5e6f5a80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T08:42:57.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 ceph-mds[73327]: -- 172.21.15.171:0/2493883795 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c664b5b860 con 0x55c668cfda80 2024-07-26T08:42:57.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: starting mds.f at 2024-07-26T08:42:57.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 bash[73308]: 8b35a22b669df59caeaf5658b05b22ad0353aa94a4f873ea6a05d8e5e6f5a80d 2024-07-26T08:42:57.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:42:56 smithi171 systemd[1]: Started Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:42:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: daemon mds.f restarted 2024-07-26T08:42:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:42:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T08:42:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:boot 2024-07-26T08:42:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:42:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:57 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: daemon mds.f restarted 2024-07-26T08:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T08:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:boot 2024-07-26T08:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:42:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:57 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: daemon mds.f restarted 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:boot 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T08:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:57 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T08:42:58.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:58 smithi044 ceph-mon[32404]: pgmap v619: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:42:58.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:58 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:resolve 2024-07-26T08:42:58.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:58 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:58 smithi116 ceph-mon[37097]: pgmap v619: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:42:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:58 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:resolve 2024-07-26T08:42:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:58 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:42:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:58 smithi171 ceph-mon[37789]: pgmap v619: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:42:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:58 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:resolve 2024-07-26T08:42:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:58 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:59 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:43:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:59 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:reconnect 2024-07-26T08:43:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:59 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:59 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:43:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:42:59 smithi044 ceph-mon[32404]: pgmap v620: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:43:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:59 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:43:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:59 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:reconnect 2024-07-26T08:43:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:59 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:59 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:43:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:42:59 smithi116 ceph-mon[37097]: pgmap v620: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:43:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:59 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:43:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:59 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:reconnect 2024-07-26T08:43:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:59 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:59 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:43:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:42:59 smithi171 ceph-mon[37789]: pgmap v620: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: *** Caught signal (Aborted) ** 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: in thread 7f80fc095640 thread_name:ms_dispatch 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f810146c6f0] 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81014b994c] 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 3: raise() 2024-07-26T08:43:00.431 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 4: abort() 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 5: /lib64/libc.so.6(+0x2871b) [0x7f810145671b] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 6: /lib64/libc.so.6(+0x37386) [0x7f8101465386] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0b1cd26b9] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0b1e9080c] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b0b1ddecde] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b0b1da0763] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0b1da1a1b] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0b1d85752] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0b1cd3a56] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0b1cd7637] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0b1cd7f5e] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0b1cbb03a] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8101d39407] 2024-07-26T08:43:00.432 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 18: (DispatchQueue::entry()+0x60f) [0x7f8101d39cf9] 2024-07-26T08:43:00.433 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8101e165a1] 2024-07-26T08:43:00.433 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 20: (Thread::entry_wrapper()+0x43) [0x7f8101bc3cdd] 2024-07-26T08:43:00.433 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8101bc3cf9] 2024-07-26T08:43:00.433 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81014b7c02] 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f810153cc40] 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.081+0000 7f80fc095640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: in thread 7f80fc095640 thread_name:ms_dispatch 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f810146c6f0] 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81014b994c] 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 3: raise() 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 4: abort() 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 5: /lib64/libc.so.6(+0x2871b) [0x7f810145671b] 2024-07-26T08:43:00.434 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 6: /lib64/libc.so.6(+0x37386) [0x7f8101465386] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0b1cd26b9] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0b1e9080c] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b0b1ddecde] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b0b1da0763] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0b1da1a1b] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0b1d85752] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0b1cd3a56] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0b1cd7637] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0b1cd7f5e] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0b1cbb03a] 2024-07-26T08:43:00.435 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8101d39407] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 18: (DispatchQueue::entry()+0x60f) [0x7f8101d39cf9] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8101e165a1] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 20: (Thread::entry_wrapper()+0x43) [0x7f8101bc3cdd] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8101bc3cf9] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81014b7c02] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f810153cc40] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 0> 2024-07-26T08:43:00.081+0000 7f80fc095640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: in thread 7f80fc095640 thread_name:ms_dispatch 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f810146c6f0] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81014b994c] 2024-07-26T08:43:00.436 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 3: raise() 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 4: abort() 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 5: /lib64/libc.so.6(+0x2871b) [0x7f810145671b] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 6: /lib64/libc.so.6(+0x37386) [0x7f8101465386] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0b1cd26b9] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0b1e9080c] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b0b1ddecde] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b0b1da0763] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0b1da1a1b] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0b1d85752] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0b1cd3a56] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0b1cd7637] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0b1cd7f5e] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0b1cbb03a] 2024-07-26T08:43:00.437 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8101d39407] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 18: (DispatchQueue::entry()+0x60f) [0x7f8101d39cf9] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8101e165a1] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 20: (Thread::entry_wrapper()+0x43) [0x7f8101bc3cdd] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8101bc3cf9] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81014b7c02] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f810153cc40] 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 0> 2024-07-26T08:43:00.081+0000 7f80fc095640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: in thread 7f80fc095640 thread_name:ms_dispatch 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:00.438 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f810146c6f0] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81014b994c] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 3: raise() 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 4: abort() 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 5: /lib64/libc.so.6(+0x2871b) [0x7f810145671b] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 6: /lib64/libc.so.6(+0x37386) [0x7f8101465386] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0b1cd26b9] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0b1e9080c] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b0b1ddecde] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b0b1da0763] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0b1da1a1b] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0b1d85752] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0b1cd3a56] 2024-07-26T08:43:00.439 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0b1cd7637] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0b1cd7f5e] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0b1cbb03a] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8101d39407] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 18: (DispatchQueue::entry()+0x60f) [0x7f8101d39cf9] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8101e165a1] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 20: (Thread::entry_wrapper()+0x43) [0x7f8101bc3cdd] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8101bc3cf9] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81014b7c02] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f810153cc40] 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b[71648]: 2024-07-26T08:43:00.440 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 podman[71688]: 2024-07-26 08:43:00.139570441 +0000 UTC m=+0.019289688 container died 6c62847eaf4ac527fd6e55638b990ab2092680b0aed3e0285df20bbcc05bdaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:43:00.987 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 podman[71688]: 2024-07-26 08:43:00.756137156 +0000 UTC m=+0.635856437 container remove 6c62847eaf4ac527fd6e55638b990ab2092680b0aed3e0285df20bbcc05bdaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:43:00.987 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:00 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:43:00.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:rejoin 2024-07-26T08:43:00.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:43:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:43:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:active 2024-07-26T08:43:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:00 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:rejoin 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:active 2024-07-26T08:43:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:00 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:rejoin 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.116:6838/2354428855,v1:172.21.15.116:6839/2354428855] up:active 2024-07-26T08:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:00 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:01.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:01 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T08:43:01.272 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:01 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.355s CPU time. 2024-07-26T08:43:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:01 smithi044 ceph-mon[32404]: pgmap v621: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 11 op/s 2024-07-26T08:43:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:01 smithi116 ceph-mon[37097]: pgmap v621: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 11 op/s 2024-07-26T08:43:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:01 smithi171 ceph-mon[37789]: pgmap v621: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 11 op/s 2024-07-26T08:43:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:04 smithi044 ceph-mon[32404]: pgmap v622: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:43:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:04 smithi116 ceph-mon[37097]: pgmap v622: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:43:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:04 smithi171 ceph-mon[37789]: pgmap v622: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:43:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:06 smithi044 ceph-mon[32404]: pgmap v623: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:06 smithi116 ceph-mon[37097]: pgmap v623: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:06 smithi171 ceph-mon[37789]: pgmap v623: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:08.388 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:08 smithi171 ceph-mon[37789]: pgmap v624: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 196 B/s wr, 14 op/s 2024-07-26T08:43:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:08 smithi044 ceph-mon[32404]: pgmap v624: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 196 B/s wr, 14 op/s 2024-07-26T08:43:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:08 smithi116 ceph-mon[37097]: pgmap v624: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 196 B/s wr, 14 op/s 2024-07-26T08:43:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:10.134+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:10.136+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:43:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:10.136+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:43:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:10.136+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:43:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:10.136+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:10 smithi044 ceph-mon[32404]: pgmap v625: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:10 smithi116 ceph-mon[37097]: pgmap v625: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:10 smithi171 ceph-mon[37789]: pgmap v625: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:11.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:11 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:43:11.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:11 smithi116 systemd[1]: Stopped Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:43:11.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:11 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.355s CPU time. 2024-07-26T08:43:11.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:11 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Start request repeated too quickly. 2024-07-26T08:43:11.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:11 smithi116 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T08:43:11.523 INFO:journalctl@ceph.mds.b.smithi116.stdout:Jul 26 08:43:11 smithi116 systemd[1]: Failed to start Ceph mds.b for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:43:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:12 smithi044 ceph-mon[32404]: pgmap v626: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:43:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:43:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:12 smithi116 ceph-mon[37097]: pgmap v626: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:43:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:43:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:12 smithi171 ceph-mon[37789]: pgmap v626: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:43:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:43:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:14 smithi044 ceph-mon[32404]: pgmap v627: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:43:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:14 smithi116 ceph-mon[37097]: pgmap v627: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:43:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:14 smithi171 ceph-mon[37789]: pgmap v627: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:43:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:15 smithi044 ceph-mon[32404]: Replacing daemon mds.b as rank 0 with standby daemon mds.f 2024-07-26T08:43:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:15 smithi044 ceph-mon[32404]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T08:43:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:15 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:15 smithi044 ceph-mon[32404]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T08:43:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:15 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:15 smithi116 ceph-mon[37097]: Replacing daemon mds.b as rank 0 with standby daemon mds.f 2024-07-26T08:43:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:15 smithi116 ceph-mon[37097]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T08:43:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:15 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:15 smithi116 ceph-mon[37097]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T08:43:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:15 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:15 smithi171 ceph-mon[37789]: Replacing daemon mds.b as rank 0 with standby daemon mds.f 2024-07-26T08:43:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:15 smithi171 ceph-mon[37789]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T08:43:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:15 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:15 smithi171 ceph-mon[37789]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T08:43:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:15 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:16 smithi044 ceph-mon[32404]: pgmap v629: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:43:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:16 smithi116 ceph-mon[37097]: pgmap v629: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:43:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:16 smithi171 ceph-mon[37789]: pgmap v629: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:43:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:17 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:resolve 2024-07-26T08:43:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:17 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:17 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:resolve 2024-07-26T08:43:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:17 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:17 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:resolve 2024-07-26T08:43:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:17 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:18 smithi044 ceph-mon[32404]: pgmap v630: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:18 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:43:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:18 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:reconnect 2024-07-26T08:43:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:18 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:18 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:43:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:18 smithi116 ceph-mon[37097]: pgmap v630: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:18 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:43:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:18 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:reconnect 2024-07-26T08:43:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:18 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:18 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:43:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:18 smithi171 ceph-mon[37789]: pgmap v630: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:18 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:43:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:18 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:reconnect 2024-07-26T08:43:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:18 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:18 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:43:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:19 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:rejoin 2024-07-26T08:43:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:19 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:19 smithi044 ceph-mon[32404]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:43:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:19 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:rejoin 2024-07-26T08:43:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:19 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:19 smithi116 ceph-mon[37097]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:43:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:rejoin 2024-07-26T08:43:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-mon[37789]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:43:19.587 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:43:19.587 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: *** Caught signal (Aborted) ** 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: in thread 7f9aff928640 thread_name:ms_dispatch 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b04cff6f0] 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b04d4c94c] 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 3: raise() 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 4: abort() 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b04ce971b] 2024-07-26T08:43:19.588 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b04cf8386] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c661cc16b9] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c661e7f80c] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55c661dcdcde] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55c661d8f763] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c661d90a1b] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c661d74752] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c661cc2a56] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c661cc6637] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c661cc6f5e] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c661caa03a] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b055cc407] 2024-07-26T08:43:19.589 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b055cccf9] 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b056a95a1] 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b05456cdd] 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b05456cf9] 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b04d4ac02] 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b04dcfc40] 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.234+0000 7f9aff928640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: in thread 7f9aff928640 thread_name:ms_dispatch 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:19.590 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b04cff6f0] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b04d4c94c] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 3: raise() 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 4: abort() 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b04ce971b] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b04cf8386] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c661cc16b9] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c661e7f80c] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55c661dcdcde] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55c661d8f763] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c661d90a1b] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c661d74752] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c661cc2a56] 2024-07-26T08:43:19.591 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c661cc6637] 2024-07-26T08:43:19.592 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c661cc6f5e] 2024-07-26T08:43:19.592 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c661caa03a] 2024-07-26T08:43:19.592 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b055cc407] 2024-07-26T08:43:19.592 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b055cccf9] 2024-07-26T08:43:19.592 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b056a95a1] 2024-07-26T08:43:19.592 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b05456cdd] 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b05456cf9] 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b04d4ac02] 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b04dcfc40] 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 0> 2024-07-26T08:43:19.234+0000 7f9aff928640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: in thread 7f9aff928640 thread_name:ms_dispatch 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.593 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b04cff6f0] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b04d4c94c] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 3: raise() 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 4: abort() 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b04ce971b] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b04cf8386] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c661cc16b9] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c661e7f80c] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55c661dcdcde] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55c661d8f763] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c661d90a1b] 2024-07-26T08:43:19.594 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c661d74752] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c661cc2a56] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c661cc6637] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c661cc6f5e] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c661caa03a] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b055cc407] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b055cccf9] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b056a95a1] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b05456cdd] 2024-07-26T08:43:19.595 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b05456cf9] 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b04d4ac02] 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b04dcfc40] 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 0> 2024-07-26T08:43:19.234+0000 7f9aff928640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: in thread 7f9aff928640 thread_name:ms_dispatch 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9b04cff6f0] 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9b04d4c94c] 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 3: raise() 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 4: abort() 2024-07-26T08:43:19.596 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9b04ce971b] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 6: /lib64/libc.so.6(+0x37386) [0x7f9b04cf8386] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c661cc16b9] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c661e7f80c] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55c661dcdcde] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55c661d8f763] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c661d90a1b] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c661d74752] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c661cc2a56] 2024-07-26T08:43:19.597 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c661cc6637] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c661cc6f5e] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c661caa03a] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9b055cc407] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 18: (DispatchQueue::entry()+0x60f) [0x7f9b055cccf9] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9b056a95a1] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 20: (Thread::entry_wrapper()+0x43) [0x7f9b05456cdd] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9b05456cf9] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9b04d4ac02] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9b04dcfc40] 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73323]: 2024-07-26T08:43:19.598 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 podman[73491]: 2024-07-26 08:43:19.29592133 +0000 UTC m=+0.020077265 container died 8b35a22b669df59caeaf5658b05b22ad0353aa94a4f873ea6a05d8e5e6f5a80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T08:43:20.152 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 podman[73491]: 2024-07-26 08:43:19.900137292 +0000 UTC m=+0.624293220 container remove 8b35a22b669df59caeaf5658b05b22ad0353aa94a4f873ea6a05d8e5e6f5a80d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-26T08:43:20.152 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:19 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:43:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:20 smithi044 ceph-mon[32404]: pgmap v631: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:20 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:20 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:43:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:20 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:active 2024-07-26T08:43:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:20 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:20 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:20 smithi116 ceph-mon[37097]: pgmap v631: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:20 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:20 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:43:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:20 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:active 2024-07-26T08:43:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:20 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:20 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:20 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:20 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.368s CPU time. 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:20 smithi171 ceph-mon[37789]: pgmap v631: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:20 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:20 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:20 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/223053470,v1:172.21.15.171:6835/223053470] up:active 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:20 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:20 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:22 smithi044 ceph-mon[32404]: pgmap v632: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:43:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:22 smithi116 ceph-mon[37097]: pgmap v632: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:43:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:22 smithi171 ceph-mon[37789]: pgmap v632: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:43:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:24 smithi044 ceph-mon[32404]: pgmap v633: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:43:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:24 smithi116 ceph-mon[37097]: pgmap v633: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:43:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:24 smithi171 ceph-mon[37789]: pgmap v633: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:43:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:25.144+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:43:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:25.144+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:25.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:25.144+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:43:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:26 smithi044 ceph-mon[32404]: pgmap v634: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:26 smithi116 ceph-mon[37097]: pgmap v634: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:26 smithi171 ceph-mon[37789]: pgmap v634: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:28 smithi044 ceph-mon[32404]: pgmap v635: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:43:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:28 smithi116 ceph-mon[37097]: pgmap v635: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:43:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:28 smithi171 ceph-mon[37789]: pgmap v635: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:43:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:30 smithi044 ceph-mon[32404]: pgmap v636: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:30 smithi116 ceph-mon[37097]: pgmap v636: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:30 smithi171 ceph-mon[37789]: pgmap v636: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:30.570 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:43:30.570 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 systemd[1]: Stopped Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:43:30.570 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.368s CPU time. 2024-07-26T08:43:30.570 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 systemd[1]: Starting Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:43:30.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 podman[73582]: 2024-07-26 08:43:30.472120372 +0000 UTC m=+0.018009904 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:43:30.837 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 podman[73582]: 2024-07-26 08:43:30.59298731 +0000 UTC m=+0.138876845 container create 1541f998e9cf9c1ae2fd1834cb051ab2fc70c642e42b30d3f0429ebbc2f93be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:43:31.108 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 podman[73582]: 2024-07-26 08:43:30.837908928 +0000 UTC m=+0.383798470 container init 1541f998e9cf9c1ae2fd1834cb051ab2fc70c642e42b30d3f0429ebbc2f93be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T08:43:31.108 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 podman[73582]: 2024-07-26 08:43:30.840536521 +0000 UTC m=+0.386426055 container start 1541f998e9cf9c1ae2fd1834cb051ab2fc70c642e42b30d3f0429ebbc2f93be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:43:31.108 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 ceph-mds[73602]: -- 172.21.15.171:0/2292562863 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ea64971860 con 0x55ea68b13180 2024-07-26T08:43:31.109 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: starting mds.f at 2024-07-26T08:43:31.109 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 bash[73582]: 1541f998e9cf9c1ae2fd1834cb051ab2fc70c642e42b30d3f0429ebbc2f93be0 2024-07-26T08:43:31.109 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:30 smithi171 systemd[1]: Started Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:43:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: daemon mds.f restarted 2024-07-26T08:43:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:43:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T08:43:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:boot 2024-07-26T08:43:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:43:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T08:43:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:31 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: daemon mds.f restarted 2024-07-26T08:43:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:43:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T08:43:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:boot 2024-07-26T08:43:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:43:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T08:43:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:31 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: daemon mds.f restarted 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:boot 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:43:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T08:43:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:31 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:32 smithi171 ceph-mon[37789]: pgmap v638: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 204 B/s wr, 13 op/s 2024-07-26T08:43:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:32 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:resolve 2024-07-26T08:43:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:32 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:32 smithi044 ceph-mon[32404]: pgmap v638: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 204 B/s wr, 13 op/s 2024-07-26T08:43:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:32 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:resolve 2024-07-26T08:43:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:32 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:32 smithi116 ceph-mon[37097]: pgmap v638: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 204 B/s wr, 13 op/s 2024-07-26T08:43:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:32 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:resolve 2024-07-26T08:43:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:32 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:33 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:43:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:33 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:reconnect 2024-07-26T08:43:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:33 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:33 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999985 2024-07-26T08:43:34.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:33 smithi044 ceph-mon[32404]: pgmap v639: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 op/s 2024-07-26T08:43:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:33 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:43:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:33 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:reconnect 2024-07-26T08:43:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:33 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:33 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999985 2024-07-26T08:43:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:33 smithi116 ceph-mon[37097]: pgmap v639: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 op/s 2024-07-26T08:43:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:33 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:43:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:33 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:reconnect 2024-07-26T08:43:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:33 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:33 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999985 2024-07-26T08:43:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:33 smithi171 ceph-mon[37789]: pgmap v639: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 op/s 2024-07-26T08:43:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:rejoin 2024-07-26T08:43:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-mon[32404]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:43:35.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:43:35.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: *** Caught signal (Aborted) ** 2024-07-26T08:43:35.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: in thread 7f1cb0c00640 thread_name:ms_dispatch 2024-07-26T08:43:35.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1cb6e3e6f0] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1cb6e8b94c] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 3: raise() 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 4: abort() 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1cb6e2871b] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 6: /lib64/libc.so.6(+0x37386) [0x7f1cb6e37386] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b33f5b06b9] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b33f76e80c] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b33f6bccde] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b33f67e763] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b33f67fa1b] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b33f663752] 2024-07-26T08:43:35.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b33f5b1a56] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b33f5b5637] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b33f5b5f5e] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b33f59903a] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1cb799b407] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 18: (DispatchQueue::entry()+0x60f) [0x7f1cb799bcf9] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1cb7a785a1] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 20: (Thread::entry_wrapper()+0x43) [0x7f1cb7825cdd] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1cb7825cf9] 2024-07-26T08:43:35.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1cb6e89c02] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1cb6f0ec40] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:34.995+0000 7f1cb0c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: in thread 7f1cb0c00640 thread_name:ms_dispatch 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1cb6e3e6f0] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1cb6e8b94c] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 3: raise() 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 4: abort() 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1cb6e2871b] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 6: /lib64/libc.so.6(+0x37386) [0x7f1cb6e37386] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b33f5b06b9] 2024-07-26T08:43:35.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b33f76e80c] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b33f6bccde] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b33f67e763] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b33f67fa1b] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b33f663752] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b33f5b1a56] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b33f5b5637] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b33f5b5f5e] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b33f59903a] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1cb799b407] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 18: (DispatchQueue::entry()+0x60f) [0x7f1cb799bcf9] 2024-07-26T08:43:35.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1cb7a785a1] 2024-07-26T08:43:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:34 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:rejoin 2024-07-26T08:43:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:34 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:34 smithi116 ceph-mon[37097]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:43:35.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 20: (Thread::entry_wrapper()+0x43) [0x7f1cb7825cdd] 2024-07-26T08:43:35.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1cb7825cf9] 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1cb6e89c02] 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1cb6f0ec40] 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:34 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 0> 2024-07-26T08:43:34.995+0000 7f1cb0c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: in thread 7f1cb0c00640 thread_name:ms_dispatch 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1cb6e3e6f0] 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1cb6e8b94c] 2024-07-26T08:43:35.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 3: raise() 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 4: abort() 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1cb6e2871b] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 6: /lib64/libc.so.6(+0x37386) [0x7f1cb6e37386] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b33f5b06b9] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b33f76e80c] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b33f6bccde] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b33f67e763] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b33f67fa1b] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b33f663752] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b33f5b1a56] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b33f5b5637] 2024-07-26T08:43:35.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b33f5b5f5e] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b33f59903a] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1cb799b407] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 18: (DispatchQueue::entry()+0x60f) [0x7f1cb799bcf9] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1cb7a785a1] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 20: (Thread::entry_wrapper()+0x43) [0x7f1cb7825cdd] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1cb7825cf9] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1cb6e89c02] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1cb6f0ec40] 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 0> 2024-07-26T08:43:34.995+0000 7f1cb0c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: in thread 7f1cb0c00640 thread_name:ms_dispatch 2024-07-26T08:43:35.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f1cb6e3e6f0] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f1cb6e8b94c] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 3: raise() 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 4: abort() 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 5: /lib64/libc.so.6(+0x2871b) [0x7f1cb6e2871b] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 6: /lib64/libc.so.6(+0x37386) [0x7f1cb6e37386] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b33f5b06b9] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b33f76e80c] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b33f6bccde] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b33f67e763] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b33f67fa1b] 2024-07-26T08:43:35.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b33f663752] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b33f5b1a56] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b33f5b5637] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b33f5b5f5e] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b33f59903a] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f1cb799b407] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 18: (DispatchQueue::entry()+0x60f) [0x7f1cb799bcf9] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1cb7a785a1] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 20: (Thread::entry_wrapper()+0x43) [0x7f1cb7825cdd] 2024-07-26T08:43:35.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1cb7825cf9] 2024-07-26T08:43:35.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 22: /lib64/libc.so.6(+0x89c02) [0x7f1cb6e89c02] 2024-07-26T08:43:35.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f1cb6f0ec40] 2024-07-26T08:43:35.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:35.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[79652]: 2024-07-26T08:43:35.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 podman[105696]: 2024-07-26 08:43:35.132995857 +0000 UTC m=+0.037619063 container died e06cea071add4b91342a9670f51f33717549b0f6b33062a248629211642e9dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T08:43:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:34 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:rejoin 2024-07-26T08:43:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:34 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:34 smithi171 ceph-mon[37789]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:43:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:43:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:active 2024-07-26T08:43:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:35 smithi044 ceph-mon[32404]: pgmap v640: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:36.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:35 smithi044 podman[105696]: 2024-07-26 08:43:35.992015867 +0000 UTC m=+0.896639122 container remove e06cea071add4b91342a9670f51f33717549b0f6b33062a248629211642e9dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:43:36.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:36 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:35 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:35 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:35 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:active 2024-07-26T08:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:35 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:35 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:35 smithi116 ceph-mon[37097]: pgmap v640: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:35 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:35 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:35 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/4084470430,v1:172.21.15.44:6841/4084470430] up:active 2024-07-26T08:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:35 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:35 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:35 smithi171 ceph-mon[37789]: pgmap v640: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:43:37.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:36 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T08:43:37.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:36 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 4.266s CPU time. 2024-07-26T08:43:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:38 smithi044 ceph-mon[32404]: pgmap v641: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:43:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:38 smithi116 ceph-mon[37097]: pgmap v641: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:43:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:38 smithi171 ceph-mon[37789]: pgmap v641: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:43:40.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:40.144+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:43:40.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:40.146+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T08:43:40.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:40.146+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T08:43:40.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:40.146+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:43:40.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:40.146+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:40 smithi044 ceph-mon[32404]: pgmap v642: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:40 smithi116 ceph-mon[37097]: pgmap v642: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:40 smithi171 ceph-mon[37789]: pgmap v642: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:42 smithi044 ceph-mon[32404]: pgmap v643: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 200 B/s wr, 14 op/s 2024-07-26T08:43:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:42 smithi116 ceph-mon[37097]: pgmap v643: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 200 B/s wr, 14 op/s 2024-07-26T08:43:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:42 smithi171 ceph-mon[37789]: pgmap v643: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 200 B/s wr, 14 op/s 2024-07-26T08:43:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:44 smithi044 ceph-mon[32404]: pgmap v644: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:44 smithi116 ceph-mon[37097]: pgmap v644: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:44 smithi171 ceph-mon[37789]: pgmap v644: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:43:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:46 smithi044 ceph-mon[32404]: pgmap v645: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:43:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:46 smithi116 ceph-mon[37097]: pgmap v645: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:43:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:46 smithi171 ceph-mon[37789]: pgmap v645: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:43:47.166 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:46 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-07-26T08:43:47.167 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:46 smithi044 systemd[1]: Stopped Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:43:47.167 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:46 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 4.266s CPU time. 2024-07-26T08:43:47.167 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:46 smithi044 systemd[1]: Starting Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:43:47.508 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 podman[105793]: 2024-07-26 08:43:47.068927374 +0000 UTC m=+0.031388504 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:43:47.508 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 podman[105793]: 2024-07-26 08:43:47.193744438 +0000 UTC m=+0.156205623 container create 6a03c47e424b7cb33fcb3a36e5b8c47f00dbad1bd2be518004b52a26229d7efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T08:43:47.508 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 podman[105793]: 2024-07-26 08:43:47.506266843 +0000 UTC m=+0.468728004 container init 6a03c47e424b7cb33fcb3a36e5b8c47f00dbad1bd2be518004b52a26229d7efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-26T08:43:47.766 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 podman[105793]: 2024-07-26 08:43:47.518071543 +0000 UTC m=+0.480532674 container start 6a03c47e424b7cb33fcb3a36e5b8c47f00dbad1bd2be518004b52a26229d7efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T08:43:47.767 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 bash[105793]: 6a03c47e424b7cb33fcb3a36e5b8c47f00dbad1bd2be518004b52a26229d7efb 2024-07-26T08:43:47.767 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 ceph-mds[105812]: -- 172.21.15.44:0/2787234891 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561e07e19860 con 0x561e0bfbb180 2024-07-26T08:43:47.767 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 systemd[1]: Started Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:43:47.767 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:43:47 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: starting mds.a at 2024-07-26T08:43:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: pgmap v646: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:43:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: daemon mds.a restarted 2024-07-26T08:43:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T08:43:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:boot 2024-07-26T08:43:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:43:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T08:43:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:43:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:48 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: pgmap v646: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: daemon mds.a restarted 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:boot 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:43:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T08:43:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:43:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:48 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: pgmap v646: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:43:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: daemon mds.a restarted 2024-07-26T08:43:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:43:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T08:43:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:boot 2024-07-26T08:43:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T08:43:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T08:43:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:43:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:48 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T08:43:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:49 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:resolve 2024-07-26T08:43:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:49 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:49 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:resolve 2024-07-26T08:43:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:49 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:49 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:resolve 2024-07-26T08:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:49 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T08:43:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:50 smithi044 ceph-mon[32404]: pgmap v648: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:43:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:50 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:43:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:50 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:reconnect 2024-07-26T08:43:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:50 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:50 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:43:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:50 smithi116 ceph-mon[37097]: pgmap v648: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:43:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:50 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:43:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:50 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:reconnect 2024-07-26T08:43:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:50 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:50 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:43:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:50 smithi171 ceph-mon[37789]: pgmap v648: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:43:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:50 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299994 2024-07-26T08:43:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:50 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:reconnect 2024-07-26T08:43:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:50 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:43:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:50 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0099998 2024-07-26T08:43:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:51 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:rejoin 2024-07-26T08:43:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:51 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:51 smithi044 ceph-mon[32404]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:43:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:51 smithi044 ceph-mon[32404]: pgmap v649: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:43:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:51 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:rejoin 2024-07-26T08:43:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:51 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:51 smithi116 ceph-mon[37097]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:43:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:51 smithi116 ceph-mon[37097]: pgmap v649: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:rejoin 2024-07-26T08:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-mon[37789]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T08:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-mon[37789]: pgmap v649: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: *** Caught signal (Aborted) ** 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: in thread 7f44d55f5640 thread_name:ms_dispatch 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44da9cc6f0] 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44daa1994c] 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 3: raise() 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 4: abort() 2024-07-26T08:43:52.088 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44da9b671b] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 6: /lib64/libc.so.6(+0x37386) [0x7f44da9c5386] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ea62a1d6b9] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ea62bdb80c] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ea62b29cde] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ea62aeb763] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ea62aeca1b] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ea62ad0752] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ea62a1ea56] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ea62a22637] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ea62a22f5e] 2024-07-26T08:43:52.089 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ea62a0603a] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44db299407] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 18: (DispatchQueue::entry()+0x60f) [0x7f44db299cf9] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44db3765a1] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 20: (Thread::entry_wrapper()+0x43) [0x7f44db123cdd] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44db123cf9] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44daa17c02] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44daa9cc40] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:51.714+0000 7f44d55f5640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: in thread 7f44d55f5640 thread_name:ms_dispatch 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44da9cc6f0] 2024-07-26T08:43:52.090 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44daa1994c] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 3: raise() 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 4: abort() 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44da9b671b] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 6: /lib64/libc.so.6(+0x37386) [0x7f44da9c5386] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ea62a1d6b9] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ea62bdb80c] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ea62b29cde] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ea62aeb763] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ea62aeca1b] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ea62ad0752] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ea62a1ea56] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ea62a22637] 2024-07-26T08:43:52.091 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ea62a22f5e] 2024-07-26T08:43:52.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ea62a0603a] 2024-07-26T08:43:52.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44db299407] 2024-07-26T08:43:52.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 18: (DispatchQueue::entry()+0x60f) [0x7f44db299cf9] 2024-07-26T08:43:52.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44db3765a1] 2024-07-26T08:43:52.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 20: (Thread::entry_wrapper()+0x43) [0x7f44db123cdd] 2024-07-26T08:43:52.092 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44db123cf9] 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44daa17c02] 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44daa9cc40] 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 0> 2024-07-26T08:43:51.714+0000 7f44d55f5640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: in thread 7f44d55f5640 thread_name:ms_dispatch 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44da9cc6f0] 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44daa1994c] 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 3: raise() 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 4: abort() 2024-07-26T08:43:52.093 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44da9b671b] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 6: /lib64/libc.so.6(+0x37386) [0x7f44da9c5386] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ea62a1d6b9] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ea62bdb80c] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ea62b29cde] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ea62aeb763] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ea62aeca1b] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ea62ad0752] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ea62a1ea56] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ea62a22637] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ea62a22f5e] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ea62a0603a] 2024-07-26T08:43:52.094 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44db299407] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 18: (DispatchQueue::entry()+0x60f) [0x7f44db299cf9] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44db3765a1] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 20: (Thread::entry_wrapper()+0x43) [0x7f44db123cdd] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44db123cf9] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44daa17c02] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44daa9cc40] 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 0> 2024-07-26T08:43:51.714+0000 7f44d55f5640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: in thread 7f44d55f5640 thread_name:ms_dispatch 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:52.095 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f44da9cc6f0] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f44daa1994c] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 3: raise() 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 4: abort() 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 5: /lib64/libc.so.6(+0x2871b) [0x7f44da9b671b] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 6: /lib64/libc.so.6(+0x37386) [0x7f44da9c5386] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ea62a1d6b9] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ea62bdb80c] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ea62b29cde] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ea62aeb763] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ea62aeca1b] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ea62ad0752] 2024-07-26T08:43:52.096 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ea62a1ea56] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ea62a22637] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ea62a22f5e] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ea62a0603a] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f44db299407] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 18: (DispatchQueue::entry()+0x60f) [0x7f44db299cf9] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f44db3765a1] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 20: (Thread::entry_wrapper()+0x43) [0x7f44db123cdd] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 21: (Thread::_entry_func(void*)+0xd) [0x7f44db123cf9] 2024-07-26T08:43:52.097 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 22: /lib64/libc.so.6(+0x89c02) [0x7f44daa17c02] 2024-07-26T08:43:52.098 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f44daa9cc40] 2024-07-26T08:43:52.098 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:43:52.098 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f[73597]: 2024-07-26T08:43:52.098 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:51 smithi171 podman[73649]: 2024-07-26 08:43:51.7805694 +0000 UTC m=+0.019435107 container died 1541f998e9cf9c1ae2fd1834cb051ab2fc70c642e42b30d3f0429ebbc2f93be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T08:43:52.728 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:52 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:52.728 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:52 smithi171 ceph-mon[37789]: Cluster is now healthy 2024-07-26T08:43:52.728 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:52 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:active 2024-07-26T08:43:52.728 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:52 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:52.729 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:52 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:52.729 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:52 smithi171 podman[73649]: 2024-07-26 08:43:52.462663191 +0000 UTC m=+0.701528890 container remove 1541f998e9cf9c1ae2fd1834cb051ab2fc70c642e42b30d3f0429ebbc2f93be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T08:43:52.729 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:52 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:43:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:52 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:52 smithi044 ceph-mon[32404]: Cluster is now healthy 2024-07-26T08:43:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:52 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:active 2024-07-26T08:43:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:52 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:52 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:52 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:43:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:52 smithi116 ceph-mon[37097]: Cluster is now healthy 2024-07-26T08:43:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:52 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.171:6834/1036519184,v1:172.21.15.171:6835/1036519184] up:active 2024-07-26T08:43:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:52 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T08:43:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:52 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:53.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:52 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:43:53.087 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:43:52 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.475s CPU time. 2024-07-26T08:43:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:53 smithi044 ceph-mon[32404]: pgmap v650: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:43:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:53 smithi116 ceph-mon[37097]: pgmap v650: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:43:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:53 smithi171 ceph-mon[37789]: pgmap v650: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:43:55.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:55.148+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:43:55.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:55.148+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:43:55.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:43:55.148+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:56 smithi044 ceph-mon[32404]: pgmap v651: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:43:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:56 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 180.549952 secs 2024-07-26T08:43:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:56 smithi044 ceph-mon[32404]: slow request 180.549951 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:43:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:56 smithi116 ceph-mon[37097]: pgmap v651: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:43:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:56 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 180.549952 secs 2024-07-26T08:43:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:56 smithi116 ceph-mon[37097]: slow request 180.549951 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:43:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:56 smithi171 ceph-mon[37789]: pgmap v651: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:43:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:56 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 180.549952 secs 2024-07-26T08:43:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:56 smithi171 ceph-mon[37789]: slow request 180.549951 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:43:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:57 smithi044 ceph-mon[32404]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T08:43:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:57 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:active 2024-07-26T08:43:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:57 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:57 smithi116 ceph-mon[37097]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T08:43:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:57 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:active 2024-07-26T08:43:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:57 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:57 smithi171 ceph-mon[37789]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T08:43:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:57 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6838/1144072288,v1:172.21.15.44:6839/1144072288] up:active 2024-07-26T08:43:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:57 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:43:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:43:58 smithi044 ceph-mon[32404]: pgmap v652: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:43:58 smithi116 ceph-mon[37097]: pgmap v652: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:43:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:43:58 smithi171 ceph-mon[37789]: pgmap v652: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:44:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:00 smithi044 ceph-mon[32404]: pgmap v653: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 178 B/s wr, 14 op/s 2024-07-26T08:44:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:00 smithi116 ceph-mon[37097]: pgmap v653: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 178 B/s wr, 14 op/s 2024-07-26T08:44:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:00 smithi171 ceph-mon[37789]: pgmap v653: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 178 B/s wr, 14 op/s 2024-07-26T08:44:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 185.550074 secs 2024-07-26T08:44:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 185.550074 secs 2024-07-26T08:44:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 185.550074 secs 2024-07-26T08:44:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:02 smithi044 ceph-mon[32404]: pgmap v654: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:02 smithi116 ceph-mon[37097]: pgmap v654: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:02 smithi171 ceph-mon[37789]: pgmap v654: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:03.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:44:02 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-07-26T08:44:03.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:44:02 smithi171 systemd[1]: Stopped Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:03.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:44:02 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Consumed 2.475s CPU time. 2024-07-26T08:44:03.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:44:02 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-07-26T08:44:03.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:44:02 smithi171 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-07-26T08:44:03.337 INFO:journalctl@ceph.mds.f.smithi171.stdout:Jul 26 08:44:02 smithi171 systemd[1]: Failed to start Ceph mds.f for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:04 smithi044 ceph-mon[32404]: pgmap v655: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:44:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:04 smithi116 ceph-mon[37097]: pgmap v655: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:44:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:04 smithi171 ceph-mon[37789]: pgmap v655: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:44:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:06 smithi044 ceph-mon[32404]: pgmap v656: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 12 op/s 2024-07-26T08:44:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 190.550177 secs 2024-07-26T08:44:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:06 smithi116 ceph-mon[37097]: pgmap v656: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 12 op/s 2024-07-26T08:44:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 190.550177 secs 2024-07-26T08:44:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:06 smithi171 ceph-mon[37789]: pgmap v656: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 12 op/s 2024-07-26T08:44:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 190.550177 secs 2024-07-26T08:44:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:08 smithi044 ceph-mon[32404]: pgmap v657: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 12 op/s 2024-07-26T08:44:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:08 smithi116 ceph-mon[37097]: pgmap v657: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 12 op/s 2024-07-26T08:44:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:08 smithi171 ceph-mon[37789]: pgmap v657: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 12 op/s 2024-07-26T08:44:10.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:10.151+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:44:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:10.151+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:10.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:10.151+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:10 smithi044 ceph-mon[32404]: pgmap v658: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:10 smithi044 ceph-mon[32404]: Replacing daemon mds.f as rank 0 with standby daemon mds.a 2024-07-26T08:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:10 smithi044 ceph-mon[32404]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T08:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:10 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:10 smithi044 ceph-mon[32404]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T08:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:10 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:10 smithi116 ceph-mon[37097]: pgmap v658: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:10 smithi116 ceph-mon[37097]: Replacing daemon mds.f as rank 0 with standby daemon mds.a 2024-07-26T08:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:10 smithi116 ceph-mon[37097]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T08:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:10 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:10 smithi116 ceph-mon[37097]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T08:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:10 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:10 smithi171 ceph-mon[37789]: pgmap v658: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:10 smithi171 ceph-mon[37789]: Replacing daemon mds.f as rank 0 with standby daemon mds.a 2024-07-26T08:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:10 smithi171 ceph-mon[37789]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-07-26T08:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:10 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:10 smithi171 ceph-mon[37789]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T08:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:10 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 195.550316 secs 2024-07-26T08:44:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 195.550316 secs 2024-07-26T08:44:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 195.550316 secs 2024-07-26T08:44:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:12 smithi044 ceph-mon[32404]: pgmap v660: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:44:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:12 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:resolve 2024-07-26T08:44:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:12 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:12 smithi116 ceph-mon[37097]: pgmap v660: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:44:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:12 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:resolve 2024-07-26T08:44:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:12 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:12 smithi171 ceph-mon[37789]: pgmap v660: 129 pgs: 129 active+clean; 60 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:44:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:12 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:resolve 2024-07-26T08:44:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:12 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:13 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:44:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:13 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:reconnect 2024-07-26T08:44:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:13 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:13 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999985 2024-07-26T08:44:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:13 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:44:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:13 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:reconnect 2024-07-26T08:44:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:13 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:13 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999985 2024-07-26T08:44:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:13 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:44:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:13 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:reconnect 2024-07-26T08:44:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:13 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:13 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999985 2024-07-26T08:44:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:14.318 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:44:14.318 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: *** Caught signal (Aborted) ** 2024-07-26T08:44:14.318 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: in thread 7f616b200640 thread_name:ms_dispatch 2024-07-26T08:44:14.318 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:14.318 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f617143e6f0] 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f617148b94c] 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 3: raise() 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 4: abort() 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: pgmap v661: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:rejoin 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:14.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:14.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 5: /lib64/libc.so.6(+0x2871b) [0x7f617142871b] 2024-07-26T08:44:14.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 6: /lib64/libc.so.6(+0x37386) [0x7f6171437386] 2024-07-26T08:44:14.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561e058026b9] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561e059c080c] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561e0590ecde] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x561e058d0763] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561e058d1a1b] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561e058b5752] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561e05803a56] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561e05807637] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561e05807f5e] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561e057eb03a] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6171f9b407] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 18: (DispatchQueue::entry()+0x60f) [0x7f6171f9bcf9] 2024-07-26T08:44:14.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61720785a1] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 20: (Thread::entry_wrapper()+0x43) [0x7f6171e25cdd] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6171e25cf9] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6171489c02] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f617150ec40] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.309+0000 7f616b200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: in thread 7f616b200640 thread_name:ms_dispatch 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f617143e6f0] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f617148b94c] 2024-07-26T08:44:14.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 3: raise() 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 4: abort() 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 5: /lib64/libc.so.6(+0x2871b) [0x7f617142871b] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 6: /lib64/libc.so.6(+0x37386) [0x7f6171437386] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561e058026b9] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561e059c080c] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561e0590ecde] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x561e058d0763] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561e058d1a1b] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561e058b5752] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561e05803a56] 2024-07-26T08:44:14.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561e05807637] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561e05807f5e] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561e057eb03a] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6171f9b407] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 18: (DispatchQueue::entry()+0x60f) [0x7f6171f9bcf9] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61720785a1] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 20: (Thread::entry_wrapper()+0x43) [0x7f6171e25cdd] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6171e25cf9] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6171489c02] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f617150ec40] 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 0> 2024-07-26T08:44:14.309+0000 7f616b200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: in thread 7f616b200640 thread_name:ms_dispatch 2024-07-26T08:44:14.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: pgmap v661: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:rejoin 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f617143e6f0] 2024-07-26T08:44:14.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f617148b94c] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 3: raise() 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 4: abort() 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 5: /lib64/libc.so.6(+0x2871b) [0x7f617142871b] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 6: /lib64/libc.so.6(+0x37386) [0x7f6171437386] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561e058026b9] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561e059c080c] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561e0590ecde] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x561e058d0763] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561e058d1a1b] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561e058b5752] 2024-07-26T08:44:14.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561e05803a56] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561e05807637] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561e05807f5e] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561e057eb03a] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6171f9b407] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 18: (DispatchQueue::entry()+0x60f) [0x7f6171f9bcf9] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61720785a1] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 20: (Thread::entry_wrapper()+0x43) [0x7f6171e25cdd] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6171e25cf9] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6171489c02] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f617150ec40] 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:14.525 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 0> 2024-07-26T08:44:14.309+0000 7f616b200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: in thread 7f616b200640 thread_name:ms_dispatch 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f617143e6f0] 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f617148b94c] 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 3: raise() 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 4: abort() 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 5: /lib64/libc.so.6(+0x2871b) [0x7f617142871b] 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 6: /lib64/libc.so.6(+0x37386) [0x7f6171437386] 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561e058026b9] 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561e059c080c] 2024-07-26T08:44:14.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561e0590ecde] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x561e058d0763] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561e058d1a1b] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561e058b5752] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561e05803a56] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561e05807637] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561e05807f5e] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561e057eb03a] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6171f9b407] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 18: (DispatchQueue::entry()+0x60f) [0x7f6171f9bcf9] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61720785a1] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 20: (Thread::entry_wrapper()+0x43) [0x7f6171e25cdd] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6171e25cf9] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6171489c02] 2024-07-26T08:44:14.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f617150ec40] 2024-07-26T08:44:14.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:14.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[105808]: 2024-07-26T08:44:14.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:14 smithi044 podman[106074]: 2024-07-26 08:44:14.466719606 +0000 UTC m=+0.048967694 container died 6a03c47e424b7cb33fcb3a36e5b8c47f00dbad1bd2be518004b52a26229d7efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: pgmap v661: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:rejoin 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:44:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:14.696+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400034c0 con 0x7fcf6c1057b0 2024-07-26T08:44:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:14.697+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400034c0 con 0x7fcf6c1057b0 2024-07-26T08:44:14.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:14.704+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0034c0 con 0x55c94889a070 2024-07-26T08:44:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:14.705+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0034c0 con 0x55c94889a070 2024-07-26T08:44:15.516 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:15 smithi044 podman[106074]: 2024-07-26 08:44:15.405519129 +0000 UTC m=+0.987767229 container remove 6a03c47e424b7cb33fcb3a36e5b8c47f00dbad1bd2be518004b52a26229d7efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T08:44:15.516 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:15 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:44:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:15 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:15 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:active 2024-07-26T08:44:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:15 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:44:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:15 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:15 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:15 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:active 2024-07-26T08:44:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:15 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:44:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:15 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:15 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:15 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/577284002,v1:172.21.15.44:6841/577284002] up:active 2024-07-26T08:44:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:15 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:44:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:15 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:16.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:16 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T08:44:16.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:16 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.848s CPU time. 2024-07-26T08:44:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:16 smithi171 ceph-mon[37789]: pgmap v662: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:44:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 200.550441 secs 2024-07-26T08:44:16.674 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:16 smithi044 ceph-mon[32404]: pgmap v662: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:44:16.674 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 200.550441 secs 2024-07-26T08:44:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:16 smithi116 ceph-mon[37097]: pgmap v662: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:44:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 200.550441 secs 2024-07-26T08:44:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:18 smithi171 ceph-mon[37789]: pgmap v663: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:44:18.594 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:18 smithi044 ceph-mon[32404]: pgmap v663: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:44:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:18 smithi116 ceph-mon[37097]: pgmap v663: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 op/s 2024-07-26T08:44:20.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:20 smithi044 ceph-mon[32404]: pgmap v664: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:44:20.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 205.550599 secs 2024-07-26T08:44:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:20 smithi171 ceph-mon[37789]: pgmap v664: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:44:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 205.550599 secs 2024-07-26T08:44:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:20 smithi116 ceph-mon[37097]: pgmap v664: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:44:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 205.550599 secs 2024-07-26T08:44:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:22 smithi171 ceph-mon[37789]: pgmap v665: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 185 B/s wr, 14 op/s 2024-07-26T08:44:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:22 smithi044 ceph-mon[32404]: pgmap v665: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 185 B/s wr, 14 op/s 2024-07-26T08:44:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:22 smithi116 ceph-mon[37097]: pgmap v665: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 185 B/s wr, 14 op/s 2024-07-26T08:44:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:24 smithi171 ceph-mon[37789]: pgmap v666: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:44:24.609 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:24 smithi044 ceph-mon[32404]: pgmap v666: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:44:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:24 smithi116 ceph-mon[37097]: pgmap v666: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:44:25.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:25.157+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:44:25.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:25.157+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:25.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:25.157+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:44:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 210.550755 secs 2024-07-26T08:44:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 210.550755 secs 2024-07-26T08:44:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 210.550755 secs 2024-07-26T08:44:26.329 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:44:26.329 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 systemd[1]: Stopped Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:26.329 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.848s CPU time. 2024-07-26T08:44:26.329 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 systemd[1]: Starting Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:44:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:26 smithi171 ceph-mon[37789]: pgmap v667: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:26.643 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:26 smithi044 ceph-mon[32404]: pgmap v667: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:26.644 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 podman[106167]: 2024-07-26 08:44:26.329868365 +0000 UTC m=+0.038614499 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:44:26.644 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 podman[106167]: 2024-07-26 08:44:26.457360212 +0000 UTC m=+0.166106358 container create 7f24e3f3290d97c78745d9843656d95cf63316d6382cf0e1d85e0ae1d2568f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-07-26T08:44:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:26 smithi116 ceph-mon[37097]: pgmap v667: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:27.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 podman[106167]: 2024-07-26 08:44:26.753550144 +0000 UTC m=+0.462296293 container init 7f24e3f3290d97c78745d9843656d95cf63316d6382cf0e1d85e0ae1d2568f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-07-26T08:44:27.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 podman[106167]: 2024-07-26 08:44:26.765907047 +0000 UTC m=+0.474653168 container start 7f24e3f3290d97c78745d9843656d95cf63316d6382cf0e1d85e0ae1d2568f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T08:44:27.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 ceph-mds[106185]: -- 172.21.15.44:0/253986648 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624e3c6bc20 con 0x5624e7e0da80 2024-07-26T08:44:27.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: starting mds.a at 2024-07-26T08:44:27.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 bash[106167]: 7f24e3f3290d97c78745d9843656d95cf63316d6382cf0e1d85e0ae1d2568f30 2024-07-26T08:44:27.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:26 smithi044 systemd[1]: Started Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: daemon mds.a restarted 2024-07-26T08:44:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T08:44:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:boot 2024-07-26T08:44:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:44:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:44:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:44:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:27 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: daemon mds.a restarted 2024-07-26T08:44:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T08:44:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:boot 2024-07-26T08:44:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:44:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:44:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:44:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:27 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: daemon mds.a restarted 2024-07-26T08:44:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T08:44:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:boot 2024-07-26T08:44:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:44:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:44:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:44:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:27 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:28 smithi171 ceph-mon[37789]: pgmap v669: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:44:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:28 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:resolve 2024-07-26T08:44:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:28 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:28.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:28 smithi044 ceph-mon[32404]: pgmap v669: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:44:28.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:28 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:resolve 2024-07-26T08:44:28.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:28 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:28 smithi116 ceph-mon[37097]: pgmap v669: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:44:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:28 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:resolve 2024-07-26T08:44:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:28 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:29 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:44:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:29 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:reconnect 2024-07-26T08:44:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:29 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:29 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:44:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:29 smithi044 ceph-mon[32404]: pgmap v670: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:44:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:29 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:44:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:29 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:reconnect 2024-07-26T08:44:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:29 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:29 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:44:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:29 smithi116 ceph-mon[37097]: pgmap v670: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:44:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:29 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:44:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:29 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:reconnect 2024-07-26T08:44:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:29 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:29 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:44:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:29 smithi171 ceph-mon[37789]: pgmap v670: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T08:44:30.453 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:44:30.453 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: *** Caught signal (Aborted) ** 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: in thread 7f2b96400640 thread_name:ms_dispatch 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2b9c63e6f0] 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2b9c68b94c] 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 3: raise() 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 4: abort() 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2b9c62871b] 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 6: /lib64/libc.so.6(+0x37386) [0x7f2b9c637386] 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56307e7366b9] 2024-07-26T08:44:30.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56307e8f480c] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56307e842cde] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56307e804763] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56307e805a1b] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56307e7e9752] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56307e737a56] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56307e73b637] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56307e73bf5e] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56307e71f03a] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2b9d19b407] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 18: (DispatchQueue::entry()+0x60f) [0x7f2b9d19bcf9] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2b9d2785a1] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 20: (Thread::entry_wrapper()+0x43) [0x7f2b9d025cdd] 2024-07-26T08:44:30.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2b9d025cf9] 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2b9c689c02] 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2b9c70ec40] 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.128+0000 7f2b96400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: in thread 7f2b96400640 thread_name:ms_dispatch 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2b9c63e6f0] 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2b9c68b94c] 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 3: raise() 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 4: abort() 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2b9c62871b] 2024-07-26T08:44:30.456 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 6: /lib64/libc.so.6(+0x37386) [0x7f2b9c637386] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56307e7366b9] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56307e8f480c] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56307e842cde] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56307e804763] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56307e805a1b] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56307e7e9752] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56307e737a56] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56307e73b637] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56307e73bf5e] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56307e71f03a] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2b9d19b407] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 18: (DispatchQueue::entry()+0x60f) [0x7f2b9d19bcf9] 2024-07-26T08:44:30.457 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2b9d2785a1] 2024-07-26T08:44:30.458 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 20: (Thread::entry_wrapper()+0x43) [0x7f2b9d025cdd] 2024-07-26T08:44:30.458 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2b9d025cf9] 2024-07-26T08:44:30.458 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2b9c689c02] 2024-07-26T08:44:30.458 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2b9c70ec40] 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 0> 2024-07-26T08:44:30.128+0000 7f2b96400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: in thread 7f2b96400640 thread_name:ms_dispatch 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2b9c63e6f0] 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2b9c68b94c] 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 3: raise() 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 4: abort() 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2b9c62871b] 2024-07-26T08:44:30.459 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 6: /lib64/libc.so.6(+0x37386) [0x7f2b9c637386] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56307e7366b9] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56307e8f480c] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56307e842cde] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56307e804763] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56307e805a1b] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56307e7e9752] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56307e737a56] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56307e73b637] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56307e73bf5e] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56307e71f03a] 2024-07-26T08:44:30.460 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2b9d19b407] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 18: (DispatchQueue::entry()+0x60f) [0x7f2b9d19bcf9] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2b9d2785a1] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 20: (Thread::entry_wrapper()+0x43) [0x7f2b9d025cdd] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2b9d025cf9] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2b9c689c02] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2b9c70ec40] 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 0> 2024-07-26T08:44:30.128+0000 7f2b96400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: in thread 7f2b96400640 thread_name:ms_dispatch 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:30.461 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2b9c63e6f0] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2b9c68b94c] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 3: raise() 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 4: abort() 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2b9c62871b] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 6: /lib64/libc.so.6(+0x37386) [0x7f2b9c637386] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56307e7366b9] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56307e8f480c] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56307e842cde] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56307e804763] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56307e805a1b] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56307e7e9752] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56307e737a56] 2024-07-26T08:44:30.462 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56307e73b637] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56307e73bf5e] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56307e71f03a] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2b9d19b407] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 18: (DispatchQueue::entry()+0x60f) [0x7f2b9d19bcf9] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2b9d2785a1] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 20: (Thread::entry_wrapper()+0x43) [0x7f2b9d025cdd] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2b9d025cf9] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2b9c689c02] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2b9c70ec40] 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[78079]: 2024-07-26T08:44:30.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:30 smithi044 podman[106221]: 2024-07-26 08:44:30.256374868 +0000 UTC m=+0.033592745 container died bd4dfc52cffd01e3bc1a5b41aebeb643578aed1231d0beeec659a99478c2146d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T08:44:31.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:31 smithi044 podman[106221]: 2024-07-26 08:44:31.093265806 +0000 UTC m=+0.870483723 container remove bd4dfc52cffd01e3bc1a5b41aebeb643578aed1231d0beeec659a99478c2146d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T08:44:31.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:31 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:rejoin 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:active 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 215.550832 secs 2024-07-26T08:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:rejoin 2024-07-26T08:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:active 2024-07-26T08:44:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 215.550832 secs 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:rejoin 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2787895500,v1:172.21.15.44:6835/2787895500] up:active 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 215.550832 secs 2024-07-26T08:44:31.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:31 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:44:31.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:31 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 4.114s CPU time. 2024-07-26T08:44:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:31 smithi044 ceph-mon[32404]: pgmap v671: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 11 op/s 2024-07-26T08:44:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:31 smithi116 ceph-mon[37097]: pgmap v671: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 11 op/s 2024-07-26T08:44:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:31 smithi171 ceph-mon[37789]: pgmap v671: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 11 op/s 2024-07-26T08:44:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:34 smithi044 ceph-mon[32404]: pgmap v672: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:44:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:34 smithi116 ceph-mon[37097]: pgmap v672: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:44:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:34 smithi171 ceph-mon[37789]: pgmap v672: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:44:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:36 smithi044 ceph-mon[32404]: pgmap v673: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:44:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 220.550999 secs 2024-07-26T08:44:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:36 smithi116 ceph-mon[37097]: pgmap v673: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:44:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 220.550999 secs 2024-07-26T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:36 smithi171 ceph-mon[37789]: pgmap v673: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:44:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 220.550999 secs 2024-07-26T08:44:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:38 smithi044 ceph-mon[32404]: pgmap v674: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 200 B/s wr, 14 op/s 2024-07-26T08:44:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:38 smithi116 ceph-mon[37097]: pgmap v674: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 200 B/s wr, 14 op/s 2024-07-26T08:44:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:38 smithi171 ceph-mon[37789]: pgmap v674: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 200 B/s wr, 14 op/s 2024-07-26T08:44:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:40.160+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:44:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:40.160+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:40.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:40.160+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:44:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:40 smithi044 ceph-mon[32404]: pgmap v675: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:44:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:40 smithi116 ceph-mon[37097]: pgmap v675: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:44:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:40 smithi171 ceph-mon[37789]: pgmap v675: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T08:44:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 225.551120 secs 2024-07-26T08:44:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 225.551120 secs 2024-07-26T08:44:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 225.551120 secs 2024-07-26T08:44:41.793 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:41 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-07-26T08:44:41.793 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:41 smithi044 systemd[1]: Stopped Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:41.793 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:41 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 4.114s CPU time. 2024-07-26T08:44:42.178 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:41 smithi044 systemd[1]: Starting Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:44:42.454 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:42 smithi044 ceph-mon[32404]: pgmap v676: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:42.454 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 podman[106321]: 2024-07-26 08:44:42.084088841 +0000 UTC m=+0.039589281 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:44:42.455 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 podman[106321]: 2024-07-26 08:44:42.279171444 +0000 UTC m=+0.234671876 container create 81394e019b796a8e2196eef9bb853d2ff9276e684f09620e441d15bb659ad756 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T08:44:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:42 smithi116 ceph-mon[37097]: pgmap v676: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:42 smithi171 ceph-mon[37789]: pgmap v676: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:44:43.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 podman[106321]: 2024-07-26 08:44:42.675628978 +0000 UTC m=+0.631129400 container init 81394e019b796a8e2196eef9bb853d2ff9276e684f09620e441d15bb659ad756 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:44:43.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 podman[106321]: 2024-07-26 08:44:42.686598288 +0000 UTC m=+0.642098710 container start 81394e019b796a8e2196eef9bb853d2ff9276e684f09620e441d15bb659ad756 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD) 2024-07-26T08:44:43.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 bash[106321]: 81394e019b796a8e2196eef9bb853d2ff9276e684f09620e441d15bb659ad756 2024-07-26T08:44:43.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 systemd[1]: Started Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:43.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 ceph-mds[106339]: -- 172.21.15.44:0/1429204940 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5649d8113860 con 0x5649dc2b5600 2024-07-26T08:44:43.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:44:42 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: starting mds.j at 2024-07-26T08:44:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: daemon mds.j restarted 2024-07-26T08:44:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T08:44:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:boot 2024-07-26T08:44:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:44:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:44:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:44:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:43 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: daemon mds.j restarted 2024-07-26T08:44:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T08:44:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:boot 2024-07-26T08:44:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:44:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:44:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:44:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:43 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: daemon mds.j restarted 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:boot 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:44:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:43 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:44:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:44 smithi044 ceph-mon[32404]: pgmap v678: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:44:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:44 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:resolve 2024-07-26T08:44:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:44 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:44 smithi116 ceph-mon[37097]: pgmap v678: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:44:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:44 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:resolve 2024-07-26T08:44:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:44 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:44 smithi171 ceph-mon[37789]: pgmap v678: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T08:44:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:44 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:resolve 2024-07-26T08:44:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:44 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:44:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:45 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:44:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:45 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:reconnect 2024-07-26T08:44:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:45 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:45 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:44:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:45 smithi044 ceph-mon[32404]: pgmap v679: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:44:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 230.551281 secs 2024-07-26T08:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:45 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:45 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:reconnect 2024-07-26T08:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:45 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:45 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:45 smithi116 ceph-mon[37097]: pgmap v679: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 230.551281 secs 2024-07-26T08:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:45 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:45 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:reconnect 2024-07-26T08:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:45 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:45 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:45 smithi171 ceph-mon[37789]: pgmap v679: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 230.551281 secs 2024-07-26T08:44:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:rejoin 2024-07-26T08:44:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-mon[32404]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:44:47.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T08:44:47.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:44:47.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T08:44:47.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T08:44:47.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: *** Caught signal (Aborted) ** 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: in thread 7eff9d200640 thread_name:ms_dispatch 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7effa343e6f0] 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2: /lib64/libc.so.6(+0x8b94c) [0x7effa348b94c] 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 3: raise() 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 4: abort() 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 5: /lib64/libc.so.6(+0x2871b) [0x7effa342871b] 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 6: /lib64/libc.so.6(+0x37386) [0x7effa3437386] 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624b833b6b9] 2024-07-26T08:44:47.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624b84f980c] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5624b8447cde] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5624b8409763] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624b840aa1b] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624b83ee752] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624b833ca56] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624b8340637] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624b8340f5e] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624b832403a] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7effa3f9b407] 2024-07-26T08:44:47.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 18: (DispatchQueue::entry()+0x60f) [0x7effa3f9bcf9] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7effa40785a1] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 20: (Thread::entry_wrapper()+0x43) [0x7effa3e25cdd] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 21: (Thread::_entry_func(void*)+0xd) [0x7effa3e25cf9] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 22: /lib64/libc.so.6(+0x89c02) [0x7effa3489c02] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 23: /lib64/libc.so.6(+0x10ec40) [0x7effa350ec40] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:46.924+0000 7eff9d200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: in thread 7eff9d200640 thread_name:ms_dispatch 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7effa343e6f0] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2: /lib64/libc.so.6(+0x8b94c) [0x7effa348b94c] 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 3: raise() 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 4: abort() 2024-07-26T08:44:47.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 5: /lib64/libc.so.6(+0x2871b) [0x7effa342871b] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 6: /lib64/libc.so.6(+0x37386) [0x7effa3437386] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624b833b6b9] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624b84f980c] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5624b8447cde] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5624b8409763] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624b840aa1b] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624b83ee752] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624b833ca56] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624b8340637] 2024-07-26T08:44:47.271 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624b8340f5e] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624b832403a] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7effa3f9b407] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 18: (DispatchQueue::entry()+0x60f) [0x7effa3f9bcf9] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7effa40785a1] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 20: (Thread::entry_wrapper()+0x43) [0x7effa3e25cdd] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 21: (Thread::_entry_func(void*)+0xd) [0x7effa3e25cf9] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 22: /lib64/libc.so.6(+0x89c02) [0x7effa3489c02] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 23: /lib64/libc.so.6(+0x10ec40) [0x7effa350ec40] 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1306> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1304> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1302> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T08:44:47.272 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1300> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1298> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 0> 2024-07-26T08:44:46.924+0000 7eff9d200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: in thread 7eff9d200640 thread_name:ms_dispatch 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7effa343e6f0] 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2: /lib64/libc.so.6(+0x8b94c) [0x7effa348b94c] 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 3: raise() 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 4: abort() 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 5: /lib64/libc.so.6(+0x2871b) [0x7effa342871b] 2024-07-26T08:44:47.273 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 6: /lib64/libc.so.6(+0x37386) [0x7effa3437386] 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:46 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:rejoin 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:46 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:46 smithi116 ceph-mon[37097]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624b833b6b9] 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624b84f980c] 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5624b8447cde] 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5624b8409763] 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624b840aa1b] 2024-07-26T08:44:47.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624b83ee752] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624b833ca56] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624b8340637] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624b8340f5e] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624b832403a] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7effa3f9b407] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 18: (DispatchQueue::entry()+0x60f) [0x7effa3f9bcf9] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7effa40785a1] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 20: (Thread::entry_wrapper()+0x43) [0x7effa3e25cdd] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 21: (Thread::_entry_func(void*)+0xd) [0x7effa3e25cf9] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 22: /lib64/libc.so.6(+0x89c02) [0x7effa3489c02] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 23: /lib64/libc.so.6(+0x10ec40) [0x7effa350ec40] 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:47.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1311> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1309> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1307> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1305> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -1303> 2024-07-26T08:44:46.909+0000 7eff9a000640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: -5> 2024-07-26T08:44:46.924+0000 7eff9d200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: in thread 7eff9d200640 thread_name:ms_dispatch 2024-07-26T08:44:47.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7effa343e6f0] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2: /lib64/libc.so.6(+0x8b94c) [0x7effa348b94c] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 3: raise() 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 4: abort() 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 5: /lib64/libc.so.6(+0x2871b) [0x7effa342871b] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 6: /lib64/libc.so.6(+0x37386) [0x7effa3437386] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624b833b6b9] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624b84f980c] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5624b8447cde] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5624b8409763] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624b840aa1b] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624b83ee752] 2024-07-26T08:44:47.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624b833ca56] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624b8340637] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624b8340f5e] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624b832403a] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7effa3f9b407] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 18: (DispatchQueue::entry()+0x60f) [0x7effa3f9bcf9] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7effa40785a1] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 20: (Thread::entry_wrapper()+0x43) [0x7effa3e25cdd] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 21: (Thread::_entry_func(void*)+0xd) [0x7effa3e25cf9] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 22: /lib64/libc.so.6(+0x89c02) [0x7effa3489c02] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 23: /lib64/libc.so.6(+0x10ec40) [0x7effa350ec40] 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:46 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106181]: 2024-07-26T08:44:47.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:47 smithi044 podman[106374]: 2024-07-26 08:44:47.072322197 +0000 UTC m=+0.036664681 container died 7f24e3f3290d97c78745d9843656d95cf63316d6382cf0e1d85e0ae1d2568f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0) 2024-07-26T08:44:47.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:46 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:rejoin 2024-07-26T08:44:47.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:46 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:44:47.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:46 smithi171 ceph-mon[37789]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:44:48.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:47 smithi044 podman[106374]: 2024-07-26 08:44:47.939666206 +0000 UTC m=+0.904008732 container remove 7f24e3f3290d97c78745d9843656d95cf63316d6382cf0e1d85e0ae1d2568f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T08:44:48.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:47 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:44:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:47 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:47 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:active 2024-07-26T08:44:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:47 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:44:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:47 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:48.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:47 smithi044 ceph-mon[32404]: pgmap v680: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:44:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:47 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:47 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:active 2024-07-26T08:44:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:47 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:44:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:47 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:47 smithi116 ceph-mon[37097]: pgmap v680: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:44:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:47 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:44:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:47 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/514689932,v1:172.21.15.44:6841/514689932] up:active 2024-07-26T08:44:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:47 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:44:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:47 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:44:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:47 smithi171 ceph-mon[37789]: pgmap v680: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 op/s 2024-07-26T08:44:48.766 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:48 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T08:44:48.766 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:48 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.859s CPU time. 2024-07-26T08:44:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:50 smithi044 ceph-mon[32404]: pgmap v681: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-07-26T08:44:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:50 smithi116 ceph-mon[37097]: pgmap v681: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-07-26T08:44:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:50 smithi171 ceph-mon[37789]: pgmap v681: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-07-26T08:44:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 235.551431 secs 2024-07-26T08:44:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 235.551431 secs 2024-07-26T08:44:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 235.551431 secs 2024-07-26T08:44:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:52 smithi044 ceph-mon[32404]: pgmap v682: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-07-26T08:44:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:52 smithi116 ceph-mon[37097]: pgmap v682: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-07-26T08:44:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:52 smithi171 ceph-mon[37789]: pgmap v682: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-07-26T08:44:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:54 smithi044 ceph-mon[32404]: pgmap v683: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-07-26T08:44:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:54 smithi116 ceph-mon[37097]: pgmap v683: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-07-26T08:44:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:54 smithi171 ceph-mon[37789]: pgmap v683: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-07-26T08:44:55.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:55.168+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:44:55.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:55.168+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:55.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:44:55.168+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:44:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:56 smithi044 ceph-mon[32404]: pgmap v684: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-07-26T08:44:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 240.551573 secs 2024-07-26T08:44:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:56 smithi116 ceph-mon[37097]: pgmap v684: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-07-26T08:44:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 240.551573 secs 2024-07-26T08:44:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:56 smithi171 ceph-mon[37789]: pgmap v684: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-07-26T08:44:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 240.551573 secs 2024-07-26T08:44:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:44:58 smithi044 ceph-mon[32404]: pgmap v685: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-07-26T08:44:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:44:58 smithi116 ceph-mon[37097]: pgmap v685: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-07-26T08:44:58.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:44:58 smithi171 ceph-mon[37789]: pgmap v685: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-07-26T08:44:58.951 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:58 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:44:58.952 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:58 smithi044 systemd[1]: Stopped Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:58.952 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:58 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.859s CPU time. 2024-07-26T08:44:58.952 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:58 smithi044 systemd[1]: Starting Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:44:59.255 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:58 smithi044 podman[106464]: 2024-07-26 08:44:58.858398167 +0000 UTC m=+0.054230652 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:44:59.255 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:58 smithi044 podman[106464]: 2024-07-26 08:44:58.975211631 +0000 UTC m=+0.171044126 container create 2d9cef1c369942162c81ac5f2c368bfe703e6b0ca15e58ae0bad78a253c5a06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T08:44:59.516 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:59 smithi044 podman[106464]: 2024-07-26 08:44:59.25454963 +0000 UTC m=+0.450382175 container init 2d9cef1c369942162c81ac5f2c368bfe703e6b0ca15e58ae0bad78a253c5a06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-07-26T08:44:59.516 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:59 smithi044 podman[106464]: 2024-07-26 08:44:59.262224879 +0000 UTC m=+0.458057358 container start 2d9cef1c369942162c81ac5f2c368bfe703e6b0ca15e58ae0bad78a253c5a06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T08:44:59.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:59 smithi044 bash[106464]: 2d9cef1c369942162c81ac5f2c368bfe703e6b0ca15e58ae0bad78a253c5a06b 2024-07-26T08:44:59.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:59 smithi044 systemd[1]: Started Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:44:59.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:59 smithi044 ceph-mds[106490]: -- 172.21.15.44:0/3559264793 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9ba063860 con 0x55a9be205a80 2024-07-26T08:44:59.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:44:59 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: starting mds.a at 2024-07-26T08:45:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: pgmap v686: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-07-26T08:45:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: daemon mds.a restarted 2024-07-26T08:45:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T08:45:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:boot 2024-07-26T08:45:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:45:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:45:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:00 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: pgmap v686: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-07-26T08:45:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: daemon mds.a restarted 2024-07-26T08:45:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T08:45:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:boot 2024-07-26T08:45:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:45:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:45:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:00 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: pgmap v686: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 KiB/s wr, 15 op/s 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: daemon mds.a restarted 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:boot 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:45:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:00 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 245.551668 secs 2024-07-26T08:45:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:01 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:resolve 2024-07-26T08:45:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:01 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 245.551668 secs 2024-07-26T08:45:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:01 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:resolve 2024-07-26T08:45:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:01 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 245.551668 secs 2024-07-26T08:45:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:01 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:resolve 2024-07-26T08:45:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:01 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:02 smithi044 ceph-mon[32404]: pgmap v688: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 11 op/s 2024-07-26T08:45:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:02 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.000999985 2024-07-26T08:45:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:02 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:reconnect 2024-07-26T08:45:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:02 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:02 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:45:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:02 smithi116 ceph-mon[37097]: pgmap v688: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 11 op/s 2024-07-26T08:45:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:02 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.000999985 2024-07-26T08:45:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:02 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:reconnect 2024-07-26T08:45:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:02 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:02 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:45:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:02 smithi171 ceph-mon[37789]: pgmap v688: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 11 op/s 2024-07-26T08:45:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:02 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.000999985 2024-07-26T08:45:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:02 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:reconnect 2024-07-26T08:45:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:02 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:02 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:45:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:rejoin 2024-07-26T08:45:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-mon[32404]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:45:03.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: *** Caught signal (Aborted) ** 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: in thread 7f0093400640 thread_name:ms_dispatch 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f009983e6f0] 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f009988b94c] 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 3: raise() 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 4: abort() 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 5: /lib64/libc.so.6(+0x2871b) [0x7f009982871b] 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 6: /lib64/libc.so.6(+0x37386) [0x7f0099837386] 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5649a523f6b9] 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5649a53fd80c] 2024-07-26T08:45:03.767 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5649a534bcde] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5649a530d763] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5649a530ea1b] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5649a52f2752] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5649a5240a56] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5649a5244637] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5649a5244f5e] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5649a522803a] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f009a39b407] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 18: (DispatchQueue::entry()+0x60f) [0x7f009a39bcf9] 2024-07-26T08:45:03.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f009a4785a1] 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 20: (Thread::entry_wrapper()+0x43) [0x7f009a225cdd] 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 21: (Thread::_entry_func(void*)+0xd) [0x7f009a225cf9] 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0099889c02] 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f009990ec40] 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.482+0000 7f0093400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: in thread 7f0093400640 thread_name:ms_dispatch 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:03.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f009983e6f0] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f009988b94c] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 3: raise() 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 4: abort() 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 5: /lib64/libc.so.6(+0x2871b) [0x7f009982871b] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 6: /lib64/libc.so.6(+0x37386) [0x7f0099837386] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5649a523f6b9] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5649a53fd80c] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5649a534bcde] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5649a530d763] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5649a530ea1b] 2024-07-26T08:45:03.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5649a52f2752] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5649a5240a56] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5649a5244637] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5649a5244f5e] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5649a522803a] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f009a39b407] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 18: (DispatchQueue::entry()+0x60f) [0x7f009a39bcf9] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f009a4785a1] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 20: (Thread::entry_wrapper()+0x43) [0x7f009a225cdd] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 21: (Thread::_entry_func(void*)+0xd) [0x7f009a225cf9] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0099889c02] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f009990ec40] 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 0> 2024-07-26T08:45:03.482+0000 7f0093400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: in thread 7f0093400640 thread_name:ms_dispatch 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f009983e6f0] 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f009988b94c] 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 3: raise() 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 4: abort() 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 5: /lib64/libc.so.6(+0x2871b) [0x7f009982871b] 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 6: /lib64/libc.so.6(+0x37386) [0x7f0099837386] 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5649a523f6b9] 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5649a53fd80c] 2024-07-26T08:45:03.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5649a534bcde] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5649a530d763] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5649a530ea1b] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5649a52f2752] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5649a5240a56] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5649a5244637] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5649a5244f5e] 2024-07-26T08:45:03.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5649a522803a] 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:03 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:rejoin 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:03 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:03 smithi116 ceph-mon[37097]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f009a39b407] 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 18: (DispatchQueue::entry()+0x60f) [0x7f009a39bcf9] 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f009a4785a1] 2024-07-26T08:45:03.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 20: (Thread::entry_wrapper()+0x43) [0x7f009a225cdd] 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 21: (Thread::_entry_func(void*)+0xd) [0x7f009a225cf9] 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0099889c02] 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f009990ec40] 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 0> 2024-07-26T08:45:03.482+0000 7f0093400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: in thread 7f0093400640 thread_name:ms_dispatch 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:03.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f009983e6f0] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f009988b94c] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 3: raise() 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 4: abort() 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 5: /lib64/libc.so.6(+0x2871b) [0x7f009982871b] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 6: /lib64/libc.so.6(+0x37386) [0x7f0099837386] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5649a523f6b9] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5649a53fd80c] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5649a534bcde] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5649a530d763] 2024-07-26T08:45:03.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5649a530ea1b] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5649a52f2752] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5649a5240a56] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5649a5244637] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5649a5244f5e] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5649a522803a] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f009a39b407] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 18: (DispatchQueue::entry()+0x60f) [0x7f009a39bcf9] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f009a4785a1] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 20: (Thread::entry_wrapper()+0x43) [0x7f009a225cdd] 2024-07-26T08:45:03.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 21: (Thread::_entry_func(void*)+0xd) [0x7f009a225cf9] 2024-07-26T08:45:03.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0099889c02] 2024-07-26T08:45:03.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f009990ec40] 2024-07-26T08:45:03.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:03.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106335]: 2024-07-26T08:45:03.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:03 smithi044 podman[106528]: 2024-07-26 08:45:03.623637628 +0000 UTC m=+0.041569321 container died 81394e019b796a8e2196eef9bb853d2ff9276e684f09620e441d15bb659ad756 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-07-26T08:45:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:03 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:rejoin 2024-07-26T08:45:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:03 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:03 smithi171 ceph-mon[37789]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:45:04.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:04 smithi044 podman[106528]: 2024-07-26 08:45:04.586047748 +0000 UTC m=+1.003979441 container remove 81394e019b796a8e2196eef9bb853d2ff9276e684f09620e441d15bb659ad756 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T08:45:04.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:04 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:45:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:04 smithi044 ceph-mon[32404]: pgmap v689: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:04 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:04 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:active 2024-07-26T08:45:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:04 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:04 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:04 smithi116 ceph-mon[37097]: pgmap v689: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:04 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:04 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:active 2024-07-26T08:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:04 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:04 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:04 smithi171 ceph-mon[37789]: pgmap v689: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:04 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:04 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3469001950,v1:172.21.15.44:6835/3469001950] up:active 2024-07-26T08:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:04 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:04 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:05.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:05 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:45:05.463 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:05 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.906s CPU time. 2024-07-26T08:45:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 250.551779 secs 2024-07-26T08:45:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 250.551779 secs 2024-07-26T08:45:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 250.551779 secs 2024-07-26T08:45:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:06 smithi044 ceph-mon[32404]: pgmap v690: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:06 smithi116 ceph-mon[37097]: pgmap v690: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:06 smithi171 ceph-mon[37789]: pgmap v690: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:08 smithi044 ceph-mon[32404]: pgmap v691: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 102 B/s wr, 13 op/s 2024-07-26T08:45:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:08 smithi116 ceph-mon[37097]: pgmap v691: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 102 B/s wr, 13 op/s 2024-07-26T08:45:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:08 smithi171 ceph-mon[37789]: pgmap v691: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 102 B/s wr, 13 op/s 2024-07-26T08:45:10.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:10.171+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:45:10.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:10.171+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:10.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:10.171+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:45:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:10 smithi044 ceph-mon[32404]: pgmap v692: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:45:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 255.551916 secs 2024-07-26T08:45:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:10 smithi116 ceph-mon[37097]: pgmap v692: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:45:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 255.551916 secs 2024-07-26T08:45:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:10 smithi171 ceph-mon[37789]: pgmap v692: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T08:45:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 255.551916 secs 2024-07-26T08:45:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:12 smithi044 ceph-mon[32404]: pgmap v693: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 174 B/s wr, 12 op/s 2024-07-26T08:45:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:12 smithi116 ceph-mon[37097]: pgmap v693: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 174 B/s wr, 12 op/s 2024-07-26T08:45:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:12 smithi171 ceph-mon[37789]: pgmap v693: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 174 B/s wr, 12 op/s 2024-07-26T08:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:14 smithi044 ceph-mon[32404]: pgmap v694: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:14 smithi116 ceph-mon[37097]: pgmap v694: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:14 smithi171 ceph-mon[37789]: pgmap v694: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T08:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 260.552059 secs 2024-07-26T08:45:15.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:15 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-07-26T08:45:15.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:15 smithi044 systemd[1]: Stopped Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:45:15.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:15 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.906s CPU time. 2024-07-26T08:45:15.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:15 smithi044 systemd[1]: Starting Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:45:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 260.552059 secs 2024-07-26T08:45:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 260.552059 secs 2024-07-26T08:45:16.265 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:15 smithi044 podman[106789]: 2024-07-26 08:45:15.893303503 +0000 UTC m=+0.034693062 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:45:16.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 podman[106789]: 2024-07-26 08:45:16.018320128 +0000 UTC m=+0.159709710 container create d69425e034c00eb0d9ca33bceda747f69629951c1528875cd2c00ba4768bfc25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:45:16.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 podman[106789]: 2024-07-26 08:45:16.306190593 +0000 UTC m=+0.447580178 container init d69425e034c00eb0d9ca33bceda747f69629951c1528875cd2c00ba4768bfc25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-07-26T08:45:16.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 podman[106789]: 2024-07-26 08:45:16.315487028 +0000 UTC m=+0.456876614 container start d69425e034c00eb0d9ca33bceda747f69629951c1528875cd2c00ba4768bfc25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:45:16.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 bash[106789]: d69425e034c00eb0d9ca33bceda747f69629951c1528875cd2c00ba4768bfc25 2024-07-26T08:45:16.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 systemd[1]: Started Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:45:16.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mds[106834]: -- 172.21.15.44:0/2541586719 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd74741c20 con 0x55cd788e3180 2024-07-26T08:45:16.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: starting mds.j at 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: pgmap v695: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 85 B/s wr, 12 op/s 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: daemon mds.j restarted 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:boot 2024-07-26T08:45:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:45:16.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:16.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:16 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: pgmap v695: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 85 B/s wr, 12 op/s 2024-07-26T08:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: daemon mds.j restarted 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:boot 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:16 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: pgmap v695: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 85 B/s wr, 12 op/s 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: daemon mds.j restarted 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T08:45:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:45:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:boot 2024-07-26T08:45:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:45:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:16 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:18.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:18 smithi044 ceph-mon[32404]: pgmap v697: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:18.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:18.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:18.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:18 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:resolve 2024-07-26T08:45:18.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:18 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:18 smithi116 ceph-mon[37097]: pgmap v697: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:18 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:resolve 2024-07-26T08:45:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:18 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:18 smithi171 ceph-mon[37789]: pgmap v697: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T08:45:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:18 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:resolve 2024-07-26T08:45:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:18 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:19 smithi044 ceph-mon[32404]: pgmap v698: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:45:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:19.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:19 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:reconnect 2024-07-26T08:45:19.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:19 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:19 smithi116 ceph-mon[37097]: pgmap v698: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:19 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:reconnect 2024-07-26T08:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:19 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:19 smithi171 ceph-mon[37789]: pgmap v698: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T08:45:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:45:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:19 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:reconnect 2024-07-26T08:45:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:19 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:45:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999984 2024-07-26T08:45:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:45:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:rejoin 2024-07-26T08:45:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:45:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:active 2024-07-26T08:45:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:20.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:45:20.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: *** Caught signal (Aborted) ** 2024-07-26T08:45:20.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: in thread 7f6950800640 thread_name:ms_dispatch 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6956a3e6f0] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6956a8b94c] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 3: raise() 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 4: abort() 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6956a2871b] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 6: /lib64/libc.so.6(+0x37386) [0x7f6956a37386] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a9a21576b9] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a9a231580c] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a9a2263cde] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a9a2225763] 2024-07-26T08:45:20.518 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a9a2226a1b] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a9a220a752] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a9a2158a56] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a9a215c637] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a9a215cf5e] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a9a214003a] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f695759b407] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 18: (DispatchQueue::entry()+0x60f) [0x7f695759bcf9] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69576785a1] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 20: (Thread::entry_wrapper()+0x43) [0x7f6957425cdd] 2024-07-26T08:45:20.519 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6957425cf9] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6956a89c02] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6956b0ec40] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.180+0000 7f6950800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: in thread 7f6950800640 thread_name:ms_dispatch 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6956a3e6f0] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6956a8b94c] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 3: raise() 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 4: abort() 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6956a2871b] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 6: /lib64/libc.so.6(+0x37386) [0x7f6956a37386] 2024-07-26T08:45:20.520 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a9a21576b9] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a9a231580c] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a9a2263cde] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a9a2225763] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a9a2226a1b] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a9a220a752] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a9a2158a56] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a9a215c637] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a9a215cf5e] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a9a214003a] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f695759b407] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 18: (DispatchQueue::entry()+0x60f) [0x7f695759bcf9] 2024-07-26T08:45:20.521 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69576785a1] 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 20: (Thread::entry_wrapper()+0x43) [0x7f6957425cdd] 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6957425cf9] 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6956a89c02] 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6956b0ec40] 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 0> 2024-07-26T08:45:20.180+0000 7f6950800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:20.522 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: in thread 7f6950800640 thread_name:ms_dispatch 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6956a3e6f0] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6956a8b94c] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 3: raise() 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 4: abort() 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6956a2871b] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 6: /lib64/libc.so.6(+0x37386) [0x7f6956a37386] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a9a21576b9] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a9a231580c] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a9a2263cde] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a9a2225763] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a9a2226a1b] 2024-07-26T08:45:20.523 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a9a220a752] 2024-07-26T08:45:20.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a9a2158a56] 2024-07-26T08:45:20.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a9a215c637] 2024-07-26T08:45:20.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a9a215cf5e] 2024-07-26T08:45:20.524 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a9a214003a] 2024-07-26T08:45:20.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:45:20.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999984 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:rejoin 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:active 2024-07-26T08:45:20.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:20 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f695759b407] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 18: (DispatchQueue::entry()+0x60f) [0x7f695759bcf9] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69576785a1] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 20: (Thread::entry_wrapper()+0x43) [0x7f6957425cdd] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6957425cf9] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6956a89c02] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6956b0ec40] 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: -13> 2024-07-26T08:45:20.180+0000 7f6950800640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: in thread 7f6950800640 thread_name:ms_dispatch 2024-07-26T08:45:20.526 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6956a3e6f0] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6956a8b94c] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 3: raise() 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 4: abort() 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6956a2871b] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 6: /lib64/libc.so.6(+0x37386) [0x7f6956a37386] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a9a21576b9] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a9a231580c] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a9a2263cde] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a9a2225763] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a9a2226a1b] 2024-07-26T08:45:20.527 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a9a220a752] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a9a2158a56] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a9a215c637] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a9a215cf5e] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a9a214003a] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f695759b407] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 18: (DispatchQueue::entry()+0x60f) [0x7f695759bcf9] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f69576785a1] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 20: (Thread::entry_wrapper()+0x43) [0x7f6957425cdd] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6957425cf9] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6956a89c02] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6956b0ec40] 2024-07-26T08:45:20.528 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:20.529 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[106481]: 2024-07-26T08:45:20.529 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:20 smithi044 podman[107129]: 2024-07-26 08:45:20.320726047 +0000 UTC m=+0.032852267 container died 2d9cef1c369942162c81ac5f2c368bfe703e6b0ca15e58ae0bad78a253c5a06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:45:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999984 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:rejoin 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/3060995873,v1:172.21.15.44:6841/3060995873] up:active 2024-07-26T08:45:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:20 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 265.552149 secs 2024-07-26T08:45:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:21 smithi044 ceph-mon[32404]: pgmap v699: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:45:21.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:21 smithi044 podman[107129]: 2024-07-26 08:45:21.271085442 +0000 UTC m=+0.983211725 container remove 2d9cef1c369942162c81ac5f2c368bfe703e6b0ca15e58ae0bad78a253c5a06b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T08:45:21.517 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:21 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:45:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 265.552149 secs 2024-07-26T08:45:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:21 smithi116 ceph-mon[37097]: pgmap v699: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:45:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 265.552149 secs 2024-07-26T08:45:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:21 smithi171 ceph-mon[37789]: pgmap v699: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T08:45:22.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:21 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T08:45:22.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:21 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.857s CPU time. 2024-07-26T08:45:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:23 smithi044 ceph-mon[32404]: pgmap v700: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 12 op/s 2024-07-26T08:45:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:23 smithi116 ceph-mon[37097]: pgmap v700: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 12 op/s 2024-07-26T08:45:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:23 smithi171 ceph-mon[37789]: pgmap v700: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 12 op/s 2024-07-26T08:45:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:25.178+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:45:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:25.178+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:45:25.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:25.178+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:25 smithi044 ceph-mon[32404]: pgmap v701: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 224 B/s wr, 16 op/s 2024-07-26T08:45:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 270.552295 secs 2024-07-26T08:45:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:25 smithi116 ceph-mon[37097]: pgmap v701: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 224 B/s wr, 16 op/s 2024-07-26T08:45:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 270.552295 secs 2024-07-26T08:45:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:25 smithi171 ceph-mon[37789]: pgmap v701: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 224 B/s wr, 16 op/s 2024-07-26T08:45:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 270.552295 secs 2024-07-26T08:45:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:27 smithi044 ceph-mon[32404]: pgmap v702: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 207 B/s wr, 14 op/s 2024-07-26T08:45:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:27 smithi116 ceph-mon[37097]: pgmap v702: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 207 B/s wr, 14 op/s 2024-07-26T08:45:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:27 smithi171 ceph-mon[37789]: pgmap v702: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 207 B/s wr, 14 op/s 2024-07-26T08:45:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:29 smithi044 ceph-mon[32404]: pgmap v703: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 183 B/s wr, 15 op/s 2024-07-26T08:45:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:29 smithi116 ceph-mon[37097]: pgmap v703: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 183 B/s wr, 15 op/s 2024-07-26T08:45:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:29 smithi171 ceph-mon[37789]: pgmap v703: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 183 B/s wr, 15 op/s 2024-07-26T08:45:30.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 275.552421 secs 2024-07-26T08:45:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 275.552421 secs 2024-07-26T08:45:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 275.552421 secs 2024-07-26T08:45:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:31 smithi044 ceph-mon[32404]: pgmap v704: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:45:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:31 smithi116 ceph-mon[37097]: pgmap v704: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:45:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:31 smithi171 ceph-mon[37789]: pgmap v704: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:45:32.449 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:45:32.449 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 systemd[1]: Stopped Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:45:32.449 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.857s CPU time. 2024-07-26T08:45:32.449 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 systemd[1]: Starting Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:45:32.762 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 podman[107231]: 2024-07-26 08:45:32.350955712 +0000 UTC m=+0.036476776 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:45:32.763 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 podman[107231]: 2024-07-26 08:45:32.498313463 +0000 UTC m=+0.183834543 container create 176f59aca70aa0e37d3e2e0c480b3d44be202e34c10250fb7c168311abf3b4fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-07-26T08:45:33.016 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 podman[107231]: 2024-07-26 08:45:32.760732014 +0000 UTC m=+0.446253097 container init 176f59aca70aa0e37d3e2e0c480b3d44be202e34c10250fb7c168311abf3b4fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux ) 2024-07-26T08:45:33.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 podman[107231]: 2024-07-26 08:45:32.771476216 +0000 UTC m=+0.456997299 container start 176f59aca70aa0e37d3e2e0c480b3d44be202e34c10250fb7c168311abf3b4fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T08:45:33.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 bash[107231]: 176f59aca70aa0e37d3e2e0c480b3d44be202e34c10250fb7c168311abf3b4fc 2024-07-26T08:45:33.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 systemd[1]: Started Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:45:33.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 ceph-mds[107250]: -- 172.21.15.44:0/1690153674 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55586fce3860 con 0x555873e85600 2024-07-26T08:45:33.017 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:32 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: starting mds.a at 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: pgmap v705: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: daemon mds.a restarted 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:boot 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:45:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:33 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: pgmap v705: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:45:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: daemon mds.a restarted 2024-07-26T08:45:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T08:45:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:boot 2024-07-26T08:45:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:45:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:45:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:33 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: pgmap v705: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: daemon mds.a restarted 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:boot 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T08:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:33 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:34 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:resolve 2024-07-26T08:45:35.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:34 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:35.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:34 smithi044 ceph-mon[32404]: pgmap v707: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:34 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:resolve 2024-07-26T08:45:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:34 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:34 smithi116 ceph-mon[37097]: pgmap v707: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:34 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:resolve 2024-07-26T08:45:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:34 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:34 smithi171 ceph-mon[37789]: pgmap v707: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:35 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:45:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:35 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:reconnect 2024-07-26T08:45:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:35 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:35 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:45:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 280.552514 secs 2024-07-26T08:45:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:35 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:45:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:35 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:reconnect 2024-07-26T08:45:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:35 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:35 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:45:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 280.552514 secs 2024-07-26T08:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:35 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:35 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:reconnect 2024-07-26T08:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:35 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:35 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 280.552514 secs 2024-07-26T08:45:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:rejoin 2024-07-26T08:45:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-mon[32404]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:45:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-mon[32404]: pgmap v708: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:37.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:45:37.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: *** Caught signal (Aborted) ** 2024-07-26T08:45:37.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: in thread 7fdc1dc00640 thread_name:ms_dispatch 2024-07-26T08:45:37.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:37.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdc23e3e6f0] 2024-07-26T08:45:37.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdc23e8b94c] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 3: raise() 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 4: abort() 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdc23e2871b] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 6: /lib64/libc.so.6(+0x37386) [0x7fdc23e37386] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd47e7d6b9] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd4803b80c] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55cd47f89cde] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55cd47f4b763] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd47f4ca1b] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd47f30752] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd47e7ea56] 2024-07-26T08:45:37.268 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd47e82637] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd47e82f5e] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd47e6603a] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdc2499b407] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 18: (DispatchQueue::entry()+0x60f) [0x7fdc2499bcf9] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdc24a785a1] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 20: (Thread::entry_wrapper()+0x43) [0x7fdc24825cdd] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdc24825cf9] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdc23e89c02] 2024-07-26T08:45:37.269 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdc23f0ec40] 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:36.994+0000 7fdc1dc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: in thread 7fdc1dc00640 thread_name:ms_dispatch 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdc23e3e6f0] 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdc23e8b94c] 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 3: raise() 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 4: abort() 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdc23e2871b] 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 6: /lib64/libc.so.6(+0x37386) [0x7fdc23e37386] 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd47e7d6b9] 2024-07-26T08:45:37.270 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd4803b80c] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55cd47f89cde] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55cd47f4b763] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd47f4ca1b] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd47f30752] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd47e7ea56] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd47e82637] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd47e82f5e] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd47e6603a] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdc2499b407] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 18: (DispatchQueue::entry()+0x60f) [0x7fdc2499bcf9] 2024-07-26T08:45:37.271 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdc24a785a1] 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 20: (Thread::entry_wrapper()+0x43) [0x7fdc24825cdd] 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdc24825cf9] 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdc23e89c02] 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdc23f0ec40] 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:36 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 0> 2024-07-26T08:45:36.994+0000 7fdc1dc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: in thread 7fdc1dc00640 thread_name:ms_dispatch 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdc23e3e6f0] 2024-07-26T08:45:37.272 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdc23e8b94c] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 3: raise() 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 4: abort() 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdc23e2871b] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 6: /lib64/libc.so.6(+0x37386) [0x7fdc23e37386] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd47e7d6b9] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd4803b80c] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55cd47f89cde] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55cd47f4b763] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd47f4ca1b] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd47f30752] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd47e7ea56] 2024-07-26T08:45:37.273 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd47e82637] 2024-07-26T08:45:37.274 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd47e82f5e] 2024-07-26T08:45:37.274 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd47e6603a] 2024-07-26T08:45:37.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:36 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:rejoin 2024-07-26T08:45:37.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:36 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:37.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:36 smithi116 ceph-mon[37097]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:36 smithi116 ceph-mon[37097]: pgmap v708: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdc2499b407] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 18: (DispatchQueue::entry()+0x60f) [0x7fdc2499bcf9] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdc24a785a1] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 20: (Thread::entry_wrapper()+0x43) [0x7fdc24825cdd] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdc24825cf9] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdc23e89c02] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdc23f0ec40] 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:37.275 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 0> 2024-07-26T08:45:36.994+0000 7fdc1dc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: in thread 7fdc1dc00640 thread_name:ms_dispatch 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdc23e3e6f0] 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdc23e8b94c] 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 3: raise() 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 4: abort() 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdc23e2871b] 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 6: /lib64/libc.so.6(+0x37386) [0x7fdc23e37386] 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cd47e7d6b9] 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cd4803b80c] 2024-07-26T08:45:37.276 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55cd47f89cde] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55cd47f4b763] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cd47f4ca1b] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cd47f30752] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cd47e7ea56] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cd47e82637] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cd47e82f5e] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cd47e6603a] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdc2499b407] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 18: (DispatchQueue::entry()+0x60f) [0x7fdc2499bcf9] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdc24a785a1] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 20: (Thread::entry_wrapper()+0x43) [0x7fdc24825cdd] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdc24825cf9] 2024-07-26T08:45:37.277 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdc23e89c02] 2024-07-26T08:45:37.278 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdc23f0ec40] 2024-07-26T08:45:37.278 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:37.278 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[106815]: 2024-07-26T08:45:37.278 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:37 smithi044 podman[107285]: 2024-07-26 08:45:37.1353989 +0000 UTC m=+0.048537803 container died d69425e034c00eb0d9ca33bceda747f69629951c1528875cd2c00ba4768bfc25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T08:45:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:36 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:rejoin 2024-07-26T08:45:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:36 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:36 smithi171 ceph-mon[37789]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:45:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:36 smithi171 ceph-mon[37789]: pgmap v708: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:45:38.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:37 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:38.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:37 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:active 2024-07-26T08:45:38.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:37 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:38.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:37 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:38.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:38 smithi044 podman[107285]: 2024-07-26 08:45:38.103619163 +0000 UTC m=+1.016758104 container remove d69425e034c00eb0d9ca33bceda747f69629951c1528875cd2c00ba4768bfc25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-26T08:45:38.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:38 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:45:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:active 2024-07-26T08:45:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:37 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:37 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:37 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/1447814352,v1:172.21.15.44:6835/1447814352] up:active 2024-07-26T08:45:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:37 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:37 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:38.969 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:38 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:45:38.969 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:38 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 4.103s CPU time. 2024-07-26T08:45:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:38 smithi044 ceph-mon[32404]: pgmap v709: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:45:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:38 smithi116 ceph-mon[37097]: pgmap v709: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:45:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:38 smithi171 ceph-mon[37789]: pgmap v709: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T08:45:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:40.181+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:45:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:40.181+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:40.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:40.181+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:45:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 285.552665 secs 2024-07-26T08:45:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 285.552665 secs 2024-07-26T08:45:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 285.552665 secs 2024-07-26T08:45:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:41 smithi044 ceph-mon[32404]: pgmap v710: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:45:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:41 smithi116 ceph-mon[37097]: pgmap v710: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:45:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:41 smithi171 ceph-mon[37789]: pgmap v710: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:45:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:43 smithi044 ceph-mon[32404]: pgmap v711: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:45:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:43 smithi116 ceph-mon[37097]: pgmap v711: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:45:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:43 smithi171 ceph-mon[37789]: pgmap v711: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T08:45:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:45 smithi044 ceph-mon[32404]: pgmap v712: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 179 B/s wr, 15 op/s 2024-07-26T08:45:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 290.552801 secs 2024-07-26T08:45:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:45 smithi116 ceph-mon[37097]: pgmap v712: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 179 B/s wr, 15 op/s 2024-07-26T08:45:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 290.552801 secs 2024-07-26T08:45:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:45 smithi171 ceph-mon[37789]: pgmap v712: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 179 B/s wr, 15 op/s 2024-07-26T08:45:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 290.552801 secs 2024-07-26T08:45:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:47 smithi044 ceph-mon[32404]: pgmap v713: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:45:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:47 smithi116 ceph-mon[37097]: pgmap v713: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:45:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:47 smithi171 ceph-mon[37789]: pgmap v713: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T08:45:49.176 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:48 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-07-26T08:45:49.176 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:48 smithi044 systemd[1]: Stopped Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:45:49.176 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:48 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 4.103s CPU time. 2024-07-26T08:45:49.176 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:48 smithi044 systemd[1]: Starting Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:45:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:49 smithi044 ceph-mon[32404]: pgmap v714: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T08:45:49.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 podman[107382]: 2024-07-26 08:45:49.079812599 +0000 UTC m=+0.035777011 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:45:49.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 podman[107382]: 2024-07-26 08:45:49.227645684 +0000 UTC m=+0.183610148 container create 5e1d6e46d403ba5fafa003c8c4156ab9cea29217d5c9bcd7764d31eeef023366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724) 2024-07-26T08:45:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:49 smithi116 ceph-mon[37097]: pgmap v714: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T08:45:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:49 smithi171 ceph-mon[37789]: pgmap v714: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T08:45:50.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 podman[107382]: 2024-07-26 08:45:49.773725416 +0000 UTC m=+0.729689890 container init 5e1d6e46d403ba5fafa003c8c4156ab9cea29217d5c9bcd7764d31eeef023366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-07-26T08:45:50.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 podman[107382]: 2024-07-26 08:45:49.785085098 +0000 UTC m=+0.741049522 container start 5e1d6e46d403ba5fafa003c8c4156ab9cea29217d5c9bcd7764d31eeef023366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T08:45:50.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 bash[107382]: 5e1d6e46d403ba5fafa003c8c4156ab9cea29217d5c9bcd7764d31eeef023366 2024-07-26T08:45:50.266 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 ceph-mds[107400]: -- 172.21.15.44:0/2951718890 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c8d165860 con 0x559c91307180 2024-07-26T08:45:50.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 systemd[1]: Started Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:45:50.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:45:49 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: starting mds.j at 2024-07-26T08:45:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: daemon mds.j restarted 2024-07-26T08:45:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T08:45:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:boot 2024-07-26T08:45:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:45:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:45:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 295.552906 secs 2024-07-26T08:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: daemon mds.j restarted 2024-07-26T08:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T08:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:boot 2024-07-26T08:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:45:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:45:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 295.552906 secs 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: daemon mds.j restarted 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:boot 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T08:45:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 295.552906 secs 2024-07-26T08:45:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:51 smithi044 ceph-mon[32404]: pgmap v716: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 op/s 2024-07-26T08:45:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:51 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:resolve 2024-07-26T08:45:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:51 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:51 smithi116 ceph-mon[37097]: pgmap v716: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 op/s 2024-07-26T08:45:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:51 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:resolve 2024-07-26T08:45:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:51 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:51 smithi171 ceph-mon[37789]: pgmap v716: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 op/s 2024-07-26T08:45:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:51 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:resolve 2024-07-26T08:45:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:51 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T08:45:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:52 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:45:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:52 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:reconnect 2024-07-26T08:45:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:52 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:52 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:45:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:52 smithi044 ceph-mon[32404]: pgmap v717: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:45:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:52 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:45:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:52 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:reconnect 2024-07-26T08:45:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:52 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:52 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:45:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:52 smithi116 ceph-mon[37097]: pgmap v717: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:45:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:52 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:45:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:52 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:reconnect 2024-07-26T08:45:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:52 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T08:45:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:52 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:45:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:52 smithi171 ceph-mon[37789]: pgmap v717: 129 pgs: 129 active+clean; 60 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:45:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:53 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:rejoin 2024-07-26T08:45:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:53 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:53 smithi044 ceph-mon[32404]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:45:54.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:45:54.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: *** Caught signal (Aborted) ** 2024-07-26T08:45:54.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: in thread 7f4877200640 thread_name:ms_dispatch 2024-07-26T08:45:54.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f487d43e6f0] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f487d48b94c] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 3: raise() 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 4: abort() 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 5: /lib64/libc.so.6(+0x2871b) [0x7f487d42871b] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 6: /lib64/libc.so.6(+0x37386) [0x7f487d437386] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55585e5006b9] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55585e6be80c] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55585e60ccde] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55585e5ce763] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55585e5cfa1b] 2024-07-26T08:45:54.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55585e5b3752] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55585e501a56] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55585e505637] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55585e505f5e] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55585e4e903a] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f487df9b407] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 18: (DispatchQueue::entry()+0x60f) [0x7f487df9bcf9] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f487e0785a1] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 20: (Thread::entry_wrapper()+0x43) [0x7f487de25cdd] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 21: (Thread::_entry_func(void*)+0xd) [0x7f487de25cf9] 2024-07-26T08:45:54.268 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 22: /lib64/libc.so.6(+0x89c02) [0x7f487d489c02] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f487d50ec40] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.007+0000 7f4877200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: in thread 7f4877200640 thread_name:ms_dispatch 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f487d43e6f0] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f487d48b94c] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 3: raise() 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 4: abort() 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 5: /lib64/libc.so.6(+0x2871b) [0x7f487d42871b] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 6: /lib64/libc.so.6(+0x37386) [0x7f487d437386] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55585e5006b9] 2024-07-26T08:45:54.269 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55585e6be80c] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55585e60ccde] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55585e5ce763] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55585e5cfa1b] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55585e5b3752] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55585e501a56] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55585e505637] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55585e505f5e] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55585e4e903a] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f487df9b407] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 18: (DispatchQueue::entry()+0x60f) [0x7f487df9bcf9] 2024-07-26T08:45:54.270 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f487e0785a1] 2024-07-26T08:45:54.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 20: (Thread::entry_wrapper()+0x43) [0x7f487de25cdd] 2024-07-26T08:45:54.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 21: (Thread::_entry_func(void*)+0xd) [0x7f487de25cf9] 2024-07-26T08:45:54.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 22: /lib64/libc.so.6(+0x89c02) [0x7f487d489c02] 2024-07-26T08:45:54.274 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f487d50ec40] 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 0> 2024-07-26T08:45:54.007+0000 7f4877200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: in thread 7f4877200640 thread_name:ms_dispatch 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f487d43e6f0] 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f487d48b94c] 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 3: raise() 2024-07-26T08:45:54.275 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 4: abort() 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 5: /lib64/libc.so.6(+0x2871b) [0x7f487d42871b] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 6: /lib64/libc.so.6(+0x37386) [0x7f487d437386] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55585e5006b9] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55585e6be80c] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55585e60ccde] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55585e5ce763] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55585e5cfa1b] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55585e5b3752] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55585e501a56] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55585e505637] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55585e505f5e] 2024-07-26T08:45:54.276 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55585e4e903a] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f487df9b407] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 18: (DispatchQueue::entry()+0x60f) [0x7f487df9bcf9] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f487e0785a1] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 20: (Thread::entry_wrapper()+0x43) [0x7f487de25cdd] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 21: (Thread::_entry_func(void*)+0xd) [0x7f487de25cf9] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 22: /lib64/libc.so.6(+0x89c02) [0x7f487d489c02] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f487d50ec40] 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 0> 2024-07-26T08:45:54.007+0000 7f4877200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: in thread 7f4877200640 thread_name:ms_dispatch 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:45:54.277 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f487d43e6f0] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f487d48b94c] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 3: raise() 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 4: abort() 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 5: /lib64/libc.so.6(+0x2871b) [0x7f487d42871b] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 6: /lib64/libc.so.6(+0x37386) [0x7f487d437386] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55585e5006b9] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55585e6be80c] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55585e60ccde] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55585e5ce763] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55585e5cfa1b] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55585e5b3752] 2024-07-26T08:45:54.278 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55585e501a56] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55585e505637] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55585e505f5e] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55585e4e903a] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f487df9b407] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 18: (DispatchQueue::entry()+0x60f) [0x7f487df9bcf9] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f487e0785a1] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 20: (Thread::entry_wrapper()+0x43) [0x7f487de25cdd] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 21: (Thread::_entry_func(void*)+0xd) [0x7f487de25cf9] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 22: /lib64/libc.so.6(+0x89c02) [0x7f487d489c02] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f487d50ec40] 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a[107246]: 2024-07-26T08:45:54.279 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:54 smithi044 podman[107435]: 2024-07-26 08:45:54.154520655 +0000 UTC m=+0.040664582 container died 176f59aca70aa0e37d3e2e0c480b3d44be202e34c10250fb7c168311abf3b4fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T08:45:54.280 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:53 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:rejoin 2024-07-26T08:45:54.280 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:53 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:54.280 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:53 smithi116 ceph-mon[37097]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:45:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:53 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:rejoin 2024-07-26T08:45:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:53 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T08:45:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:53 smithi171 ceph-mon[37789]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T08:45:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:55.187+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:45:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:55.187+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:45:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:45:55.187+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:55.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:55 smithi044 podman[107435]: 2024-07-26 08:45:55.088545739 +0000 UTC m=+0.974689682 container remove 176f59aca70aa0e37d3e2e0c480b3d44be202e34c10250fb7c168311abf3b4fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:45:55.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:55 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:45:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:active 2024-07-26T08:45:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-mon[32404]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:54 smithi044 ceph-mon[32404]: pgmap v718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:45:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:54 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:54 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:active 2024-07-26T08:45:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:54 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:54 smithi116 ceph-mon[37097]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:54 smithi116 ceph-mon[37097]: pgmap v718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:45:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:54 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:45:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:54 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6840/2363748898,v1:172.21.15.44:6841/2363748898] up:active 2024-07-26T08:45:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:54 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T08:45:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:54 smithi171 ceph-mon[37789]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T08:45:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:54 smithi171 ceph-mon[37789]: pgmap v718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 10 op/s 2024-07-26T08:45:55.987 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:55 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T08:45:55.987 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:45:55 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.982s CPU time. 2024-07-26T08:45:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 300.553000 secs 2024-07-26T08:45:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 300.553000 secs 2024-07-26T08:45:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 300.553000 secs 2024-07-26T08:45:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:56 smithi044 ceph-mon[32404]: pgmap v719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:45:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:56 smithi116 ceph-mon[37097]: pgmap v719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:45:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:56 smithi171 ceph-mon[37789]: pgmap v719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 11 op/s 2024-07-26T08:45:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:45:59 smithi044 ceph-mon[32404]: pgmap v720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 307 B/s wr, 14 op/s 2024-07-26T08:45:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:45:59 smithi116 ceph-mon[37097]: pgmap v720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 307 B/s wr, 14 op/s 2024-07-26T08:45:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:45:59 smithi171 ceph-mon[37789]: pgmap v720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 307 B/s wr, 14 op/s 2024-07-26T08:46:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 305.553163 secs 2024-07-26T08:46:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 305.553163 secs 2024-07-26T08:46:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 305.553163 secs 2024-07-26T08:46:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:01 smithi044 ceph-mon[32404]: pgmap v721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 295 B/s wr, 14 op/s 2024-07-26T08:46:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:01 smithi116 ceph-mon[37097]: pgmap v721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 295 B/s wr, 14 op/s 2024-07-26T08:46:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:01 smithi171 ceph-mon[37789]: pgmap v721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 295 B/s wr, 14 op/s 2024-07-26T08:46:03.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:03 smithi044 ceph-mon[32404]: pgmap v722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 13 op/s 2024-07-26T08:46:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:03 smithi116 ceph-mon[37097]: pgmap v722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 13 op/s 2024-07-26T08:46:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:03 smithi171 ceph-mon[37789]: pgmap v722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 13 op/s 2024-07-26T08:46:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:05 smithi044 ceph-mon[32404]: pgmap v723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T08:46:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 310.553321 secs 2024-07-26T08:46:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:05 smithi116 ceph-mon[37097]: pgmap v723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T08:46:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 310.553321 secs 2024-07-26T08:46:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:05 smithi171 ceph-mon[37789]: pgmap v723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T08:46:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 310.553321 secs 2024-07-26T08:46:06.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:46:05 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:46:06.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:46:05 smithi044 systemd[1]: Stopped Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:46:06.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:46:05 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Consumed 3.982s CPU time. 2024-07-26T08:46:06.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:46:05 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-07-26T08:46:06.266 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:46:05 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T08:46:06.267 INFO:journalctl@ceph.mds.a.smithi044.stdout:Jul 26 08:46:05 smithi044 systemd[1]: Failed to start Ceph mds.a for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:46:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:07 smithi044 ceph-mon[32404]: pgmap v724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 12 op/s 2024-07-26T08:46:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:07 smithi116 ceph-mon[37097]: pgmap v724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 12 op/s 2024-07-26T08:46:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:07 smithi171 ceph-mon[37789]: pgmap v724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 255 B/s wr, 12 op/s 2024-07-26T08:46:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:09 smithi044 ceph-mon[32404]: pgmap v725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T08:46:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:09 smithi116 ceph-mon[37097]: pgmap v725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T08:46:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:09 smithi171 ceph-mon[37789]: pgmap v725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T08:46:10.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:10.190+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:46:10.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:10.190+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:46:10.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:10.190+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:10 smithi044 ceph-mon[32404]: Replacing daemon mds.a as rank 0 with standby daemon mds.j 2024-07-26T08:46:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:10 smithi044 ceph-mon[32404]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T08:46:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:10 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:10.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:10 smithi044 ceph-mon[32404]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T08:46:10.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:10 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:10.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 315.553475 secs 2024-07-26T08:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:10 smithi116 ceph-mon[37097]: Replacing daemon mds.a as rank 0 with standby daemon mds.j 2024-07-26T08:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:10 smithi116 ceph-mon[37097]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T08:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:10 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:10 smithi116 ceph-mon[37097]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T08:46:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:10 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 315.553475 secs 2024-07-26T08:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:10 smithi171 ceph-mon[37789]: Replacing daemon mds.a as rank 0 with standby daemon mds.j 2024-07-26T08:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:10 smithi171 ceph-mon[37789]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T08:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:10 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:10 smithi171 ceph-mon[37789]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T08:46:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:10 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 315.553475 secs 2024-07-26T08:46:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:11 smithi044 ceph-mon[32404]: pgmap v727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:46:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:11 smithi116 ceph-mon[37097]: pgmap v727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:46:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:11 smithi171 ceph-mon[37789]: pgmap v727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:46:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:12 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:resolve 2024-07-26T08:46:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:12 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:12 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:resolve 2024-07-26T08:46:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:12 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:12 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:resolve 2024-07-26T08:46:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:12 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:13 smithi044 ceph-mon[32404]: pgmap v728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 op/s 2024-07-26T08:46:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:13 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:46:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:13 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:reconnect 2024-07-26T08:46:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:13 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:13 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:46:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:13 smithi116 ceph-mon[37097]: pgmap v728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 op/s 2024-07-26T08:46:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:13 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:46:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:13 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:reconnect 2024-07-26T08:46:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:13 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:13 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:46:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:13 smithi171 ceph-mon[37789]: pgmap v728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 op/s 2024-07-26T08:46:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:13 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00299995 2024-07-26T08:46:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:13 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:reconnect 2024-07-26T08:46:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:13 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:13 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0119998 2024-07-26T08:46:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:rejoin 2024-07-26T08:46:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-mon[32404]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: *** Caught signal (Aborted) ** 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: in thread 7f4d45400640 thread_name:ms_dispatch 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d4b63e6f0] 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d4b68b94c] 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 3: raise() 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 4: abort() 2024-07-26T08:46:14.768 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d4b62871b] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d4b637386] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559c758056b9] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559c759c380c] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559c75911cde] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559c758d3763] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559c758d4a1b] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559c758b8752] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559c75806a56] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559c7580a637] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559c7580af5e] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559c757ee03a] 2024-07-26T08:46:14.769 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d4c19b407] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d4c19bcf9] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d4c2785a1] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d4c025cdd] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d4c025cf9] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d4b689c02] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d4b70ec40] 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.521+0000 7f4d45400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:14.770 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: in thread 7f4d45400640 thread_name:ms_dispatch 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d4b63e6f0] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d4b68b94c] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 3: raise() 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 4: abort() 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d4b62871b] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d4b637386] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559c758056b9] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559c759c380c] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559c75911cde] 2024-07-26T08:46:14.771 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559c758d3763] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559c758d4a1b] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559c758b8752] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559c75806a56] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559c7580a637] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559c7580af5e] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559c757ee03a] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d4c19b407] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d4c19bcf9] 2024-07-26T08:46:14.772 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d4c2785a1] 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:14 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:rejoin 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:14 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:14 smithi116 ceph-mon[37097]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d4c025cdd] 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d4c025cf9] 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d4b689c02] 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d4b70ec40] 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 0> 2024-07-26T08:46:14.521+0000 7f4d45400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:14.773 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: in thread 7f4d45400640 thread_name:ms_dispatch 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d4b63e6f0] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d4b68b94c] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 3: raise() 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 4: abort() 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d4b62871b] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d4b637386] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559c758056b9] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559c759c380c] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559c75911cde] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559c758d3763] 2024-07-26T08:46:14.774 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559c758d4a1b] 2024-07-26T08:46:14.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559c758b8752] 2024-07-26T08:46:14.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559c75806a56] 2024-07-26T08:46:14.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559c7580a637] 2024-07-26T08:46:14.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559c7580af5e] 2024-07-26T08:46:14.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559c757ee03a] 2024-07-26T08:46:14.775 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d4c19b407] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d4c19bcf9] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d4c2785a1] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d4c025cdd] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d4c025cf9] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d4b689c02] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d4b70ec40] 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 0> 2024-07-26T08:46:14.521+0000 7f4d45400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: in thread 7f4d45400640 thread_name:ms_dispatch 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.776 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d4b63e6f0] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d4b68b94c] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 3: raise() 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 4: abort() 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d4b62871b] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d4b637386] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559c758056b9] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559c759c380c] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559c75911cde] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559c758d3763] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559c758d4a1b] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559c758b8752] 2024-07-26T08:46:14.777 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559c75806a56] 2024-07-26T08:46:14.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559c7580a637] 2024-07-26T08:46:14.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559c7580af5e] 2024-07-26T08:46:14.778 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559c757ee03a] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d4c19b407] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d4c19bcf9] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d4c2785a1] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d4c025cdd] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d4c025cf9] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d4b689c02] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d4b70ec40] 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107396]: 2024-07-26T08:46:14.779 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:14 smithi044 podman[107484]: 2024-07-26 08:46:14.652377685 +0000 UTC m=+0.036835697 container died 5e1d6e46d403ba5fafa003c8c4156ab9cea29217d5c9bcd7764d31eeef023366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T08:46:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:14 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:rejoin 2024-07-26T08:46:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:14 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:14 smithi171 ceph-mon[37789]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:15.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:15 smithi044 podman[107484]: 2024-07-26 08:46:15.420691121 +0000 UTC m=+0.805149162 container remove 5e1d6e46d403ba5fafa003c8c4156ab9cea29217d5c9bcd7764d31eeef023366 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T08:46:15.766 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:15 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:46:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:15 smithi044 ceph-mon[32404]: pgmap v729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 op/s 2024-07-26T08:46:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:15 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:15 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:active 2024-07-26T08:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:15 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 320.553585 secs 2024-07-26T08:46:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:15 smithi116 ceph-mon[37097]: pgmap v729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 op/s 2024-07-26T08:46:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:15 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:15 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:active 2024-07-26T08:46:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:15 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 320.553585 secs 2024-07-26T08:46:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:15 smithi171 ceph-mon[37789]: pgmap v729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 op/s 2024-07-26T08:46:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:15 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:15 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/3903976434,v1:172.21.15.44:6835/3903976434] up:active 2024-07-26T08:46:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:15 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 320.553585 secs 2024-07-26T08:46:16.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:16 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:46:16.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:16 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.866s CPU time. 2024-07-26T08:46:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:17 smithi044 ceph-mon[32404]: pgmap v730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 op/s 2024-07-26T08:46:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:17 smithi116 ceph-mon[37097]: pgmap v730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 op/s 2024-07-26T08:46:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:17 smithi171 ceph-mon[37789]: pgmap v730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 op/s 2024-07-26T08:46:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:19 smithi044 ceph-mon[32404]: pgmap v731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:19 smithi116 ceph-mon[37097]: pgmap v731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:19 smithi171 ceph-mon[37789]: pgmap v731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:46:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:46:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:20 smithi044 ceph-mon[32404]: pgmap v732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 216 B/s wr, 13 op/s 2024-07-26T08:46:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:20 smithi044 ceph-mon[32404]: pgmap v733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 220 B/s wr, 13 op/s 2024-07-26T08:46:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:46:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 325.553730 secs 2024-07-26T08:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:20 smithi116 ceph-mon[37097]: pgmap v732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 216 B/s wr, 13 op/s 2024-07-26T08:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:20 smithi116 ceph-mon[37097]: pgmap v733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 220 B/s wr, 13 op/s 2024-07-26T08:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 325.553730 secs 2024-07-26T08:46:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:46:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:46:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:20 smithi171 ceph-mon[37789]: pgmap v732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 216 B/s wr, 13 op/s 2024-07-26T08:46:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:20 smithi171 ceph-mon[37789]: pgmap v733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 220 B/s wr, 13 op/s 2024-07-26T08:46:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:46:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 325.553730 secs 2024-07-26T08:46:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:22 smithi044 ceph-mon[32404]: pgmap v734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 220 B/s wr, 13 op/s 2024-07-26T08:46:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:22 smithi116 ceph-mon[37097]: pgmap v734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 220 B/s wr, 13 op/s 2024-07-26T08:46:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:22 smithi171 ceph-mon[37789]: pgmap v734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 220 B/s wr, 13 op/s 2024-07-26T08:46:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:24 smithi044 ceph-mon[32404]: pgmap v735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 220 B/s wr, 12 op/s 2024-07-26T08:46:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:24 smithi116 ceph-mon[37097]: pgmap v735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 220 B/s wr, 12 op/s 2024-07-26T08:46:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:24 smithi171 ceph-mon[37789]: pgmap v735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 220 B/s wr, 12 op/s 2024-07-26T08:46:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:25.191+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:46:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:25.191+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:46:25.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:25.191+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 330.553830 secs 2024-07-26T08:46:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 330.553830 secs 2024-07-26T08:46:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 330.553830 secs 2024-07-26T08:46:26.670 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:26 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-07-26T08:46:26.670 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:26 smithi044 systemd[1]: Stopped Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:46:26.670 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:26 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.866s CPU time. 2024-07-26T08:46:26.670 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:26 smithi044 systemd[1]: Starting Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:46:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:26 smithi044 ceph-mon[32404]: pgmap v736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 220 B/s wr, 11 op/s 2024-07-26T08:46:27.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:26 smithi044 podman[107707]: 2024-07-26 08:46:26.574947191 +0000 UTC m=+0.035944243 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:46:27.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:26 smithi044 podman[107707]: 2024-07-26 08:46:26.706095179 +0000 UTC m=+0.167092261 container create 57372c99addb929dab2af2d72bf409e29ef1589b5916895e35a22bd11009dbe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T08:46:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:26 smithi116 ceph-mon[37097]: pgmap v736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 220 B/s wr, 11 op/s 2024-07-26T08:46:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:26 smithi171 ceph-mon[37789]: pgmap v736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 220 B/s wr, 11 op/s 2024-07-26T08:46:27.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:27 smithi044 podman[107707]: 2024-07-26 08:46:27.018147747 +0000 UTC m=+0.479144832 container init 57372c99addb929dab2af2d72bf409e29ef1589b5916895e35a22bd11009dbe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-26T08:46:27.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:27 smithi044 podman[107707]: 2024-07-26 08:46:27.030757827 +0000 UTC m=+0.491754936 container start 57372c99addb929dab2af2d72bf409e29ef1589b5916895e35a22bd11009dbe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-07-26T08:46:27.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:27 smithi044 bash[107707]: 57372c99addb929dab2af2d72bf409e29ef1589b5916895e35a22bd11009dbe3 2024-07-26T08:46:27.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:27 smithi044 systemd[1]: Started Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:46:27.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mds[107725]: -- 172.21.15.44:0/2929490504 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a665edd860 con 0x55a66a07f180 2024-07-26T08:46:27.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: starting mds.j at 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: daemon mds.j restarted 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:boot 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:27 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: daemon mds.j restarted 2024-07-26T08:46:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T08:46:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:boot 2024-07-26T08:46:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:46:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:46:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:46:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:27 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: daemon mds.j restarted 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:boot 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:46:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:27 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:28 smithi044 ceph-mon[32404]: pgmap v738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 8 op/s 2024-07-26T08:46:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:28 smithi116 ceph-mon[37097]: pgmap v738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 8 op/s 2024-07-26T08:46:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:28 smithi171 ceph-mon[37789]: pgmap v738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 8 op/s 2024-07-26T08:46:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:29 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:resolve 2024-07-26T08:46:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:29 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:29 smithi044 ceph-mon[32404]: pgmap v739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8 op/s 2024-07-26T08:46:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:29 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:resolve 2024-07-26T08:46:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:29 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:29 smithi116 ceph-mon[37097]: pgmap v739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8 op/s 2024-07-26T08:46:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:29 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:resolve 2024-07-26T08:46:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:29 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:29 smithi171 ceph-mon[37789]: pgmap v739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8 op/s 2024-07-26T08:46:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:30 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:reconnect 2024-07-26T08:46:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:30 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:30 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999984 2024-07-26T08:46:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:30 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0109998 2024-07-26T08:46:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 335.553962 secs 2024-07-26T08:46:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:30 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:reconnect 2024-07-26T08:46:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:30 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:30 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999984 2024-07-26T08:46:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:30 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0109998 2024-07-26T08:46:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 335.553962 secs 2024-07-26T08:46:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:30 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:reconnect 2024-07-26T08:46:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:30 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:30 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.00999984 2024-07-26T08:46:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:30 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.0109998 2024-07-26T08:46:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 335.553962 secs 2024-07-26T08:46:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:rejoin 2024-07-26T08:46:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-mon[32404]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-mon[32404]: pgmap v740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9 op/s 2024-07-26T08:46:32.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:46:32.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: *** Caught signal (Aborted) ** 2024-07-26T08:46:32.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: in thread 7f330c200640 thread_name:ms_dispatch 2024-07-26T08:46:32.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f331243e6f0] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f331248b94c] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 3: raise() 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 4: abort() 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 5: /lib64/libc.so.6(+0x2871b) [0x7f331242871b] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 6: /lib64/libc.so.6(+0x37386) [0x7f3312437386] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a62e5e06b9] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a62e79e80c] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a62e6eccde] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a62e6ae763] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a62e6afa1b] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a62e693752] 2024-07-26T08:46:32.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a62e5e1a56] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a62e5e5637] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a62e5e5f5e] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a62e5c903a] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3312f9b407] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 18: (DispatchQueue::entry()+0x60f) [0x7f3312f9bcf9] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f33130785a1] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 20: (Thread::entry_wrapper()+0x43) [0x7f3312e25cdd] 2024-07-26T08:46:32.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3312e25cf9] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3312489c02] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f331250ec40] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:31.784+0000 7f330c200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: in thread 7f330c200640 thread_name:ms_dispatch 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f331243e6f0] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f331248b94c] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 3: raise() 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 4: abort() 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 5: /lib64/libc.so.6(+0x2871b) [0x7f331242871b] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 6: /lib64/libc.so.6(+0x37386) [0x7f3312437386] 2024-07-26T08:46:32.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a62e5e06b9] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a62e79e80c] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a62e6eccde] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a62e6ae763] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a62e6afa1b] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a62e693752] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a62e5e1a56] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a62e5e5637] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a62e5e5f5e] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a62e5c903a] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3312f9b407] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 18: (DispatchQueue::entry()+0x60f) [0x7f3312f9bcf9] 2024-07-26T08:46:32.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f33130785a1] 2024-07-26T08:46:32.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 20: (Thread::entry_wrapper()+0x43) [0x7f3312e25cdd] 2024-07-26T08:46:32.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3312e25cf9] 2024-07-26T08:46:32.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3312489c02] 2024-07-26T08:46:32.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f331250ec40] 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 0> 2024-07-26T08:46:31.784+0000 7f330c200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: in thread 7f330c200640 thread_name:ms_dispatch 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f331243e6f0] 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f331248b94c] 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 3: raise() 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 4: abort() 2024-07-26T08:46:32.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 5: /lib64/libc.so.6(+0x2871b) [0x7f331242871b] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 6: /lib64/libc.so.6(+0x37386) [0x7f3312437386] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a62e5e06b9] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a62e79e80c] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a62e6eccde] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a62e6ae763] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a62e6afa1b] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a62e693752] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a62e5e1a56] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a62e5e5637] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a62e5e5f5e] 2024-07-26T08:46:32.024 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a62e5c903a] 2024-07-26T08:46:32.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:31 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:rejoin 2024-07-26T08:46:32.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:31 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:32.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:31 smithi116 ceph-mon[37097]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:32.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:31 smithi116 ceph-mon[37097]: pgmap v740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9 op/s 2024-07-26T08:46:32.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3312f9b407] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 18: (DispatchQueue::entry()+0x60f) [0x7f3312f9bcf9] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f33130785a1] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 20: (Thread::entry_wrapper()+0x43) [0x7f3312e25cdd] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3312e25cf9] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3312489c02] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f331250ec40] 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 0> 2024-07-26T08:46:31.784+0000 7f330c200640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:32.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: in thread 7f330c200640 thread_name:ms_dispatch 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f331243e6f0] 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f331248b94c] 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 3: raise() 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 4: abort() 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 5: /lib64/libc.so.6(+0x2871b) [0x7f331242871b] 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 6: /lib64/libc.so.6(+0x37386) [0x7f3312437386] 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55a62e5e06b9] 2024-07-26T08:46:32.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55a62e79e80c] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55a62e6eccde] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55a62e6ae763] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55a62e6afa1b] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55a62e693752] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55a62e5e1a56] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55a62e5e5637] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55a62e5e5f5e] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55a62e5c903a] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3312f9b407] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 18: (DispatchQueue::entry()+0x60f) [0x7f3312f9bcf9] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f33130785a1] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 20: (Thread::entry_wrapper()+0x43) [0x7f3312e25cdd] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3312e25cf9] 2024-07-26T08:46:32.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3312489c02] 2024-07-26T08:46:32.029 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f331250ec40] 2024-07-26T08:46:32.029 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:32.029 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107721]: 2024-07-26T08:46:32.029 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:31 smithi044 podman[107760]: 2024-07-26 08:46:31.947226177 +0000 UTC m=+0.036690810 container died 57372c99addb929dab2af2d72bf409e29ef1589b5916895e35a22bd11009dbe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-07-26T08:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:31 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:rejoin 2024-07-26T08:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:31 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:31 smithi171 ceph-mon[37789]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:31 smithi171 ceph-mon[37789]: pgmap v740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9 op/s 2024-07-26T08:46:33.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:32 smithi044 podman[107760]: 2024-07-26 08:46:32.622430527 +0000 UTC m=+0.711895171 container remove 57372c99addb929dab2af2d72bf409e29ef1589b5916895e35a22bd11009dbe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T08:46:33.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:32 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:46:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:32 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:32 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:active 2024-07-26T08:46:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:32 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:32 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:32 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:active 2024-07-26T08:46:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:32 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:32 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:32 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/160946197,v1:172.21.15.44:6835/160946197] up:active 2024-07-26T08:46:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:32 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:33.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:33 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:46:33.516 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:33 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.706s CPU time. 2024-07-26T08:46:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:33 smithi044 ceph-mon[32404]: pgmap v741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:46:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:33 smithi116 ceph-mon[37097]: pgmap v741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:46:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:33 smithi171 ceph-mon[37789]: pgmap v741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T08:46:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 340.554106 secs 2024-07-26T08:46:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 340.554106 secs 2024-07-26T08:46:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 340.554106 secs 2024-07-26T08:46:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:36 smithi044 ceph-mon[32404]: pgmap v742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:36 smithi116 ceph-mon[37097]: pgmap v742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:36 smithi171 ceph-mon[37789]: pgmap v742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:38 smithi044 ceph-mon[32404]: pgmap v743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 195 B/s wr, 13 op/s 2024-07-26T08:46:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:38 smithi116 ceph-mon[37097]: pgmap v743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 195 B/s wr, 13 op/s 2024-07-26T08:46:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:38 smithi171 ceph-mon[37789]: pgmap v743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 195 B/s wr, 13 op/s 2024-07-26T08:46:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:40.194+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:46:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:40.194+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:46:40.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:40.194+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:40 smithi044 ceph-mon[32404]: pgmap v744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 345.554268 secs 2024-07-26T08:46:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:40 smithi116 ceph-mon[37097]: pgmap v744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 345.554268 secs 2024-07-26T08:46:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:40 smithi171 ceph-mon[37789]: pgmap v744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 345.554268 secs 2024-07-26T08:46:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:42 smithi044 ceph-mon[32404]: pgmap v745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:42 smithi116 ceph-mon[37097]: pgmap v745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:42 smithi171 ceph-mon[37789]: pgmap v745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:43.681 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-07-26T08:46:43.681 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 systemd[1]: Stopped Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:46:43.682 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.706s CPU time. 2024-07-26T08:46:43.682 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 systemd[1]: Starting Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968... 2024-07-26T08:46:43.955 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 podman[107854]: 2024-07-26 08:46:43.586983155 +0000 UTC m=+0.037969062 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T08:46:43.955 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 podman[107854]: 2024-07-26 08:46:43.724659154 +0000 UTC m=+0.175645089 container create 7335153f76f346ee05a99a276e11b309e35714b8f6fd869770d46d7cea39b390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T08:46:44.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 podman[107854]: 2024-07-26 08:46:43.95363533 +0000 UTC m=+0.404621265 container init 7335153f76f346ee05a99a276e11b309e35714b8f6fd869770d46d7cea39b390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T08:46:44.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:43 smithi044 podman[107854]: 2024-07-26 08:46:43.965147973 +0000 UTC m=+0.416133935 container start 7335153f76f346ee05a99a276e11b309e35714b8f6fd869770d46d7cea39b390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T08:46:44.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:44 smithi044 bash[107854]: 7335153f76f346ee05a99a276e11b309e35714b8f6fd869770d46d7cea39b390 2024-07-26T08:46:44.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:44 smithi044 systemd[1]: Started Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:46:44.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mds[107873]: -- 172.21.15.44:0/4107270635 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc5e4c1860 con 0x55dc62663180 2024-07-26T08:46:44.267 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: starting mds.j at 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: pgmap v746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: daemon mds.j restarted 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:boot 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:46:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:46:45.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:46:45.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:44 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: pgmap v746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:46:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: daemon mds.j restarted 2024-07-26T08:46:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T08:46:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:boot 2024-07-26T08:46:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:46:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:46:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:46:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:44 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: pgmap v746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: daemon mds.j restarted 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:boot 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=h=up:active,4=c=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T08:46:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:44 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T08:46:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 350.554409 secs 2024-07-26T08:46:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:46 smithi044 ceph-mon[32404]: pgmap v748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 9 op/s 2024-07-26T08:46:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:46 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:resolve 2024-07-26T08:46:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:46 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 350.554409 secs 2024-07-26T08:46:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:46 smithi116 ceph-mon[37097]: pgmap v748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 9 op/s 2024-07-26T08:46:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:46 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:resolve 2024-07-26T08:46:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:46 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 350.554409 secs 2024-07-26T08:46:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:46 smithi171 ceph-mon[37789]: pgmap v748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 9 op/s 2024-07-26T08:46:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:46 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:resolve 2024-07-26T08:46:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:46 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T08:46:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:47 smithi044 ceph-mon[32404]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:46:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:47 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:reconnect 2024-07-26T08:46:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:47 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:47 smithi044 ceph-mon[32404]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:46:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:47 smithi116 ceph-mon[37097]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:46:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:47 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:reconnect 2024-07-26T08:46:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:47 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:47 smithi116 ceph-mon[37097]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:46:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:47 smithi171 ceph-mon[37789]: reconnect by client.25535 172.21.15.44:0/3615857024 after 0.00399994 2024-07-26T08:46:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:47 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:reconnect 2024-07-26T08:46:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:47 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T08:46:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:47 smithi171 ceph-mon[37789]: reconnect by client.25387 172.21.15.44:0/1790199663 after 0.0109998 2024-07-26T08:46:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-mon[32404]: pgmap v749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9 op/s 2024-07-26T08:46:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:rejoin 2024-07-26T08:46:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-mon[32404]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-mon[32404]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:49.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T08:46:49.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: *** Caught signal (Aborted) ** 2024-07-26T08:46:49.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: in thread 7f50e4400640 thread_name:ms_dispatch 2024-07-26T08:46:49.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:49.017 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50ea63e6f0] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50ea68b94c] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 3: raise() 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 4: abort() 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50ea62871b] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 6: /lib64/libc.so.6(+0x37386) [0x7f50ea637386] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc4070d6b9] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc408cb80c] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc40819cde] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc407db763] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc407dca1b] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc407c0752] 2024-07-26T08:46:49.018 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc4070ea56] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc40712637] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc40712f5e] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc406f603a] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50eb19b407] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 18: (DispatchQueue::entry()+0x60f) [0x7f50eb19bcf9] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50eb2785a1] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 20: (Thread::entry_wrapper()+0x43) [0x7f50eb025cdd] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50eb025cf9] 2024-07-26T08:46:49.019 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50ea689c02] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50ea70ec40] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:48.759+0000 7f50e4400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: in thread 7f50e4400640 thread_name:ms_dispatch 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50ea63e6f0] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50ea68b94c] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 3: raise() 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 4: abort() 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50ea62871b] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 6: /lib64/libc.so.6(+0x37386) [0x7f50ea637386] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc4070d6b9] 2024-07-26T08:46:49.020 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc408cb80c] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc40819cde] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc407db763] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc407dca1b] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc407c0752] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc4070ea56] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc40712637] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc40712f5e] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc406f603a] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50eb19b407] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 18: (DispatchQueue::entry()+0x60f) [0x7f50eb19bcf9] 2024-07-26T08:46:49.021 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50eb2785a1] 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 20: (Thread::entry_wrapper()+0x43) [0x7f50eb025cdd] 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50eb025cf9] 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50ea689c02] 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50ea70ec40] 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 0> 2024-07-26T08:46:48.759+0000 7f50e4400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: in thread 7f50e4400640 thread_name:ms_dispatch 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50ea63e6f0] 2024-07-26T08:46:49.022 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50ea68b94c] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 3: raise() 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 4: abort() 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50ea62871b] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 6: /lib64/libc.so.6(+0x37386) [0x7f50ea637386] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc4070d6b9] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc408cb80c] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc40819cde] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc407db763] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc407dca1b] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc407c0752] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc4070ea56] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc40712637] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc40712f5e] 2024-07-26T08:46:49.023 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc406f603a] 2024-07-26T08:46:49.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:48 smithi116 ceph-mon[37097]: pgmap v749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9 op/s 2024-07-26T08:46:49.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:48 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:rejoin 2024-07-26T08:46:49.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:48 smithi116 ceph-mon[37097]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:49.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:48 smithi116 ceph-mon[37097]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50eb19b407] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 18: (DispatchQueue::entry()+0x60f) [0x7f50eb19bcf9] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50eb2785a1] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 20: (Thread::entry_wrapper()+0x43) [0x7f50eb025cdd] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50eb025cf9] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50ea689c02] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50ea70ec40] 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:49.025 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 0> 2024-07-26T08:46:48.759+0000 7f50e4400640 -1 *** Caught signal (Aborted) ** 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: in thread 7f50e4400640 thread_name:ms_dispatch 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f50ea63e6f0] 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f50ea68b94c] 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 3: raise() 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 4: abort() 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 5: /lib64/libc.so.6(+0x2871b) [0x7f50ea62871b] 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 6: /lib64/libc.so.6(+0x37386) [0x7f50ea637386] 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc4070d6b9] 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc408cb80c] 2024-07-26T08:46:49.026 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc40819cde] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc407db763] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc407dca1b] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc407c0752] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc4070ea56] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc40712637] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc40712f5e] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc406f603a] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f50eb19b407] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 18: (DispatchQueue::entry()+0x60f) [0x7f50eb19bcf9] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f50eb2785a1] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 20: (Thread::entry_wrapper()+0x43) [0x7f50eb025cdd] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 21: (Thread::_entry_func(void*)+0xd) [0x7f50eb025cf9] 2024-07-26T08:46:49.027 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 22: /lib64/libc.so.6(+0x89c02) [0x7f50ea689c02] 2024-07-26T08:46:49.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f50ea70ec40] 2024-07-26T08:46:49.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T08:46:49.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j[107869]: 2024-07-26T08:46:49.028 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:48 smithi044 podman[107908]: 2024-07-26 08:46:48.900368039 +0000 UTC m=+0.036016466 container died 7335153f76f346ee05a99a276e11b309e35714b8f6fd869770d46d7cea39b390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-07-26T08:46:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:48 smithi171 ceph-mon[37789]: pgmap v749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9 op/s 2024-07-26T08:46:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:48 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:rejoin 2024-07-26T08:46:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:48 smithi171 ceph-mon[37789]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T08:46:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:48 smithi171 ceph-mon[37789]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-26T08:46:50.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:49 smithi044 podman[107908]: 2024-07-26 08:46:49.834073637 +0000 UTC m=+0.969722104 container remove 7335153f76f346ee05a99a276e11b309e35714b8f6fd869770d46d7cea39b390 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-10471e4e-4b28-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T08:46:50.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:49 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T08:46:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:49 smithi044 ceph-mon[32404]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:49 smithi044 ceph-mon[32404]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:active 2024-07-26T08:46:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:49 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:49 smithi044 ceph-mon[32404]: pgmap v750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T08:46:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:49 smithi116 ceph-mon[37097]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:49 smithi116 ceph-mon[37097]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:active 2024-07-26T08:46:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:49 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:49 smithi116 ceph-mon[37097]: pgmap v750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T08:46:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:49 smithi171 ceph-mon[37789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T08:46:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:49 smithi171 ceph-mon[37789]: mds.? [v2:172.21.15.44:6834/2506744542,v1:172.21.15.44:6835/2506744542] up:active 2024-07-26T08:46:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:49 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:46:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:49 smithi171 ceph-mon[37789]: pgmap v750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T08:46:50.742 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:50 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:46:50.743 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:46:50 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.742s CPU time. 2024-07-26T08:46:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 355.554546 secs 2024-07-26T08:46:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 355.554546 secs 2024-07-26T08:46:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 355.554546 secs 2024-07-26T08:46:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:51 smithi044 ceph-mon[32404]: pgmap v751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:46:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:51 smithi116 ceph-mon[37097]: pgmap v751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:46:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:51 smithi171 ceph-mon[37789]: pgmap v751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T08:46:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:54 smithi044 ceph-mon[32404]: pgmap v752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:54 smithi116 ceph-mon[37097]: pgmap v752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:54 smithi171 ceph-mon[37789]: pgmap v752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T08:46:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:55.200+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:46:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:55.200+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:46:55.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:46:55.200+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:55 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 360.554674 secs 2024-07-26T08:46:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:55 smithi044 ceph-mon[32404]: slow request 360.554673 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:46:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:55 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 360.554674 secs 2024-07-26T08:46:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:55 smithi116 ceph-mon[37097]: slow request 360.554673 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:46:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:55 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 360.554674 secs 2024-07-26T08:46:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:55 smithi171 ceph-mon[37789]: slow request 360.554673 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:46:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:56 smithi044 ceph-mon[32404]: pgmap v753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 177 B/s wr, 12 op/s 2024-07-26T08:46:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:56 smithi116 ceph-mon[37097]: pgmap v753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 177 B/s wr, 12 op/s 2024-07-26T08:46:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:56 smithi171 ceph-mon[37789]: pgmap v753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 177 B/s wr, 12 op/s 2024-07-26T08:46:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:58 smithi044 ceph-mon[32404]: pgmap v754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:58 smithi116 ceph-mon[37097]: pgmap v754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:46:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:58 smithi171 ceph-mon[37789]: pgmap v754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T08:47:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:46:59 smithi044 ceph-mon[32404]: pgmap v755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:47:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:46:59 smithi116 ceph-mon[37097]: pgmap v755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:46:59 smithi171 ceph-mon[37789]: pgmap v755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 365.554839 secs 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:47:00 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:47:00 smithi044 systemd[1]: Stopped Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:47:00 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Consumed 3.742s CPU time. 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:47:00 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:47:00 smithi044 systemd[1]: ceph-10471e4e-4b28-11ef-bcb7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T08:47:01.016 INFO:journalctl@ceph.mds.j.smithi044.stdout:Jul 26 08:47:00 smithi044 systemd[1]: Failed to start Ceph mds.j for 10471e4e-4b28-11ef-bcb7-c7b262605968. 2024-07-26T08:47:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 365.554839 secs 2024-07-26T08:47:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 365.554839 secs 2024-07-26T08:47:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:01 smithi171 ceph-mon[37789]: pgmap v756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:47:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:01 smithi044 ceph-mon[32404]: pgmap v756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:47:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:01 smithi116 ceph-mon[37097]: pgmap v756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-26T08:47:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:04 smithi044 ceph-mon[32404]: pgmap v757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:47:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:04 smithi116 ceph-mon[37097]: pgmap v757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:47:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:04 smithi171 ceph-mon[37789]: pgmap v757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T08:47:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:06 smithi044 ceph-mon[32404]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:47:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 370.554975 secs 2024-07-26T08:47:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:06 smithi044 ceph-mon[32404]: pgmap v758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 8 op/s 2024-07-26T08:47:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:06 smithi116 ceph-mon[37097]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:47:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 370.554975 secs 2024-07-26T08:47:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:06 smithi116 ceph-mon[37097]: pgmap v758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 8 op/s 2024-07-26T08:47:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:06 smithi171 ceph-mon[37789]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T08:47:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 370.554975 secs 2024-07-26T08:47:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:06 smithi171 ceph-mon[37789]: pgmap v758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 8 op/s 2024-07-26T08:47:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:08 smithi044 ceph-mon[32404]: pgmap v759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:08 smithi116 ceph-mon[37097]: pgmap v759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:08 smithi171 ceph-mon[37789]: pgmap v759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:10.203+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:47:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:10.203+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:47:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:10.203+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:10 smithi044 ceph-mon[32404]: pgmap v760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 375.555119 secs 2024-07-26T08:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:10 smithi116 ceph-mon[37097]: pgmap v760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 375.555119 secs 2024-07-26T08:47:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:10 smithi171 ceph-mon[37789]: pgmap v760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 375.555119 secs 2024-07-26T08:47:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:12 smithi044 ceph-mon[32404]: pgmap v761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:12 smithi116 ceph-mon[37097]: pgmap v761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:12 smithi171 ceph-mon[37789]: pgmap v761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:47:14.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:14.727+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40005850 con 0x7fcf6c1057b0 2024-07-26T08:47:14.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:14.728+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40005850 con 0x7fcf6c1057b0 2024-07-26T08:47:14.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:14.730+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c005850 con 0x55c94889a070 2024-07-26T08:47:14.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:14.731+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c005850 con 0x55c94889a070 2024-07-26T08:47:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:14 smithi044 ceph-mon[32404]: pgmap v762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:14 smithi116 ceph-mon[37097]: pgmap v762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:14 smithi171 ceph-mon[37789]: pgmap v762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:47:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 380.555278 secs 2024-07-26T08:47:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:15 smithi044 ceph-mon[32404]: pgmap v763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 380.555278 secs 2024-07-26T08:47:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:15 smithi116 ceph-mon[37097]: pgmap v763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 380.555278 secs 2024-07-26T08:47:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:15 smithi171 ceph-mon[37789]: pgmap v763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:18 smithi044 ceph-mon[32404]: pgmap v764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:18 smithi116 ceph-mon[37097]: pgmap v764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:18 smithi171 ceph-mon[37789]: pgmap v764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:20 smithi044 ceph-mon[32404]: pgmap v765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:47:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 385.555365 secs 2024-07-26T08:47:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:20 smithi116 ceph-mon[37097]: pgmap v765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:47:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 385.555365 secs 2024-07-26T08:47:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:20 smithi171 ceph-mon[37789]: pgmap v765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:47:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 385.555365 secs 2024-07-26T08:47:22.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:22.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:22.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:47:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:47:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:22 smithi044 ceph-mon[32404]: pgmap v766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:47:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:22 smithi044 ceph-mon[32404]: pgmap v767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:47:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:22 smithi116 ceph-mon[37097]: pgmap v766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:47:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:22 smithi116 ceph-mon[37097]: pgmap v767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:47:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:22 smithi171 ceph-mon[37789]: pgmap v766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:47:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:22 smithi171 ceph-mon[37789]: pgmap v767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:47:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:23 smithi116 ceph-mon[37097]: pgmap v768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:47:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:23 smithi171 ceph-mon[37789]: pgmap v768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:47:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:23 smithi044 ceph-mon[32404]: pgmap v768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:47:25.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:25.211+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:47:25.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:25.211+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:47:25.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:25.211+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:25 smithi044 ceph-mon[32404]: pgmap v769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 390.555505 secs 2024-07-26T08:47:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:25 smithi116 ceph-mon[37097]: pgmap v769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 390.555505 secs 2024-07-26T08:47:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:25 smithi171 ceph-mon[37789]: pgmap v769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 390.555505 secs 2024-07-26T08:47:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:27 smithi044 ceph-mon[32404]: pgmap v770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:27 smithi116 ceph-mon[37097]: pgmap v770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:27 smithi171 ceph-mon[37789]: pgmap v770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:29 smithi044 ceph-mon[32404]: pgmap v771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:29 smithi116 ceph-mon[37097]: pgmap v771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:29 smithi171 ceph-mon[37789]: pgmap v771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:47:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 395.555669 secs 2024-07-26T08:47:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 395.555669 secs 2024-07-26T08:47:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 395.555669 secs 2024-07-26T08:47:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:31 smithi044 ceph-mon[32404]: pgmap v772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:47:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:31 smithi116 ceph-mon[37097]: pgmap v772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:47:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:31 smithi171 ceph-mon[37789]: pgmap v772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:47:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:33 smithi044 ceph-mon[32404]: pgmap v773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:33 smithi116 ceph-mon[37097]: pgmap v773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:33 smithi171 ceph-mon[37789]: pgmap v773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:35 smithi044 ceph-mon[32404]: pgmap v774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 400.555776 secs 2024-07-26T08:47:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:35 smithi116 ceph-mon[37097]: pgmap v774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 400.555776 secs 2024-07-26T08:47:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:35 smithi171 ceph-mon[37789]: pgmap v774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 400.555776 secs 2024-07-26T08:47:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:37 smithi044 ceph-mon[32404]: pgmap v775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:37 smithi116 ceph-mon[37097]: pgmap v775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:37 smithi171 ceph-mon[37789]: pgmap v775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:40.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:40.214+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:47:40.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:40.214+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:47:40.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:40.214+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:40 smithi044 ceph-mon[32404]: pgmap v776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:40 smithi116 ceph-mon[37097]: pgmap v776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:40 smithi171 ceph-mon[37789]: pgmap v776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 405.555917 secs 2024-07-26T08:47:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 405.555917 secs 2024-07-26T08:47:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 405.555917 secs 2024-07-26T08:47:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:42 smithi116 ceph-mon[37097]: pgmap v777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:42 smithi171 ceph-mon[37789]: pgmap v777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:42 smithi044 ceph-mon[32404]: pgmap v777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:47:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:44 smithi171 ceph-mon[37789]: pgmap v778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:44 smithi044 ceph-mon[32404]: pgmap v778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:44 smithi116 ceph-mon[37097]: pgmap v778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:46 smithi171 ceph-mon[37789]: pgmap v779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 410.556053 secs 2024-07-26T08:47:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:46 smithi044 ceph-mon[32404]: pgmap v779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 410.556053 secs 2024-07-26T08:47:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:46 smithi116 ceph-mon[37097]: pgmap v779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 410.556053 secs 2024-07-26T08:47:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:48 smithi171 ceph-mon[37789]: pgmap v780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:48 smithi044 ceph-mon[32404]: pgmap v780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:48 smithi116 ceph-mon[37097]: pgmap v780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:50 smithi171 ceph-mon[37789]: pgmap v781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:50 smithi044 ceph-mon[32404]: pgmap v781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:50 smithi116 ceph-mon[37097]: pgmap v781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 415.556194 secs 2024-07-26T08:47:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 415.556194 secs 2024-07-26T08:47:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 415.556194 secs 2024-07-26T08:47:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:52 smithi171 ceph-mon[37789]: pgmap v782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:47:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:52 smithi044 ceph-mon[32404]: pgmap v782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:47:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:52 smithi116 ceph-mon[37097]: pgmap v782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:47:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:54 smithi171 ceph-mon[37789]: pgmap v783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:54 smithi044 ceph-mon[32404]: pgmap v783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:54.638 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:54 smithi116 ceph-mon[37097]: pgmap v783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:55.222+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:47:55.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:55.222+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:47:55.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:47:55.222+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:56 smithi171 ceph-mon[37789]: pgmap v784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 420.556357 secs 2024-07-26T08:47:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:56 smithi044 ceph-mon[32404]: pgmap v784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 420.556357 secs 2024-07-26T08:47:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:56 smithi116 ceph-mon[37097]: pgmap v784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:47:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 420.556357 secs 2024-07-26T08:47:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:47:58 smithi171 ceph-mon[37789]: pgmap v785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:47:58 smithi044 ceph-mon[32404]: pgmap v785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:47:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:47:58 smithi116 ceph-mon[37097]: pgmap v785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:00 smithi171 ceph-mon[37789]: pgmap v786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:00 smithi044 ceph-mon[32404]: pgmap v786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:00 smithi116 ceph-mon[37097]: pgmap v786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 425.556481 secs 2024-07-26T08:48:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 425.556481 secs 2024-07-26T08:48:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 425.556481 secs 2024-07-26T08:48:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:02 smithi044 ceph-mon[32404]: pgmap v787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:02 smithi116 ceph-mon[37097]: pgmap v787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:02 smithi171 ceph-mon[37789]: pgmap v787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:04 smithi044 ceph-mon[32404]: pgmap v788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:04 smithi116 ceph-mon[37097]: pgmap v788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:04 smithi171 ceph-mon[37789]: pgmap v788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:06 smithi044 ceph-mon[32404]: pgmap v789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:48:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 430.556602 secs 2024-07-26T08:48:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:06 smithi116 ceph-mon[37097]: pgmap v789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:48:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 430.556602 secs 2024-07-26T08:48:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:06 smithi171 ceph-mon[37789]: pgmap v789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:48:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 430.556602 secs 2024-07-26T08:48:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:08 smithi044 ceph-mon[32404]: pgmap v790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:08 smithi116 ceph-mon[37097]: pgmap v790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:08 smithi171 ceph-mon[37789]: pgmap v790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:10.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:10.225+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:48:10.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:10.225+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:48:10.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:10.225+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:10.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:10 smithi171 ceph-mon[37789]: pgmap v791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:10 smithi044 ceph-mon[32404]: pgmap v791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:10 smithi116 ceph-mon[37097]: pgmap v791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 435.556742 secs 2024-07-26T08:48:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 435.556742 secs 2024-07-26T08:48:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 435.556742 secs 2024-07-26T08:48:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:12 smithi044 ceph-mon[32404]: pgmap v792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:48:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:12 smithi116 ceph-mon[37097]: pgmap v792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:48:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:12 smithi171 ceph-mon[37789]: pgmap v792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:48:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:14 smithi044 ceph-mon[32404]: pgmap v793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:14 smithi116 ceph-mon[37097]: pgmap v793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:14 smithi171 ceph-mon[37789]: pgmap v793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:48:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:16 smithi044 ceph-mon[32404]: pgmap v794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 440.556840 secs 2024-07-26T08:48:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:16 smithi116 ceph-mon[37097]: pgmap v794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 440.556840 secs 2024-07-26T08:48:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:16 smithi171 ceph-mon[37789]: pgmap v794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:48:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 440.556840 secs 2024-07-26T08:48:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:18 smithi044 ceph-mon[32404]: pgmap v795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:18 smithi116 ceph-mon[37097]: pgmap v795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:18 smithi171 ceph-mon[37789]: pgmap v795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:20 smithi044 ceph-mon[32404]: pgmap v796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:20 smithi116 ceph-mon[37097]: pgmap v796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:20 smithi171 ceph-mon[37789]: pgmap v796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 445.556929 secs 2024-07-26T08:48:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 445.556929 secs 2024-07-26T08:48:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 445.556929 secs 2024-07-26T08:48:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:22 smithi044 ceph-mon[32404]: pgmap v797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:22 smithi116 ceph-mon[37097]: pgmap v797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:22 smithi171 ceph-mon[37789]: pgmap v797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:23 smithi044 ceph-mon[32404]: pgmap v798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:48:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:23 smithi116 ceph-mon[37097]: pgmap v798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:48:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:23 smithi171 ceph-mon[37789]: pgmap v798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:48:25.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:25.232+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:48:25.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:25.232+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:48:25.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:25.232+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:25 smithi044 ceph-mon[32404]: pgmap v799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:25 smithi116 ceph-mon[37097]: pgmap v799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:25 smithi171 ceph-mon[37789]: pgmap v799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 450.557092 secs 2024-07-26T08:48:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 450.557092 secs 2024-07-26T08:48:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 450.557092 secs 2024-07-26T08:48:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:27 smithi044 ceph-mon[32404]: pgmap v800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:48:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:27 smithi116 ceph-mon[37097]: pgmap v800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:48:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:27 smithi171 ceph-mon[37789]: pgmap v800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:48:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:29 smithi044 ceph-mon[32404]: pgmap v801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:29 smithi116 ceph-mon[37097]: pgmap v801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:29 smithi171 ceph-mon[37789]: pgmap v801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:31 smithi044 ceph-mon[32404]: pgmap v802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:48:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 455.557177 secs 2024-07-26T08:48:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:31 smithi116 ceph-mon[37097]: pgmap v802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:48:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 455.557177 secs 2024-07-26T08:48:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:31 smithi171 ceph-mon[37789]: pgmap v802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:48:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 455.557177 secs 2024-07-26T08:48:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:33 smithi044 ceph-mon[32404]: pgmap v803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:33 smithi116 ceph-mon[37097]: pgmap v803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:33 smithi171 ceph-mon[37789]: pgmap v803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:48:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:35 smithi044 ceph-mon[32404]: pgmap v804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:35 smithi116 ceph-mon[37097]: pgmap v804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:35 smithi171 ceph-mon[37789]: pgmap v804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 460.557320 secs 2024-07-26T08:48:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 460.557320 secs 2024-07-26T08:48:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 460.557320 secs 2024-07-26T08:48:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:37 smithi044 ceph-mon[32404]: pgmap v805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:37 smithi116 ceph-mon[37097]: pgmap v805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:37 smithi171 ceph-mon[37789]: pgmap v805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:39.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:39 smithi044 ceph-mon[32404]: pgmap v806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:39 smithi116 ceph-mon[37097]: pgmap v806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:39 smithi171 ceph-mon[37789]: pgmap v806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:40.235+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:48:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:40.235+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:48:40.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:40.235+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:41 smithi044 ceph-mon[32404]: pgmap v807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 465.557458 secs 2024-07-26T08:48:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:41 smithi116 ceph-mon[37097]: pgmap v807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 465.557458 secs 2024-07-26T08:48:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:41 smithi171 ceph-mon[37789]: pgmap v807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 465.557458 secs 2024-07-26T08:48:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:43 smithi044 ceph-mon[32404]: pgmap v808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:43 smithi116 ceph-mon[37097]: pgmap v808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:43 smithi171 ceph-mon[37789]: pgmap v808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:45 smithi044 ceph-mon[32404]: pgmap v809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:45 smithi116 ceph-mon[37097]: pgmap v809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:45 smithi171 ceph-mon[37789]: pgmap v809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 470.557657 secs 2024-07-26T08:48:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 470.557657 secs 2024-07-26T08:48:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 470.557657 secs 2024-07-26T08:48:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:47 smithi171 ceph-mon[37789]: pgmap v810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:47 smithi044 ceph-mon[32404]: pgmap v810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:47 smithi116 ceph-mon[37097]: pgmap v810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:49 smithi171 ceph-mon[37789]: pgmap v811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:49 smithi044 ceph-mon[32404]: pgmap v811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:49 smithi116 ceph-mon[37097]: pgmap v811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:51 smithi171 ceph-mon[37789]: pgmap v812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 475.557811 secs 2024-07-26T08:48:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:51 smithi044 ceph-mon[32404]: pgmap v812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 475.557811 secs 2024-07-26T08:48:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:51 smithi116 ceph-mon[37097]: pgmap v812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 475.557811 secs 2024-07-26T08:48:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:53 smithi171 ceph-mon[37789]: pgmap v813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:53 smithi044 ceph-mon[32404]: pgmap v813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:53 smithi116 ceph-mon[37097]: pgmap v813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:55.242+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:55.242+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:48:55.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:48:55.242+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:55 smithi044 ceph-mon[32404]: pgmap v814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:55 smithi171 ceph-mon[37789]: pgmap v814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:55 smithi116 ceph-mon[37097]: pgmap v814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:48:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 480.557906 secs 2024-07-26T08:48:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 480.557906 secs 2024-07-26T08:48:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 480.557906 secs 2024-07-26T08:48:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:57 smithi171 ceph-mon[37789]: pgmap v815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:57 smithi044 ceph-mon[32404]: pgmap v815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:57 smithi116 ceph-mon[37097]: pgmap v815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:48:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:48:59 smithi171 ceph-mon[37789]: pgmap v816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:48:59 smithi044 ceph-mon[32404]: pgmap v816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:48:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:48:59 smithi116 ceph-mon[37097]: pgmap v816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:01 smithi171 ceph-mon[37789]: pgmap v817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 485.558036 secs 2024-07-26T08:49:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:01 smithi044 ceph-mon[32404]: pgmap v817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 485.558036 secs 2024-07-26T08:49:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:01 smithi116 ceph-mon[37097]: pgmap v817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 485.558036 secs 2024-07-26T08:49:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:03 smithi171 ceph-mon[37789]: pgmap v818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:03 smithi044 ceph-mon[32404]: pgmap v818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:03 smithi116 ceph-mon[37097]: pgmap v818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:05 smithi044 ceph-mon[32404]: pgmap v819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:05 smithi116 ceph-mon[37097]: pgmap v819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:05 smithi171 ceph-mon[37789]: pgmap v819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:06.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 490.558195 secs 2024-07-26T08:49:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 490.558195 secs 2024-07-26T08:49:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 490.558195 secs 2024-07-26T08:49:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:07 smithi044 ceph-mon[32404]: pgmap v820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:07 smithi116 ceph-mon[37097]: pgmap v820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:07 smithi171 ceph-mon[37789]: pgmap v820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:09 smithi044 ceph-mon[32404]: pgmap v821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:09 smithi116 ceph-mon[37097]: pgmap v821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:09 smithi171 ceph-mon[37789]: pgmap v821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:10.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:10.245+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:49:10.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:10.245+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:49:10.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:10.245+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 495.558354 secs 2024-07-26T08:49:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 495.558354 secs 2024-07-26T08:49:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 495.558354 secs 2024-07-26T08:49:11.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:11 smithi171 ceph-mon[37789]: pgmap v822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:11 smithi044 ceph-mon[32404]: pgmap v822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:11 smithi116 ceph-mon[37097]: pgmap v822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:13 smithi044 ceph-mon[32404]: pgmap v823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:13 smithi116 ceph-mon[37097]: pgmap v823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:13 smithi171 ceph-mon[37789]: pgmap v823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:15 smithi044 ceph-mon[32404]: pgmap v824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 500.558518 secs 2024-07-26T08:49:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:15 smithi116 ceph-mon[37097]: pgmap v824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 500.558518 secs 2024-07-26T08:49:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:15 smithi171 ceph-mon[37789]: pgmap v824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 500.558518 secs 2024-07-26T08:49:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:17 smithi044 ceph-mon[32404]: pgmap v825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:17 smithi116 ceph-mon[37097]: pgmap v825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:17 smithi171 ceph-mon[37789]: pgmap v825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:19 smithi044 ceph-mon[32404]: pgmap v826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:19 smithi116 ceph-mon[37097]: pgmap v826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:19 smithi171 ceph-mon[37789]: pgmap v826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:49:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 505.558681 secs 2024-07-26T08:49:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 505.558681 secs 2024-07-26T08:49:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 505.558681 secs 2024-07-26T08:49:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:21 smithi044 ceph-mon[32404]: pgmap v827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:21 smithi116 ceph-mon[37097]: pgmap v827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:21 smithi171 ceph-mon[37789]: pgmap v827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:23 smithi044 ceph-mon[32404]: pgmap v828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:23 smithi116 ceph-mon[37097]: pgmap v828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:23 smithi171 ceph-mon[37789]: pgmap v828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:24 smithi044 ceph-mon[32404]: pgmap v829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:49:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:24 smithi044 ceph-mon[32404]: pgmap v830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:49:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:24 smithi116 ceph-mon[37097]: pgmap v829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:49:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:24 smithi116 ceph-mon[37097]: pgmap v830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:49:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:24 smithi171 ceph-mon[37789]: pgmap v829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:49:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:24 smithi171 ceph-mon[37789]: pgmap v830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:49:25.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:25.246+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:49:25.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:25.247+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:49:25.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:25.247+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 510.558842 secs 2024-07-26T08:49:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 510.558842 secs 2024-07-26T08:49:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 510.558842 secs 2024-07-26T08:49:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:26 smithi044 ceph-mon[32404]: pgmap v831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:26 smithi116 ceph-mon[37097]: pgmap v831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:26 smithi171 ceph-mon[37789]: pgmap v831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:28 smithi044 ceph-mon[32404]: pgmap v832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:28 smithi116 ceph-mon[37097]: pgmap v832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:28 smithi171 ceph-mon[37789]: pgmap v832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:29.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:29.750+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40005b10 con 0x7fcf6c1057b0 2024-07-26T08:49:29.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:29.750+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40005b10 con 0x7fcf6c1057b0 2024-07-26T08:49:29.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:29.753+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c005b50 con 0x55c94889a070 2024-07-26T08:49:29.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:29.754+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c005b50 con 0x55c94889a070 2024-07-26T08:49:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:30 smithi044 ceph-mon[32404]: pgmap v833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 515.558998 secs 2024-07-26T08:49:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:30 smithi116 ceph-mon[37097]: pgmap v833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 515.558998 secs 2024-07-26T08:49:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:30 smithi171 ceph-mon[37789]: pgmap v833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:49:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 515.558998 secs 2024-07-26T08:49:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:32 smithi116 ceph-mon[37097]: pgmap v834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:32 smithi171 ceph-mon[37789]: pgmap v834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:32 smithi044 ceph-mon[32404]: pgmap v834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:49:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:34 smithi171 ceph-mon[37789]: pgmap v835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:49:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:34 smithi044 ceph-mon[32404]: pgmap v835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:49:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:34 smithi116 ceph-mon[37097]: pgmap v835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:49:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 520.559100 secs 2024-07-26T08:49:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 520.559100 secs 2024-07-26T08:49:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 520.559100 secs 2024-07-26T08:49:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:36 smithi171 ceph-mon[37789]: pgmap v836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:36.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:36 smithi116 ceph-mon[37097]: pgmap v836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:36 smithi044 ceph-mon[32404]: pgmap v836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:38.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:38 smithi171 ceph-mon[37789]: pgmap v837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:38 smithi044 ceph-mon[32404]: pgmap v837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:38 smithi116 ceph-mon[37097]: pgmap v837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:40.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:40.250+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:49:40.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:40.251+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:49:40.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:40.251+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:40 smithi171 ceph-mon[37789]: pgmap v838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 525.559258 secs 2024-07-26T08:49:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:40 smithi044 ceph-mon[32404]: pgmap v838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 525.559258 secs 2024-07-26T08:49:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:40 smithi116 ceph-mon[37097]: pgmap v838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 525.559258 secs 2024-07-26T08:49:42.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:42 smithi044 ceph-mon[32404]: pgmap v839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:42 smithi171 ceph-mon[37789]: pgmap v839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:42 smithi116 ceph-mon[37097]: pgmap v839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:44 smithi171 ceph-mon[37789]: pgmap v840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:49:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:44 smithi044 ceph-mon[32404]: pgmap v840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:49:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:44 smithi116 ceph-mon[37097]: pgmap v840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:49:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 530.559410 secs 2024-07-26T08:49:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 530.559410 secs 2024-07-26T08:49:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 530.559410 secs 2024-07-26T08:49:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:46 smithi171 ceph-mon[37789]: pgmap v841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:46 smithi044 ceph-mon[32404]: pgmap v841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:46 smithi116 ceph-mon[37097]: pgmap v841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:48 smithi171 ceph-mon[37789]: pgmap v842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:48 smithi044 ceph-mon[32404]: pgmap v842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:48 smithi116 ceph-mon[37097]: pgmap v842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:50 smithi044 ceph-mon[32404]: pgmap v843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 535.559573 secs 2024-07-26T08:49:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:50 smithi116 ceph-mon[37097]: pgmap v843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 535.559573 secs 2024-07-26T08:49:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:50 smithi171 ceph-mon[37789]: pgmap v843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:49:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 535.559573 secs 2024-07-26T08:49:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:52 smithi044 ceph-mon[32404]: pgmap v844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:52 smithi116 ceph-mon[37097]: pgmap v844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:52 smithi171 ceph-mon[37789]: pgmap v844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:54 smithi044 ceph-mon[32404]: pgmap v845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:49:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:54 smithi116 ceph-mon[37097]: pgmap v845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:49:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:54 smithi171 ceph-mon[37789]: pgmap v845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:49:55.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:55.253+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:49:55.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:55.253+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:49:55.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:49:55.253+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 540.559718 secs 2024-07-26T08:49:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 540.559718 secs 2024-07-26T08:49:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 540.559718 secs 2024-07-26T08:49:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:56 smithi044 ceph-mon[32404]: pgmap v846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:56 smithi116 ceph-mon[37097]: pgmap v846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:56 smithi171 ceph-mon[37789]: pgmap v846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:49:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:49:58 smithi044 ceph-mon[32404]: pgmap v847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:49:58 smithi116 ceph-mon[37097]: pgmap v847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:49:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:49:58 smithi171 ceph-mon[37789]: pgmap v847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:00.628 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: pgmap v848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:00.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:50:00.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-07-26T08:50:00.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: daemon mds.b on smithi116 is in error state 2024-07-26T08:50:00.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: daemon mds.f on smithi171 is in error state 2024-07-26T08:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: pgmap v848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-07-26T08:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: daemon mds.b on smithi116 is in error state 2024-07-26T08:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: daemon mds.f on smithi171 is in error state 2024-07-26T08:50:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:50:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: mds.g(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T08:50:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 545.559804 secs 2024-07-26T08:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: pgmap v848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T08:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-07-26T08:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: daemon mds.b on smithi116 is in error state 2024-07-26T08:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: daemon mds.f on smithi171 is in error state 2024-07-26T08:50:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:50:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: mds.g(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T08:50:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 545.559804 secs 2024-07-26T08:50:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T08:50:01.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: mds.g(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T08:50:01.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 545.559804 secs 2024-07-26T08:50:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:02 smithi044 ceph-mon[32404]: pgmap v849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:02 smithi116 ceph-mon[37097]: pgmap v849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:02 smithi171 ceph-mon[37789]: pgmap v849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:04 smithi044 ceph-mon[32404]: pgmap v850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:04 smithi116 ceph-mon[37097]: pgmap v850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:04 smithi171 ceph-mon[37789]: pgmap v850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 550.559952 secs 2024-07-26T08:50:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 550.559952 secs 2024-07-26T08:50:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 550.559952 secs 2024-07-26T08:50:06.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:06 smithi116 ceph-mon[37097]: pgmap v851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:06 smithi044 ceph-mon[32404]: pgmap v851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:06 smithi171 ceph-mon[37789]: pgmap v851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:08 smithi044 ceph-mon[32404]: pgmap v852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:08 smithi116 ceph-mon[37097]: pgmap v852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:08 smithi171 ceph-mon[37789]: pgmap v852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:10.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:10.255+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:50:10.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:10.255+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:50:10.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:10.255+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:10 smithi044 ceph-mon[32404]: pgmap v853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 555.560094 secs 2024-07-26T08:50:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:10 smithi116 ceph-mon[37097]: pgmap v853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 555.560094 secs 2024-07-26T08:50:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:10 smithi171 ceph-mon[37789]: pgmap v853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 555.560094 secs 2024-07-26T08:50:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:12 smithi044 ceph-mon[32404]: pgmap v854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:12 smithi116 ceph-mon[37097]: pgmap v854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:12 smithi171 ceph-mon[37789]: pgmap v854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:14.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:14.757+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40005df0 con 0x7fcf6c1057b0 2024-07-26T08:50:14.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:14.758+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40005df0 con 0x7fcf6c1057b0 2024-07-26T08:50:14.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:14.760+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c005e50 con 0x55c94889a070 2024-07-26T08:50:14.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:14.761+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c005e50 con 0x55c94889a070 2024-07-26T08:50:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:14 smithi044 ceph-mon[32404]: pgmap v855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:14 smithi116 ceph-mon[37097]: pgmap v855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:14 smithi171 ceph-mon[37789]: pgmap v855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 560.560197 secs 2024-07-26T08:50:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 560.560197 secs 2024-07-26T08:50:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 560.560197 secs 2024-07-26T08:50:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:16 smithi044 ceph-mon[32404]: pgmap v856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:16 smithi116 ceph-mon[37097]: pgmap v856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:16 smithi171 ceph-mon[37789]: pgmap v856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:18 smithi044 ceph-mon[32404]: pgmap v857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:18 smithi116 ceph-mon[37097]: pgmap v857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:18 smithi171 ceph-mon[37789]: pgmap v857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:20 smithi044 ceph-mon[32404]: pgmap v858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 565.560335 secs 2024-07-26T08:50:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:20 smithi116 ceph-mon[37097]: pgmap v858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 565.560335 secs 2024-07-26T08:50:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:20 smithi171 ceph-mon[37789]: pgmap v858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 565.560335 secs 2024-07-26T08:50:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:22 smithi044 ceph-mon[32404]: pgmap v859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:22 smithi116 ceph-mon[37097]: pgmap v859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:22 smithi171 ceph-mon[37789]: pgmap v859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:24 smithi044 ceph-mon[32404]: pgmap v860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:24 smithi116 ceph-mon[37097]: pgmap v860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:24 smithi171 ceph-mon[37789]: pgmap v860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:50:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:25.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:25.256+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:50:25.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:25.256+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:50:25.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:25.256+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:25 smithi044 ceph-mon[32404]: pgmap v861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:50:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:25 smithi044 ceph-mon[32404]: pgmap v862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:50:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 570.560477 secs 2024-07-26T08:50:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:25 smithi116 ceph-mon[37097]: pgmap v861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:50:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:25 smithi116 ceph-mon[37097]: pgmap v862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:50:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 570.560477 secs 2024-07-26T08:50:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:25 smithi171 ceph-mon[37789]: pgmap v861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:50:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:25 smithi171 ceph-mon[37789]: pgmap v862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:50:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 570.560477 secs 2024-07-26T08:50:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:27 smithi044 ceph-mon[32404]: pgmap v863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:50:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:27 smithi116 ceph-mon[37097]: pgmap v863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:50:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:27 smithi171 ceph-mon[37789]: pgmap v863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:50:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:29 smithi044 ceph-mon[32404]: pgmap v864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:29 smithi116 ceph-mon[37097]: pgmap v864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:29 smithi171 ceph-mon[37789]: pgmap v864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 575.560658 secs 2024-07-26T08:50:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 575.560658 secs 2024-07-26T08:50:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 575.560658 secs 2024-07-26T08:50:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:31 smithi044 ceph-mon[32404]: pgmap v865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:31 smithi116 ceph-mon[37097]: pgmap v865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:31 smithi171 ceph-mon[37789]: pgmap v865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:50:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:32 smithi171 ceph-mon[37789]: pgmap v866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:50:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:32 smithi044 ceph-mon[32404]: pgmap v866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:50:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:32 smithi116 ceph-mon[37097]: pgmap v866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:50:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:34 smithi171 ceph-mon[37789]: pgmap v867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:34 smithi044 ceph-mon[32404]: pgmap v867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:34 smithi116 ceph-mon[37097]: pgmap v867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:50:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 580.560814 secs 2024-07-26T08:50:36.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 580.560814 secs 2024-07-26T08:50:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 580.560814 secs 2024-07-26T08:50:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:36 smithi171 ceph-mon[37789]: pgmap v868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:36 smithi044 ceph-mon[32404]: pgmap v868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:36 smithi116 ceph-mon[37097]: pgmap v868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:38 smithi171 ceph-mon[37789]: pgmap v869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:38 smithi044 ceph-mon[32404]: pgmap v869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:38 smithi116 ceph-mon[37097]: pgmap v869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:40.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:40.259+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:50:40.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:40.259+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:50:40.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:40.259+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 585.560898 secs 2024-07-26T08:50:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 585.560898 secs 2024-07-26T08:50:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 585.560898 secs 2024-07-26T08:50:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:41 smithi044 ceph-mon[32404]: pgmap v870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:41 smithi116 ceph-mon[37097]: pgmap v870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:41 smithi171 ceph-mon[37789]: pgmap v870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:42 smithi171 ceph-mon[37789]: pgmap v871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:42 smithi044 ceph-mon[32404]: pgmap v871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:42 smithi116 ceph-mon[37097]: pgmap v871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:44 smithi171 ceph-mon[37789]: pgmap v872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:44 smithi044 ceph-mon[32404]: pgmap v872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:44 smithi116 ceph-mon[37097]: pgmap v872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 590.561059 secs 2024-07-26T08:50:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 590.561059 secs 2024-07-26T08:50:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 590.561059 secs 2024-07-26T08:50:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:47 smithi171 ceph-mon[37789]: pgmap v873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:47 smithi044 ceph-mon[32404]: pgmap v873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:47 smithi116 ceph-mon[37097]: pgmap v873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:48 smithi171 ceph-mon[37789]: pgmap v874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:48 smithi044 ceph-mon[32404]: pgmap v874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:48 smithi116 ceph-mon[37097]: pgmap v874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 595.561225 secs 2024-07-26T08:50:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 595.561225 secs 2024-07-26T08:50:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 595.561225 secs 2024-07-26T08:50:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:51 smithi044 ceph-mon[32404]: pgmap v875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:51 smithi116 ceph-mon[37097]: pgmap v875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:51 smithi171 ceph-mon[37789]: pgmap v875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:50:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:52 smithi171 ceph-mon[37789]: pgmap v876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:52 smithi044 ceph-mon[32404]: pgmap v876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:52 smithi116 ceph-mon[37097]: pgmap v876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:50:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:54 smithi171 ceph-mon[37789]: pgmap v877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:54 smithi044 ceph-mon[32404]: pgmap v877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:55.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:55.267+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:50:55.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:55.267+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:55.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:50:55.267+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:50:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:54 smithi116 ceph-mon[37097]: pgmap v877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:50:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 600.561342 secs 2024-07-26T08:50:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 600.561342 secs 2024-07-26T08:50:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 600.561342 secs 2024-07-26T08:50:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:56 smithi171 ceph-mon[37789]: pgmap v878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:56 smithi044 ceph-mon[32404]: pgmap v878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:56 smithi116 ceph-mon[37097]: pgmap v878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:50:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:50:58 smithi171 ceph-mon[37789]: pgmap v879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:50:58 smithi044 ceph-mon[32404]: pgmap v879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:50:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:50:58 smithi116 ceph-mon[37097]: pgmap v879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 605.561501 secs 2024-07-26T08:51:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 605.561501 secs 2024-07-26T08:51:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 605.561501 secs 2024-07-26T08:51:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:01 smithi044 ceph-mon[32404]: pgmap v880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:01 smithi116 ceph-mon[37097]: pgmap v880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:01 smithi171 ceph-mon[37789]: pgmap v880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:02 smithi171 ceph-mon[37789]: pgmap v881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:02 smithi044 ceph-mon[32404]: pgmap v881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:02 smithi116 ceph-mon[37097]: pgmap v881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:04 smithi171 ceph-mon[37789]: pgmap v882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:04 smithi044 ceph-mon[32404]: pgmap v882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:04 smithi116 ceph-mon[37097]: pgmap v882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 610.561672 secs 2024-07-26T08:51:06.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 610.561672 secs 2024-07-26T08:51:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 610.561672 secs 2024-07-26T08:51:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:06 smithi171 ceph-mon[37789]: pgmap v883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:06 smithi044 ceph-mon[32404]: pgmap v883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:06 smithi116 ceph-mon[37097]: pgmap v883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:08 smithi171 ceph-mon[37789]: pgmap v884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:08 smithi044 ceph-mon[32404]: pgmap v884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:08 smithi116 ceph-mon[37097]: pgmap v884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:10.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:10.269+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:51:10.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:10.269+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:51:10.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:10.269+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 615.561761 secs 2024-07-26T08:51:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 615.561761 secs 2024-07-26T08:51:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 615.561761 secs 2024-07-26T08:51:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:11 smithi044 ceph-mon[32404]: pgmap v885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:11 smithi116 ceph-mon[37097]: pgmap v885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:11 smithi171 ceph-mon[37789]: pgmap v885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:12 smithi171 ceph-mon[37789]: pgmap v886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:12 smithi044 ceph-mon[32404]: pgmap v886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:12 smithi116 ceph-mon[37097]: pgmap v886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:14 smithi171 ceph-mon[37789]: pgmap v887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:14 smithi044 ceph-mon[32404]: pgmap v887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:14 smithi116 ceph-mon[37097]: pgmap v887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 620.561916 secs 2024-07-26T08:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 620.561916 secs 2024-07-26T08:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 620.561916 secs 2024-07-26T08:51:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:16 smithi171 ceph-mon[37789]: pgmap v888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:16 smithi044 ceph-mon[32404]: pgmap v888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:16 smithi116 ceph-mon[37097]: pgmap v888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:51:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:18 smithi171 ceph-mon[37789]: pgmap v889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:18 smithi044 ceph-mon[32404]: pgmap v889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:18 smithi116 ceph-mon[37097]: pgmap v889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:51:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 625.562051 secs 2024-07-26T08:51:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 625.562051 secs 2024-07-26T08:51:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 625.562051 secs 2024-07-26T08:51:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:21 smithi044 ceph-mon[32404]: pgmap v890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:21 smithi116 ceph-mon[37097]: pgmap v890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:21 smithi171 ceph-mon[37789]: pgmap v890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:22 smithi171 ceph-mon[37789]: pgmap v891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:22 smithi044 ceph-mon[32404]: pgmap v891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:22 smithi116 ceph-mon[37097]: pgmap v891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:24 smithi171 ceph-mon[37789]: pgmap v892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:24 smithi044 ceph-mon[32404]: pgmap v892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:24 smithi116 ceph-mon[37097]: pgmap v892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:51:25.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:25.277+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:51:25.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:25.277+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:51:25.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:25.277+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:25.822 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 630.562170 secs 2024-07-26T08:51:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 630.562170 secs 2024-07-26T08:51:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 630.562170 secs 2024-07-26T08:51:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: pgmap v893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: pgmap v894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:51:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:51:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: pgmap v893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:51:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: pgmap v894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:51:27.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:51:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: pgmap v893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:51:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: pgmap v894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:51:27.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:51:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:29 smithi044 ceph-mon[32404]: pgmap v895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:29 smithi116 ceph-mon[37097]: pgmap v895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:29 smithi171 ceph-mon[37789]: pgmap v895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:31 smithi044 ceph-mon[32404]: pgmap v896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 635.562331 secs 2024-07-26T08:51:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:31 smithi116 ceph-mon[37097]: pgmap v896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 635.562331 secs 2024-07-26T08:51:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:31 smithi171 ceph-mon[37789]: pgmap v896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 635.562331 secs 2024-07-26T08:51:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:33 smithi044 ceph-mon[32404]: pgmap v897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:33 smithi116 ceph-mon[37097]: pgmap v897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:33 smithi171 ceph-mon[37789]: pgmap v897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:35 smithi044 ceph-mon[32404]: pgmap v898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:35 smithi116 ceph-mon[37097]: pgmap v898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:35 smithi171 ceph-mon[37789]: pgmap v898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:51:36.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 640.562441 secs 2024-07-26T08:51:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 640.562441 secs 2024-07-26T08:51:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 640.562441 secs 2024-07-26T08:51:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:37 smithi044 ceph-mon[32404]: pgmap v899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:51:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:37 smithi116 ceph-mon[37097]: pgmap v899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:51:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:37 smithi171 ceph-mon[37789]: pgmap v899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:51:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:39 smithi044 ceph-mon[32404]: pgmap v900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:39 smithi116 ceph-mon[37097]: pgmap v900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:39 smithi171 ceph-mon[37789]: pgmap v900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:40.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:40.280+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:51:40.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:40.280+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:40.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:40.280+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:51:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:41 smithi044 ceph-mon[32404]: pgmap v901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 645.562572 secs 2024-07-26T08:51:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:41 smithi116 ceph-mon[37097]: pgmap v901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 645.562572 secs 2024-07-26T08:51:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:41 smithi171 ceph-mon[37789]: pgmap v901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 645.562572 secs 2024-07-26T08:51:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:43 smithi044 ceph-mon[32404]: pgmap v902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:43 smithi116 ceph-mon[37097]: pgmap v902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:43 smithi171 ceph-mon[37789]: pgmap v902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:45 smithi044 ceph-mon[32404]: pgmap v903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:45 smithi116 ceph-mon[37097]: pgmap v903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:45 smithi171 ceph-mon[37789]: pgmap v903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 650.562707 secs 2024-07-26T08:51:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 650.562707 secs 2024-07-26T08:51:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 650.562707 secs 2024-07-26T08:51:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:47 smithi044 ceph-mon[32404]: pgmap v904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:47 smithi116 ceph-mon[37097]: pgmap v904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:47 smithi171 ceph-mon[37789]: pgmap v904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:49 smithi044 ceph-mon[32404]: pgmap v905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:49 smithi116 ceph-mon[37097]: pgmap v905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:49 smithi171 ceph-mon[37789]: pgmap v905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:51 smithi044 ceph-mon[32404]: pgmap v906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 655.562865 secs 2024-07-26T08:51:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:51 smithi116 ceph-mon[37097]: pgmap v906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 655.562865 secs 2024-07-26T08:51:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:51 smithi171 ceph-mon[37789]: pgmap v906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 655.562865 secs 2024-07-26T08:51:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:53 smithi044 ceph-mon[32404]: pgmap v907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:53 smithi116 ceph-mon[37097]: pgmap v907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:53 smithi171 ceph-mon[37789]: pgmap v907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:55.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:55.288+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:51:55.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:55.288+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:55.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:51:55.288+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:51:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:55 smithi044 ceph-mon[32404]: pgmap v908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:55 smithi116 ceph-mon[37097]: pgmap v908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:55 smithi171 ceph-mon[37789]: pgmap v908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:51:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 660.563035 secs 2024-07-26T08:51:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 660.563035 secs 2024-07-26T08:51:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 660.563035 secs 2024-07-26T08:51:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:57 smithi044 ceph-mon[32404]: pgmap v909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:57 smithi116 ceph-mon[37097]: pgmap v909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:57 smithi171 ceph-mon[37789]: pgmap v909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:51:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:51:59 smithi044 ceph-mon[32404]: pgmap v910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:51:59 smithi116 ceph-mon[37097]: pgmap v910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:51:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:51:59 smithi171 ceph-mon[37789]: pgmap v910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:52:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:01 smithi044 ceph-mon[32404]: pgmap v911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 665.563144 secs 2024-07-26T08:52:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:01 smithi116 ceph-mon[37097]: pgmap v911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 665.563144 secs 2024-07-26T08:52:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:01 smithi171 ceph-mon[37789]: pgmap v911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 665.563144 secs 2024-07-26T08:52:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:03 smithi044 ceph-mon[32404]: pgmap v912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:03 smithi116 ceph-mon[37097]: pgmap v912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:03 smithi171 ceph-mon[37789]: pgmap v912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:05 smithi044 ceph-mon[32404]: pgmap v913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:05 smithi116 ceph-mon[37097]: pgmap v913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:05 smithi171 ceph-mon[37789]: pgmap v913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 670.563262 secs 2024-07-26T08:52:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 670.563262 secs 2024-07-26T08:52:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 670.563262 secs 2024-07-26T08:52:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:07 smithi044 ceph-mon[32404]: pgmap v914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:07 smithi116 ceph-mon[37097]: pgmap v914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:07 smithi171 ceph-mon[37789]: pgmap v914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:09 smithi044 ceph-mon[32404]: pgmap v915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:52:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:09 smithi116 ceph-mon[37097]: pgmap v915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:52:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:09 smithi171 ceph-mon[37789]: pgmap v915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:52:10.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:10.291+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:52:10.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:10.291+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:52:10.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:10.291+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:11 smithi044 ceph-mon[32404]: pgmap v916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 675.563344 secs 2024-07-26T08:52:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:11 smithi116 ceph-mon[37097]: pgmap v916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 675.563344 secs 2024-07-26T08:52:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:11 smithi171 ceph-mon[37789]: pgmap v916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 675.563344 secs 2024-07-26T08:52:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:13 smithi044 ceph-mon[32404]: pgmap v917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:13 smithi116 ceph-mon[37097]: pgmap v917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:13 smithi171 ceph-mon[37789]: pgmap v917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:15 smithi116 ceph-mon[37097]: pgmap v918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:15 smithi171 ceph-mon[37789]: pgmap v918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:15 smithi044 ceph-mon[32404]: pgmap v918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 680.563456 secs 2024-07-26T08:52:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 680.563456 secs 2024-07-26T08:52:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 680.563456 secs 2024-07-26T08:52:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:17 smithi171 ceph-mon[37789]: pgmap v919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:17 smithi044 ceph-mon[32404]: pgmap v919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:17 smithi116 ceph-mon[37097]: pgmap v919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:19 smithi171 ceph-mon[37789]: pgmap v920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:52:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:19 smithi044 ceph-mon[32404]: pgmap v920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:52:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:19 smithi116 ceph-mon[37097]: pgmap v920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T08:52:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:21 smithi171 ceph-mon[37789]: pgmap v921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 685.563554 secs 2024-07-26T08:52:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:21 smithi044 ceph-mon[32404]: pgmap v921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 685.563554 secs 2024-07-26T08:52:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:21 smithi116 ceph-mon[37097]: pgmap v921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 685.563554 secs 2024-07-26T08:52:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:23 smithi171 ceph-mon[37789]: pgmap v922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:23 smithi044 ceph-mon[32404]: pgmap v922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:23 smithi116 ceph-mon[37097]: pgmap v922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:25.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:25.299+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:52:25.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:25.299+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:52:25.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:25.299+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:25.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:25 smithi171 ceph-mon[37789]: pgmap v923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:25 smithi044 ceph-mon[32404]: pgmap v923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:25 smithi116 ceph-mon[37097]: pgmap v923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:52:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 690.563715 secs 2024-07-26T08:52:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 690.563715 secs 2024-07-26T08:52:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 690.563715 secs 2024-07-26T08:52:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: pgmap v924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:27.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:52:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: pgmap v924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T08:52:27.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T08:52:27.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:52:27.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T08:52:27.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:27.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:27.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:52:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: pgmap v924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:27.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:52:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:28 smithi171 ceph-mon[37789]: pgmap v925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:52:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:28 smithi171 ceph-mon[37789]: pgmap v926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:28 smithi044 ceph-mon[32404]: pgmap v925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:52:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:28 smithi044 ceph-mon[32404]: pgmap v926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:28 smithi116 ceph-mon[37097]: pgmap v925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:52:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:28 smithi116 ceph-mon[37097]: pgmap v926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:30 smithi171 ceph-mon[37789]: pgmap v927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:30 smithi044 ceph-mon[32404]: pgmap v927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:30 smithi116 ceph-mon[37097]: pgmap v927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 695.563849 secs 2024-07-26T08:52:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 695.563849 secs 2024-07-26T08:52:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 695.563849 secs 2024-07-26T08:52:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:32 smithi044 ceph-mon[32404]: pgmap v928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:32 smithi116 ceph-mon[37097]: pgmap v928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:32 smithi171 ceph-mon[37789]: pgmap v928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:34 smithi044 ceph-mon[32404]: pgmap v929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:34 smithi116 ceph-mon[37097]: pgmap v929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:34 smithi171 ceph-mon[37789]: pgmap v929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 700.564014 secs 2024-07-26T08:52:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:36 smithi044 ceph-mon[32404]: pgmap v930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 700.564014 secs 2024-07-26T08:52:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:36 smithi116 ceph-mon[37097]: pgmap v930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 700.564014 secs 2024-07-26T08:52:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:36 smithi171 ceph-mon[37789]: pgmap v930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:52:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:38 smithi044 ceph-mon[32404]: pgmap v931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:38 smithi116 ceph-mon[37097]: pgmap v931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:38 smithi171 ceph-mon[37789]: pgmap v931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:52:40.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:40.302+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:52:40.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:40.302+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:52:40.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:40.302+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:40 smithi044 ceph-mon[32404]: pgmap v932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:40 smithi116 ceph-mon[37097]: pgmap v932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:40 smithi171 ceph-mon[37789]: pgmap v932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:52:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 705.564156 secs 2024-07-26T08:52:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 705.564156 secs 2024-07-26T08:52:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 705.564156 secs 2024-07-26T08:52:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:42 smithi044 ceph-mon[32404]: pgmap v933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:42 smithi116 ceph-mon[37097]: pgmap v933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:42 smithi171 ceph-mon[37789]: pgmap v933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:44 smithi044 ceph-mon[32404]: pgmap v934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T08:52:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:44 smithi116 ceph-mon[37097]: pgmap v934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T08:52:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:44 smithi171 ceph-mon[37789]: pgmap v934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T08:52:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 710.564318 secs 2024-07-26T08:52:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:46 smithi044 ceph-mon[32404]: pgmap v935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 710.564318 secs 2024-07-26T08:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:46 smithi116 ceph-mon[37097]: pgmap v935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:52:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 710.564318 secs 2024-07-26T08:52:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:46 smithi171 ceph-mon[37789]: pgmap v935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:52:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:48 smithi044 ceph-mon[32404]: pgmap v936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:48 smithi116 ceph-mon[37097]: pgmap v936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:48 smithi171 ceph-mon[37789]: pgmap v936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:50 smithi044 ceph-mon[32404]: pgmap v937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 715.564461 secs 2024-07-26T08:52:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:50 smithi116 ceph-mon[37097]: pgmap v937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 715.564461 secs 2024-07-26T08:52:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:50 smithi171 ceph-mon[37789]: pgmap v937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:52:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 715.564461 secs 2024-07-26T08:52:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:52 smithi044 ceph-mon[32404]: pgmap v938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:52 smithi116 ceph-mon[37097]: pgmap v938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:52 smithi171 ceph-mon[37789]: pgmap v938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:52:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:54 smithi044 ceph-mon[32404]: pgmap v939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:52:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:54 smithi116 ceph-mon[37097]: pgmap v939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:52:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:54 smithi171 ceph-mon[37789]: pgmap v939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:52:55.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:55.310+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:52:55.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:55.310+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:52:55.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:52:55.310+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:55 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 720.564574 secs 2024-07-26T08:52:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:55 smithi044 ceph-mon[32404]: slow request 720.564573 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:52:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:55 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 720.564574 secs 2024-07-26T08:52:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:55 smithi116 ceph-mon[37097]: slow request 720.564573 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:52:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:55 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 720.564574 secs 2024-07-26T08:52:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:55 smithi171 ceph-mon[37789]: slow request 720.564573 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T08:52:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:56 smithi044 ceph-mon[32404]: pgmap v940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:56 smithi116 ceph-mon[37097]: pgmap v940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:56 smithi171 ceph-mon[37789]: pgmap v940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:52:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:52:58 smithi044 ceph-mon[32404]: pgmap v941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:52:58 smithi116 ceph-mon[37097]: pgmap v941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:52:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:52:58 smithi171 ceph-mon[37789]: pgmap v941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:00 smithi044 ceph-mon[32404]: pgmap v942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 725.564726 secs 2024-07-26T08:53:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:00 smithi116 ceph-mon[37097]: pgmap v942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 725.564726 secs 2024-07-26T08:53:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:00 smithi171 ceph-mon[37789]: pgmap v942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 725.564726 secs 2024-07-26T08:53:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:02 smithi044 ceph-mon[32404]: pgmap v943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:02 smithi116 ceph-mon[37097]: pgmap v943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:02 smithi171 ceph-mon[37789]: pgmap v943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:04 smithi044 ceph-mon[32404]: pgmap v944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:04 smithi116 ceph-mon[37097]: pgmap v944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:04 smithi171 ceph-mon[37789]: pgmap v944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 730.564888 secs 2024-07-26T08:53:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 730.564888 secs 2024-07-26T08:53:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 730.564888 secs 2024-07-26T08:53:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:06 smithi044 ceph-mon[32404]: pgmap v945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:06 smithi116 ceph-mon[37097]: pgmap v945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:06 smithi171 ceph-mon[37789]: pgmap v945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:08 smithi044 ceph-mon[32404]: pgmap v946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:08 smithi116 ceph-mon[37097]: pgmap v946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:08 smithi171 ceph-mon[37789]: pgmap v946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:10.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:10.313+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:53:10.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:10.313+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:53:10.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:10.313+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:10 smithi044 ceph-mon[32404]: pgmap v947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 735.564988 secs 2024-07-26T08:53:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:10 smithi116 ceph-mon[37097]: pgmap v947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 735.564988 secs 2024-07-26T08:53:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:10 smithi171 ceph-mon[37789]: pgmap v947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 735.564988 secs 2024-07-26T08:53:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:12 smithi044 ceph-mon[32404]: pgmap v948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:12 smithi116 ceph-mon[37097]: pgmap v948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:12 smithi171 ceph-mon[37789]: pgmap v948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:14 smithi044 ceph-mon[32404]: pgmap v949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:14 smithi116 ceph-mon[37097]: pgmap v949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:14.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:14.789+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c006020 con 0x55c94889a070 2024-07-26T08:53:14.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:14.789+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006030 con 0x7fcf6c1057b0 2024-07-26T08:53:14.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:14.789+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006030 con 0x7fcf6c1057b0 2024-07-26T08:53:14.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:14.789+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c006020 con 0x55c94889a070 2024-07-26T08:53:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:14 smithi171 ceph-mon[37789]: pgmap v949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 740.565143 secs 2024-07-26T08:53:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 740.565143 secs 2024-07-26T08:53:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 740.565143 secs 2024-07-26T08:53:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:16 smithi044 ceph-mon[32404]: pgmap v950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:16 smithi116 ceph-mon[37097]: pgmap v950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:16 smithi171 ceph-mon[37789]: pgmap v950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:18 smithi116 ceph-mon[37097]: pgmap v951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:18 smithi171 ceph-mon[37789]: pgmap v951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:18 smithi044 ceph-mon[32404]: pgmap v951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:20 smithi171 ceph-mon[37789]: pgmap v952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 745.565301 secs 2024-07-26T08:53:20.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:20 smithi044 ceph-mon[32404]: pgmap v952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:20.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 745.565301 secs 2024-07-26T08:53:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:20 smithi116 ceph-mon[37097]: pgmap v952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 745.565301 secs 2024-07-26T08:53:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:22 smithi171 ceph-mon[37789]: pgmap v953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:22 smithi044 ceph-mon[32404]: pgmap v953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:22 smithi116 ceph-mon[37097]: pgmap v953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:24 smithi171 ceph-mon[37789]: pgmap v954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:24 smithi044 ceph-mon[32404]: pgmap v954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:24 smithi116 ceph-mon[37097]: pgmap v954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:53:25.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:25.321+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:53:25.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:25.321+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:25.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:25.321+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:53:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 750.565434 secs 2024-07-26T08:53:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 750.565434 secs 2024-07-26T08:53:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 750.565434 secs 2024-07-26T08:53:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:26 smithi171 ceph-mon[37789]: pgmap v955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:26 smithi044 ceph-mon[32404]: pgmap v955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:26 smithi116 ceph-mon[37097]: pgmap v955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:28 smithi171 ceph-mon[37789]: pgmap v956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:28.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:53:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:28 smithi044 ceph-mon[32404]: pgmap v956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:53:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:28 smithi116 ceph-mon[37097]: pgmap v956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:53:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:29 smithi171 ceph-mon[37789]: pgmap v957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:53:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:29 smithi171 ceph-mon[37789]: pgmap v958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:29 smithi044 ceph-mon[32404]: pgmap v957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:53:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:29 smithi044 ceph-mon[32404]: pgmap v958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:29 smithi116 ceph-mon[37097]: pgmap v957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:53:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:29 smithi116 ceph-mon[37097]: pgmap v958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 755.565591 secs 2024-07-26T08:53:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 755.565591 secs 2024-07-26T08:53:31.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 755.565591 secs 2024-07-26T08:53:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:31 smithi044 ceph-mon[32404]: pgmap v959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:31 smithi116 ceph-mon[37097]: pgmap v959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:31 smithi171 ceph-mon[37789]: pgmap v959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:33 smithi044 ceph-mon[32404]: pgmap v960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:53:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:33 smithi116 ceph-mon[37097]: pgmap v960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:53:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:33 smithi171 ceph-mon[37789]: pgmap v960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T08:53:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:35 smithi044 ceph-mon[32404]: pgmap v961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 760.565699 secs 2024-07-26T08:53:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:35 smithi116 ceph-mon[37097]: pgmap v961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 760.565699 secs 2024-07-26T08:53:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:35 smithi171 ceph-mon[37789]: pgmap v961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:53:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 760.565699 secs 2024-07-26T08:53:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:37 smithi044 ceph-mon[32404]: pgmap v962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:53:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:37 smithi116 ceph-mon[37097]: pgmap v962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:53:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:37 smithi171 ceph-mon[37789]: pgmap v962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:53:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:39 smithi044 ceph-mon[32404]: pgmap v963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:39 smithi116 ceph-mon[37097]: pgmap v963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:39 smithi171 ceph-mon[37789]: pgmap v963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:53:40.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:40.324+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:53:40.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:40.324+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:53:40.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:40.324+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 765.565802 secs 2024-07-26T08:53:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 765.565802 secs 2024-07-26T08:53:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 765.565802 secs 2024-07-26T08:53:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:41 smithi044 ceph-mon[32404]: pgmap v964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:41 smithi116 ceph-mon[37097]: pgmap v964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:41 smithi171 ceph-mon[37789]: pgmap v964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:43 smithi044 ceph-mon[32404]: pgmap v965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:43 smithi116 ceph-mon[37097]: pgmap v965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:43 smithi171 ceph-mon[37789]: pgmap v965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:53:46.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:45 smithi044 ceph-mon[32404]: pgmap v966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:46.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 770.565966 secs 2024-07-26T08:53:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:45 smithi116 ceph-mon[37097]: pgmap v966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 770.565966 secs 2024-07-26T08:53:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:45 smithi171 ceph-mon[37789]: pgmap v966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:53:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 770.565966 secs 2024-07-26T08:53:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:47 smithi044 ceph-mon[32404]: pgmap v967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:47 smithi116 ceph-mon[37097]: pgmap v967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:47 smithi171 ceph-mon[37789]: pgmap v967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:49 smithi044 ceph-mon[32404]: pgmap v968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:53:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:49 smithi116 ceph-mon[37097]: pgmap v968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:53:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:49 smithi171 ceph-mon[37789]: pgmap v968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:53:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 775.566125 secs 2024-07-26T08:53:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 775.566125 secs 2024-07-26T08:53:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 775.566125 secs 2024-07-26T08:53:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:51 smithi044 ceph-mon[32404]: pgmap v969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:51 smithi116 ceph-mon[37097]: pgmap v969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:51 smithi171 ceph-mon[37789]: pgmap v969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:53:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:53 smithi044 ceph-mon[32404]: pgmap v970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:53:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:53 smithi116 ceph-mon[37097]: pgmap v970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:53:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:53 smithi171 ceph-mon[37789]: pgmap v970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:53:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:55.332+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:53:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:55.332+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:55.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:53:55.332+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:53:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:55 smithi044 ceph-mon[32404]: pgmap v971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:53:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 780.566291 secs 2024-07-26T08:53:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:55 smithi116 ceph-mon[37097]: pgmap v971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:53:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 780.566291 secs 2024-07-26T08:53:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:55 smithi171 ceph-mon[37789]: pgmap v971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:53:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 780.566291 secs 2024-07-26T08:53:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:57 smithi044 ceph-mon[32404]: pgmap v972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:57 smithi116 ceph-mon[37097]: pgmap v972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:53:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:57 smithi171 ceph-mon[37789]: pgmap v972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:53:59 smithi044 ceph-mon[32404]: pgmap v973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:54:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:53:59 smithi116 ceph-mon[37097]: pgmap v973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:54:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:53:59 smithi171 ceph-mon[37789]: pgmap v973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T08:54:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 785.566433 secs 2024-07-26T08:54:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 785.566433 secs 2024-07-26T08:54:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 785.566433 secs 2024-07-26T08:54:02.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:01 smithi044 ceph-mon[32404]: pgmap v974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:54:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:01 smithi116 ceph-mon[37097]: pgmap v974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:54:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:01 smithi171 ceph-mon[37789]: pgmap v974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:54:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:03 smithi044 ceph-mon[32404]: pgmap v975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:03 smithi116 ceph-mon[37097]: pgmap v975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:03 smithi171 ceph-mon[37789]: pgmap v975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:54:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:05 smithi044 ceph-mon[32404]: pgmap v976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 790.566594 secs 2024-07-26T08:54:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:05 smithi116 ceph-mon[37097]: pgmap v976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 790.566594 secs 2024-07-26T08:54:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:05 smithi171 ceph-mon[37789]: pgmap v976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 790.566594 secs 2024-07-26T08:54:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:07 smithi044 ceph-mon[32404]: pgmap v977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:07 smithi116 ceph-mon[37097]: pgmap v977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:07 smithi171 ceph-mon[37789]: pgmap v977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:09 smithi044 ceph-mon[32404]: pgmap v978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:09 smithi116 ceph-mon[37097]: pgmap v978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:09 smithi171 ceph-mon[37789]: pgmap v978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:10.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:10.335+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:54:10.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:10.335+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:54:10.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:10.335+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 795.566728 secs 2024-07-26T08:54:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 795.566728 secs 2024-07-26T08:54:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 795.566728 secs 2024-07-26T08:54:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:11 smithi044 ceph-mon[32404]: pgmap v979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:11 smithi116 ceph-mon[37097]: pgmap v979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:11 smithi171 ceph-mon[37789]: pgmap v979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:13 smithi044 ceph-mon[32404]: pgmap v980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:13 smithi116 ceph-mon[37097]: pgmap v980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:13 smithi171 ceph-mon[37789]: pgmap v980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:15 smithi116 ceph-mon[37097]: pgmap v981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 800.566826 secs 2024-07-26T08:54:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:15 smithi171 ceph-mon[37789]: pgmap v981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 800.566826 secs 2024-07-26T08:54:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:15 smithi044 ceph-mon[32404]: pgmap v981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 800.566826 secs 2024-07-26T08:54:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:17 smithi171 ceph-mon[37789]: pgmap v982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:17 smithi044 ceph-mon[32404]: pgmap v982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:17 smithi116 ceph-mon[37097]: pgmap v982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:19 smithi171 ceph-mon[37789]: pgmap v983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:19 smithi044 ceph-mon[32404]: pgmap v983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:19 smithi116 ceph-mon[37097]: pgmap v983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 805.566982 secs 2024-07-26T08:54:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 805.566982 secs 2024-07-26T08:54:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 805.566982 secs 2024-07-26T08:54:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:21 smithi171 ceph-mon[37789]: pgmap v984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:21 smithi044 ceph-mon[32404]: pgmap v984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:21 smithi116 ceph-mon[37097]: pgmap v984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:23 smithi171 ceph-mon[37789]: pgmap v985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:23 smithi044 ceph-mon[32404]: pgmap v985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:23 smithi116 ceph-mon[37097]: pgmap v985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:25.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:25.344+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:54:25.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:25.344+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:25.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:25.344+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:54:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:25 smithi171 ceph-mon[37789]: pgmap v986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 810.567128 secs 2024-07-26T08:54:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:25 smithi044 ceph-mon[32404]: pgmap v986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 810.567128 secs 2024-07-26T08:54:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:25 smithi116 ceph-mon[37097]: pgmap v986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 810.567128 secs 2024-07-26T08:54:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:26 smithi044 ceph-mon[32404]: pgmap v987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:26 smithi116 ceph-mon[37097]: pgmap v987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:26 smithi171 ceph-mon[37789]: pgmap v987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:54:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:29 smithi044 ceph-mon[32404]: pgmap v988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:29 smithi116 ceph-mon[37097]: pgmap v988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:29 smithi171 ceph-mon[37789]: pgmap v988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:54:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 815.567260 secs 2024-07-26T08:54:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 815.567260 secs 2024-07-26T08:54:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 815.567260 secs 2024-07-26T08:54:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:54:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:31 smithi044 ceph-mon[32404]: pgmap v989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:31 smithi044 ceph-mon[32404]: pgmap v990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:31 smithi116 ceph-mon[37097]: pgmap v989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:31 smithi116 ceph-mon[37097]: pgmap v990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:31 smithi171 ceph-mon[37789]: pgmap v989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:31 smithi171 ceph-mon[37789]: pgmap v990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:33 smithi044 ceph-mon[32404]: pgmap v991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:33 smithi116 ceph-mon[37097]: pgmap v991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:33 smithi171 ceph-mon[37789]: pgmap v991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:35 smithi044 ceph-mon[32404]: pgmap v992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 820.567403 secs 2024-07-26T08:54:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:35 smithi116 ceph-mon[37097]: pgmap v992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 820.567403 secs 2024-07-26T08:54:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:35 smithi171 ceph-mon[37789]: pgmap v992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 820.567403 secs 2024-07-26T08:54:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:37 smithi044 ceph-mon[32404]: pgmap v993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:54:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:37 smithi116 ceph-mon[37097]: pgmap v993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:54:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:37 smithi171 ceph-mon[37789]: pgmap v993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:54:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:39 smithi044 ceph-mon[32404]: pgmap v994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:39 smithi116 ceph-mon[37097]: pgmap v994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:39 smithi171 ceph-mon[37789]: pgmap v994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T08:54:40.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:40.347+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:54:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:40.347+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:54:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:40.347+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 825.567545 secs 2024-07-26T08:54:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 825.567545 secs 2024-07-26T08:54:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 825.567545 secs 2024-07-26T08:54:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:41 smithi044 ceph-mon[32404]: pgmap v995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:41 smithi116 ceph-mon[37097]: pgmap v995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:41 smithi171 ceph-mon[37789]: pgmap v995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:54:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:43 smithi044 ceph-mon[32404]: pgmap v996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:43 smithi116 ceph-mon[37097]: pgmap v996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:43 smithi171 ceph-mon[37789]: pgmap v996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:45 smithi044 ceph-mon[32404]: pgmap v997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 830.567690 secs 2024-07-26T08:54:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:45 smithi116 ceph-mon[37097]: pgmap v997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 830.567690 secs 2024-07-26T08:54:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:45 smithi171 ceph-mon[37789]: pgmap v997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 830.567690 secs 2024-07-26T08:54:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:47 smithi044 ceph-mon[32404]: pgmap v998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:47 smithi116 ceph-mon[37097]: pgmap v998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:47 smithi171 ceph-mon[37789]: pgmap v998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:49 smithi044 ceph-mon[32404]: pgmap v999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:54:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:49 smithi116 ceph-mon[37097]: pgmap v999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:54:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:49 smithi171 ceph-mon[37789]: pgmap v999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:54:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 835.567827 secs 2024-07-26T08:54:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 835.567827 secs 2024-07-26T08:54:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 835.567827 secs 2024-07-26T08:54:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:51 smithi044 ceph-mon[32404]: pgmap v1000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:51 smithi116 ceph-mon[37097]: pgmap v1000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:51 smithi171 ceph-mon[37789]: pgmap v1000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:53 smithi044 ceph-mon[32404]: pgmap v1001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:53 smithi116 ceph-mon[37097]: pgmap v1001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:53 smithi171 ceph-mon[37789]: pgmap v1001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:54:55.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:55.353+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:54:55.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:55.353+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:55.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:54:55.353+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:54:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:55 smithi044 ceph-mon[32404]: pgmap v1002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 840.567931 secs 2024-07-26T08:54:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:55 smithi116 ceph-mon[37097]: pgmap v1002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 840.567931 secs 2024-07-26T08:54:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:55 smithi171 ceph-mon[37789]: pgmap v1002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:54:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 840.567931 secs 2024-07-26T08:54:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:57 smithi044 ceph-mon[32404]: pgmap v1003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:57 smithi116 ceph-mon[37097]: pgmap v1003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:54:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:57 smithi171 ceph-mon[37789]: pgmap v1003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:54:59 smithi044 ceph-mon[32404]: pgmap v1004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:54:59 smithi116 ceph-mon[37097]: pgmap v1004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:54:59 smithi171 ceph-mon[37789]: pgmap v1004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 845.568064 secs 2024-07-26T08:55:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 845.568064 secs 2024-07-26T08:55:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 845.568064 secs 2024-07-26T08:55:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:01 smithi044 ceph-mon[32404]: pgmap v1005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:01 smithi116 ceph-mon[37097]: pgmap v1005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:01 smithi171 ceph-mon[37789]: pgmap v1005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:03 smithi116 ceph-mon[37097]: pgmap v1006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:03 smithi171 ceph-mon[37789]: pgmap v1006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:03 smithi044 ceph-mon[32404]: pgmap v1006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:06.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:05 smithi044 ceph-mon[32404]: pgmap v1007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:06.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 850.568214 secs 2024-07-26T08:55:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:05 smithi171 ceph-mon[37789]: pgmap v1007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 850.568214 secs 2024-07-26T08:55:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:05 smithi116 ceph-mon[37097]: pgmap v1007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 850.568214 secs 2024-07-26T08:55:08.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:07 smithi116 ceph-mon[37097]: pgmap v1008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:07 smithi171 ceph-mon[37789]: pgmap v1008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:07 smithi044 ceph-mon[32404]: pgmap v1008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:09 smithi171 ceph-mon[37789]: pgmap v1009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:09 smithi044 ceph-mon[32404]: pgmap v1009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:09 smithi116 ceph-mon[37097]: pgmap v1009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:10.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:10.355+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:55:10.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:10.355+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:10.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:10.355+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:55:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 855.568311 secs 2024-07-26T08:55:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 855.568311 secs 2024-07-26T08:55:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 855.568311 secs 2024-07-26T08:55:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:11 smithi171 ceph-mon[37789]: pgmap v1010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:11 smithi044 ceph-mon[32404]: pgmap v1010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:11 smithi116 ceph-mon[37097]: pgmap v1010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:13 smithi171 ceph-mon[37789]: pgmap v1011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:13 smithi044 ceph-mon[32404]: pgmap v1011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:13 smithi116 ceph-mon[37097]: pgmap v1011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:15 smithi171 ceph-mon[37789]: pgmap v1012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 860.568472 secs 2024-07-26T08:55:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:15 smithi044 ceph-mon[32404]: pgmap v1012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 860.568472 secs 2024-07-26T08:55:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:15 smithi116 ceph-mon[37097]: pgmap v1012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 860.568472 secs 2024-07-26T08:55:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:17 smithi171 ceph-mon[37789]: pgmap v1013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:17 smithi044 ceph-mon[32404]: pgmap v1013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:17 smithi116 ceph-mon[37097]: pgmap v1013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:18 smithi171 ceph-mon[37789]: pgmap v1014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:18 smithi044 ceph-mon[32404]: pgmap v1014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:18 smithi116 ceph-mon[37097]: pgmap v1014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 865.568664 secs 2024-07-26T08:55:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 865.568664 secs 2024-07-26T08:55:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 865.568664 secs 2024-07-26T08:55:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:21 smithi044 ceph-mon[32404]: pgmap v1015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:21 smithi116 ceph-mon[37097]: pgmap v1015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:21 smithi171 ceph-mon[37789]: pgmap v1015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:23 smithi044 ceph-mon[32404]: pgmap v1016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:23 smithi116 ceph-mon[37097]: pgmap v1016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:23 smithi171 ceph-mon[37789]: pgmap v1016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:25.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:25.363+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:55:25.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:25.363+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:25.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:25.363+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:55:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:25 smithi044 ceph-mon[32404]: pgmap v1017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 870.568823 secs 2024-07-26T08:55:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:25 smithi116 ceph-mon[37097]: pgmap v1017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 870.568823 secs 2024-07-26T08:55:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:25 smithi171 ceph-mon[37789]: pgmap v1017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 870.568823 secs 2024-07-26T08:55:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:27 smithi044 ceph-mon[32404]: pgmap v1018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:27 smithi116 ceph-mon[37097]: pgmap v1018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:27 smithi171 ceph-mon[37789]: pgmap v1018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:29 smithi044 ceph-mon[32404]: pgmap v1019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:29 smithi116 ceph-mon[37097]: pgmap v1019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:29 smithi171 ceph-mon[37789]: pgmap v1019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T08:55:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 875.568983 secs 2024-07-26T08:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: pgmap v1020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 875.568983 secs 2024-07-26T08:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: pgmap v1020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 875.568983 secs 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: pgmap v1020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:31.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:33 smithi044 ceph-mon[32404]: pgmap v1021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:33 smithi116 ceph-mon[37097]: pgmap v1021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:33 smithi171 ceph-mon[37789]: pgmap v1021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:35 smithi044 ceph-mon[32404]: pgmap v1022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 880.569069 secs 2024-07-26T08:55:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:35 smithi116 ceph-mon[37097]: pgmap v1022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 880.569069 secs 2024-07-26T08:55:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:35 smithi171 ceph-mon[37789]: pgmap v1022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 880.569069 secs 2024-07-26T08:55:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: pgmap v1023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:55:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: pgmap v1024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: pgmap v1023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:55:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: pgmap v1024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: pgmap v1023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:55:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: pgmap v1024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:55:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:38 smithi171 ceph-mon[37789]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:55:38.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:38 smithi116 ceph-mon[37097]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:55:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:38 smithi044 ceph-mon[32404]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T08:55:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:39 smithi044 ceph-mon[32404]: pgmap v1025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:39 smithi116 ceph-mon[37097]: pgmap v1025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:39 smithi171 ceph-mon[37789]: pgmap v1025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:40.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:40.367+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:55:40.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:40.367+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:40.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:40.367+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:55:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:41 smithi044 ceph-mon[32404]: pgmap v1026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 885.569233 secs 2024-07-26T08:55:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:41 smithi116 ceph-mon[37097]: pgmap v1026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 885.569233 secs 2024-07-26T08:55:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:41 smithi171 ceph-mon[37789]: pgmap v1026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:55:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 885.569233 secs 2024-07-26T08:55:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:43 smithi044 ceph-mon[32404]: pgmap v1027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:43 smithi116 ceph-mon[37097]: pgmap v1027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:43 smithi171 ceph-mon[37789]: pgmap v1027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:45 smithi044 ceph-mon[32404]: pgmap v1028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:45 smithi116 ceph-mon[37097]: pgmap v1028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:45 smithi171 ceph-mon[37789]: pgmap v1028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 890.569388 secs 2024-07-26T08:55:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 890.569388 secs 2024-07-26T08:55:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 890.569388 secs 2024-07-26T08:55:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:47 smithi044 ceph-mon[32404]: pgmap v1029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:55:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:47 smithi116 ceph-mon[37097]: pgmap v1029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:55:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:47 smithi171 ceph-mon[37789]: pgmap v1029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:55:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:49 smithi044 ceph-mon[32404]: pgmap v1030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:55:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:49 smithi116 ceph-mon[37097]: pgmap v1030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:55:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:49 smithi171 ceph-mon[37789]: pgmap v1030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:55:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 895.569527 secs 2024-07-26T08:55:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 895.569527 secs 2024-07-26T08:55:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 895.569527 secs 2024-07-26T08:55:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:51 smithi044 ceph-mon[32404]: pgmap v1031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:51 smithi116 ceph-mon[37097]: pgmap v1031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:51 smithi171 ceph-mon[37789]: pgmap v1031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:53 smithi044 ceph-mon[32404]: pgmap v1032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:53 smithi116 ceph-mon[37097]: pgmap v1032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:53 smithi171 ceph-mon[37789]: pgmap v1032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:55.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:55.370+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:55:55.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:55.370+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:55.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:55:55.370+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:55:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:55 smithi044 ceph-mon[32404]: pgmap v1033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 900.569688 secs 2024-07-26T08:55:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:55 smithi116 ceph-mon[37097]: pgmap v1033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 900.569688 secs 2024-07-26T08:55:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:55 smithi171 ceph-mon[37789]: pgmap v1033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:55:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 900.569688 secs 2024-07-26T08:55:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:57 smithi044 ceph-mon[32404]: pgmap v1034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:57 smithi116 ceph-mon[37097]: pgmap v1034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:57 smithi171 ceph-mon[37789]: pgmap v1034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:55:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:55:59 smithi044 ceph-mon[32404]: pgmap v1035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:55:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:55:59 smithi116 ceph-mon[37097]: pgmap v1035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:55:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:55:59 smithi171 ceph-mon[37789]: pgmap v1035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 905.569847 secs 2024-07-26T08:56:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 905.569847 secs 2024-07-26T08:56:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 905.569847 secs 2024-07-26T08:56:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:01 smithi044 ceph-mon[32404]: pgmap v1036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:01 smithi116 ceph-mon[37097]: pgmap v1036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:01 smithi171 ceph-mon[37789]: pgmap v1036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:03 smithi044 ceph-mon[32404]: pgmap v1037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:03 smithi116 ceph-mon[37097]: pgmap v1037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:03 smithi171 ceph-mon[37789]: pgmap v1037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:05 smithi044 ceph-mon[32404]: pgmap v1038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 910.570001 secs 2024-07-26T08:56:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:05 smithi116 ceph-mon[37097]: pgmap v1038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 910.570001 secs 2024-07-26T08:56:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:05 smithi171 ceph-mon[37789]: pgmap v1038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 910.570001 secs 2024-07-26T08:56:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:07 smithi116 ceph-mon[37097]: pgmap v1039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:07 smithi171 ceph-mon[37789]: pgmap v1039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:07 smithi044 ceph-mon[32404]: pgmap v1039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:09 smithi171 ceph-mon[37789]: pgmap v1040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:09 smithi044 ceph-mon[32404]: pgmap v1040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:09 smithi116 ceph-mon[37097]: pgmap v1040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:10.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:10.373+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:56:10.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:10.373+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:10.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:10.373+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:56:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 915.570114 secs 2024-07-26T08:56:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 915.570114 secs 2024-07-26T08:56:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 915.570114 secs 2024-07-26T08:56:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:11 smithi171 ceph-mon[37789]: pgmap v1041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:11.938 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:11 smithi044 ceph-mon[32404]: pgmap v1041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:11 smithi116 ceph-mon[37097]: pgmap v1041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:13 smithi171 ceph-mon[37789]: pgmap v1042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:13 smithi044 ceph-mon[32404]: pgmap v1042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:13 smithi116 ceph-mon[37097]: pgmap v1042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:14.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf72600640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T08:56:14.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf72600640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:56:14.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf72600640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b6250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcf64005fd0 tx=0x7fcf64009500 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:56:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf72600640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b6250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcf64005fd0 tx=0x7fcf64009500 comp rx=0 tx=0).stop 2024-07-26T08:56:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf72600640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b6250 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T08:56:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:56:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b6250 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:56:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.675+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:56:14.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.676+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:56:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.679+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf58005ef0 tx=0x7fcf5802a3f0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:56:14.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.820+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006290 con 0x7fcf6c1057b0 2024-07-26T08:56:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.820+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0062e0 con 0x55c94889a070 2024-07-26T08:56:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.821+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0062e0 con 0x55c94889a070 2024-07-26T08:56:14.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:14.821+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006290 con 0x7fcf6c1057b0 2024-07-26T08:56:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:15 smithi171 ceph-mon[37789]: pgmap v1043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 920.570280 secs 2024-07-26T08:56:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:15 smithi044 ceph-mon[32404]: pgmap v1043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 920.570280 secs 2024-07-26T08:56:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:15 smithi116 ceph-mon[37097]: pgmap v1043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 920.570280 secs 2024-07-26T08:56:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:17 smithi171 ceph-mon[37789]: pgmap v1044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:17 smithi044 ceph-mon[32404]: pgmap v1044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:17 smithi116 ceph-mon[37097]: pgmap v1044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:19 smithi171 ceph-mon[37789]: pgmap v1045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:19 smithi044 ceph-mon[32404]: pgmap v1045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:19 smithi116 ceph-mon[37097]: pgmap v1045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 925.570381 secs 2024-07-26T08:56:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 925.570381 secs 2024-07-26T08:56:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 925.570381 secs 2024-07-26T08:56:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:21 smithi171 ceph-mon[37789]: pgmap v1046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:21 smithi044 ceph-mon[32404]: pgmap v1046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:21 smithi116 ceph-mon[37097]: pgmap v1046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:23 smithi171 ceph-mon[37789]: pgmap v1047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:23 smithi044 ceph-mon[32404]: pgmap v1047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:23 smithi116 ceph-mon[37097]: pgmap v1047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:25.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:25.380+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:56:25.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:25.380+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:25.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:25.380+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:56:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:25 smithi044 ceph-mon[32404]: pgmap v1048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 930.570542 secs 2024-07-26T08:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:25 smithi116 ceph-mon[37097]: pgmap v1048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 930.570542 secs 2024-07-26T08:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:25 smithi171 ceph-mon[37789]: pgmap v1048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 930.570542 secs 2024-07-26T08:56:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:27 smithi044 ceph-mon[32404]: pgmap v1049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:27 smithi116 ceph-mon[37097]: pgmap v1049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:27 smithi171 ceph-mon[37789]: pgmap v1049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T08:56:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:29 smithi044 ceph-mon[32404]: pgmap v1050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:29 smithi116 ceph-mon[37097]: pgmap v1050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:29 smithi171 ceph-mon[37789]: pgmap v1050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 935.570700 secs 2024-07-26T08:56:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 935.570700 secs 2024-07-26T08:56:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 935.570700 secs 2024-07-26T08:56:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:31 smithi044 ceph-mon[32404]: pgmap v1051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:31 smithi116 ceph-mon[37097]: pgmap v1051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:31 smithi171 ceph-mon[37789]: pgmap v1051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:33 smithi044 ceph-mon[32404]: pgmap v1052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:33 smithi116 ceph-mon[37097]: pgmap v1052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:33 smithi171 ceph-mon[37789]: pgmap v1052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:35 smithi044 ceph-mon[32404]: pgmap v1053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 940.570841 secs 2024-07-26T08:56:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:35 smithi116 ceph-mon[37097]: pgmap v1053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 940.570841 secs 2024-07-26T08:56:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:35 smithi171 ceph-mon[37789]: pgmap v1053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 940.570841 secs 2024-07-26T08:56:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:37.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:37 smithi044 ceph-mon[32404]: pgmap v1054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:37.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:37.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:37.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:56:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:37 smithi116 ceph-mon[37097]: pgmap v1054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:56:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:37 smithi171 ceph-mon[37789]: pgmap v1054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:56:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:38 smithi044 ceph-mon[32404]: pgmap v1055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:38 smithi044 ceph-mon[32404]: pgmap v1056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:38 smithi116 ceph-mon[37097]: pgmap v1055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:38 smithi116 ceph-mon[37097]: pgmap v1056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:38 smithi171 ceph-mon[37789]: pgmap v1055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T08:56:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:38 smithi171 ceph-mon[37789]: pgmap v1056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:40.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:40.383+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:56:40.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:40.383+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:56:40.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:40.383+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:40 smithi044 ceph-mon[32404]: pgmap v1057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 945.571007 secs 2024-07-26T08:56:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:40 smithi116 ceph-mon[37097]: pgmap v1057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 945.571007 secs 2024-07-26T08:56:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:40 smithi171 ceph-mon[37789]: pgmap v1057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 945.571007 secs 2024-07-26T08:56:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:42 smithi044 ceph-mon[32404]: pgmap v1058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:42 smithi116 ceph-mon[37097]: pgmap v1058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:42 smithi171 ceph-mon[37789]: pgmap v1058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:56:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:44 smithi044 ceph-mon[32404]: pgmap v1059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:44 smithi116 ceph-mon[37097]: pgmap v1059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:44 smithi171 ceph-mon[37789]: pgmap v1059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:45.987 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 950.571154 secs 2024-07-26T08:56:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 950.571154 secs 2024-07-26T08:56:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 950.571154 secs 2024-07-26T08:56:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:46 smithi044 ceph-mon[32404]: pgmap v1060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:46 smithi116 ceph-mon[37097]: pgmap v1060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:46 smithi171 ceph-mon[37789]: pgmap v1060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:56:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:48 smithi044 ceph-mon[32404]: pgmap v1061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:48 smithi116 ceph-mon[37097]: pgmap v1061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:48 smithi171 ceph-mon[37789]: pgmap v1061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:56:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:50 smithi044 ceph-mon[32404]: pgmap v1062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 955.571257 secs 2024-07-26T08:56:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:50 smithi116 ceph-mon[37097]: pgmap v1062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 955.571257 secs 2024-07-26T08:56:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:50 smithi171 ceph-mon[37789]: pgmap v1062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:56:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 955.571257 secs 2024-07-26T08:56:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:52 smithi044 ceph-mon[32404]: pgmap v1063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:52 smithi116 ceph-mon[37097]: pgmap v1063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:52 smithi171 ceph-mon[37789]: pgmap v1063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:54 smithi044 ceph-mon[32404]: pgmap v1064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:54 smithi116 ceph-mon[37097]: pgmap v1064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:54 smithi171 ceph-mon[37789]: pgmap v1064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:56:55.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:55.391+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:56:55.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:55.391+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:56:55.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:56:55.391+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 960.571420 secs 2024-07-26T08:56:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 960.571420 secs 2024-07-26T08:56:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 960.571420 secs 2024-07-26T08:56:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:56 smithi044 ceph-mon[32404]: pgmap v1065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:56 smithi116 ceph-mon[37097]: pgmap v1065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:56 smithi171 ceph-mon[37789]: pgmap v1065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:56:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:56:58 smithi044 ceph-mon[32404]: pgmap v1066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:56:58 smithi116 ceph-mon[37097]: pgmap v1066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:56:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:56:58 smithi171 ceph-mon[37789]: pgmap v1066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:00 smithi044 ceph-mon[32404]: pgmap v1067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 965.571577 secs 2024-07-26T08:57:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:00 smithi116 ceph-mon[37097]: pgmap v1067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 965.571577 secs 2024-07-26T08:57:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:00 smithi171 ceph-mon[37789]: pgmap v1067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 965.571577 secs 2024-07-26T08:57:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:02 smithi044 ceph-mon[32404]: pgmap v1068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:02 smithi116 ceph-mon[37097]: pgmap v1068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:02 smithi171 ceph-mon[37789]: pgmap v1068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:04 smithi044 ceph-mon[32404]: pgmap v1069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:04 smithi116 ceph-mon[37097]: pgmap v1069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:04 smithi171 ceph-mon[37789]: pgmap v1069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 970.571739 secs 2024-07-26T08:57:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 970.571739 secs 2024-07-26T08:57:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 970.571739 secs 2024-07-26T08:57:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:06 smithi044 ceph-mon[32404]: pgmap v1070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:06 smithi116 ceph-mon[37097]: pgmap v1070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:06 smithi171 ceph-mon[37789]: pgmap v1070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:08 smithi116 ceph-mon[37097]: pgmap v1071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:08 smithi171 ceph-mon[37789]: pgmap v1071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:08 smithi044 ceph-mon[32404]: pgmap v1071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:10.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:10.394+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:57:10.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:10.394+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:57:10.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:10.394+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:10 smithi171 ceph-mon[37789]: pgmap v1072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 975.571834 secs 2024-07-26T08:57:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:10 smithi044 ceph-mon[32404]: pgmap v1072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 975.571834 secs 2024-07-26T08:57:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:10 smithi116 ceph-mon[37097]: pgmap v1072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 975.571834 secs 2024-07-26T08:57:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:12 smithi171 ceph-mon[37789]: pgmap v1073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:12 smithi044 ceph-mon[32404]: pgmap v1073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:12 smithi116 ceph-mon[37097]: pgmap v1073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:14 smithi171 ceph-mon[37789]: pgmap v1074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:14 smithi044 ceph-mon[32404]: pgmap v1074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:14 smithi116 ceph-mon[37097]: pgmap v1074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 980.571979 secs 2024-07-26T08:57:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 980.571979 secs 2024-07-26T08:57:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 980.571979 secs 2024-07-26T08:57:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:16 smithi171 ceph-mon[37789]: pgmap v1075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:16 smithi044 ceph-mon[32404]: pgmap v1075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:16 smithi116 ceph-mon[37097]: pgmap v1075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:18 smithi171 ceph-mon[37789]: pgmap v1076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:18 smithi044 ceph-mon[32404]: pgmap v1076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:18 smithi116 ceph-mon[37097]: pgmap v1076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:20 smithi171 ceph-mon[37789]: pgmap v1077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 985.572109 secs 2024-07-26T08:57:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:20 smithi044 ceph-mon[32404]: pgmap v1077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 985.572109 secs 2024-07-26T08:57:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:20 smithi116 ceph-mon[37097]: pgmap v1077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 985.572109 secs 2024-07-26T08:57:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:22 smithi171 ceph-mon[37789]: pgmap v1078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:22 smithi044 ceph-mon[32404]: pgmap v1078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:22 smithi116 ceph-mon[37097]: pgmap v1078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:24 smithi171 ceph-mon[37789]: pgmap v1079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:24 smithi044 ceph-mon[32404]: pgmap v1079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:24 smithi116 ceph-mon[37097]: pgmap v1079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:25.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:25.395+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:57:25.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:25.395+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:25.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:25.395+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:57:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 990.572272 secs 2024-07-26T08:57:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 990.572272 secs 2024-07-26T08:57:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 990.572272 secs 2024-07-26T08:57:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:26 smithi044 ceph-mon[32404]: pgmap v1080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:26 smithi116 ceph-mon[37097]: pgmap v1080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:26 smithi171 ceph-mon[37789]: pgmap v1080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:28 smithi044 ceph-mon[32404]: pgmap v1081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:28 smithi116 ceph-mon[37097]: pgmap v1081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:28 smithi171 ceph-mon[37789]: pgmap v1081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:29 smithi044 ceph-mon[32404]: pgmap v1082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:29 smithi116 ceph-mon[37097]: pgmap v1082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:29 smithi171 ceph-mon[37789]: pgmap v1082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 995.572411 secs 2024-07-26T08:57:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 995.572411 secs 2024-07-26T08:57:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 995.572411 secs 2024-07-26T08:57:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:31 smithi044 ceph-mon[32404]: pgmap v1083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:31 smithi116 ceph-mon[37097]: pgmap v1083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:31 smithi171 ceph-mon[37789]: pgmap v1083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:34 smithi044 ceph-mon[32404]: pgmap v1084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:34 smithi116 ceph-mon[37097]: pgmap v1084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:34 smithi171 ceph-mon[37789]: pgmap v1084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:57:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1000.572570 secs 2024-07-26T08:57:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1000.572570 secs 2024-07-26T08:57:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1000.572570 secs 2024-07-26T08:57:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:36 smithi116 ceph-mon[37097]: pgmap v1085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:36 smithi171 ceph-mon[37789]: pgmap v1085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:36 smithi044 ceph-mon[32404]: pgmap v1085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:38 smithi116 ceph-mon[37097]: pgmap v1086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:38.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:38 smithi044 ceph-mon[32404]: pgmap v1086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:38.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:38 smithi171 ceph-mon[37789]: pgmap v1086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:39 smithi171 ceph-mon[37789]: pgmap v1087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:57:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:39 smithi171 ceph-mon[37789]: pgmap v1088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:39.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:57:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:39 smithi044 ceph-mon[32404]: pgmap v1087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:57:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:39 smithi044 ceph-mon[32404]: pgmap v1088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:57:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:39 smithi116 ceph-mon[37097]: pgmap v1087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T08:57:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:39 smithi116 ceph-mon[37097]: pgmap v1088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:57:40.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:40.398+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:57:40.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:40.398+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:57:40.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:40.398+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1005.572722 secs 2024-07-26T08:57:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1005.572722 secs 2024-07-26T08:57:40.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1005.572722 secs 2024-07-26T08:57:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:41 smithi171 ceph-mon[37789]: pgmap v1089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:41 smithi044 ceph-mon[32404]: pgmap v1089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:41 smithi116 ceph-mon[37097]: pgmap v1089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:43 smithi044 ceph-mon[32404]: pgmap v1090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:43 smithi116 ceph-mon[37097]: pgmap v1090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:43 smithi171 ceph-mon[37789]: pgmap v1090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:44.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:44.837+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T08:57:44.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:44.837+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T08:57:44.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:44.838+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T08:57:44.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:44.838+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T08:57:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:45 smithi044 ceph-mon[32404]: pgmap v1091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1010.572877 secs 2024-07-26T08:57:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:45 smithi116 ceph-mon[37097]: pgmap v1091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1010.572877 secs 2024-07-26T08:57:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:45 smithi171 ceph-mon[37789]: pgmap v1091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:57:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1010.572877 secs 2024-07-26T08:57:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:47 smithi044 ceph-mon[32404]: pgmap v1092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:57:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:47 smithi116 ceph-mon[37097]: pgmap v1092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:57:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:47 smithi171 ceph-mon[37789]: pgmap v1092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T08:57:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:49 smithi044 ceph-mon[32404]: pgmap v1093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:57:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:49 smithi116 ceph-mon[37097]: pgmap v1093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:57:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:49 smithi171 ceph-mon[37789]: pgmap v1093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:57:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1015.573020 secs 2024-07-26T08:57:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1015.573020 secs 2024-07-26T08:57:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1015.573020 secs 2024-07-26T08:57:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:51 smithi044 ceph-mon[32404]: pgmap v1094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:51 smithi116 ceph-mon[37097]: pgmap v1094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:51 smithi171 ceph-mon[37789]: pgmap v1094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:57:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:53 smithi044 ceph-mon[32404]: pgmap v1095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:53 smithi116 ceph-mon[37097]: pgmap v1095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:53 smithi171 ceph-mon[37789]: pgmap v1095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:57:55.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:55.406+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:57:55.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:55.406+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:57:55.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:57:55.406+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:55 smithi044 ceph-mon[32404]: pgmap v1096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1020.573167 secs 2024-07-26T08:57:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:55 smithi116 ceph-mon[37097]: pgmap v1096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1020.573167 secs 2024-07-26T08:57:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:55 smithi171 ceph-mon[37789]: pgmap v1096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:57:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1020.573167 secs 2024-07-26T08:57:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:57 smithi044 ceph-mon[32404]: pgmap v1097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:57 smithi116 ceph-mon[37097]: pgmap v1097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:57:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:57 smithi171 ceph-mon[37789]: pgmap v1097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:57:59 smithi044 ceph-mon[32404]: pgmap v1098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:57:59 smithi116 ceph-mon[37097]: pgmap v1098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:57:59 smithi171 ceph-mon[37789]: pgmap v1098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1025.573321 secs 2024-07-26T08:58:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1025.573321 secs 2024-07-26T08:58:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1025.573321 secs 2024-07-26T08:58:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:01 smithi116 ceph-mon[37097]: pgmap v1099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:01 smithi171 ceph-mon[37789]: pgmap v1099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:01 smithi044 ceph-mon[32404]: pgmap v1099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:03 smithi171 ceph-mon[37789]: pgmap v1100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:03 smithi044 ceph-mon[32404]: pgmap v1100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:03 smithi116 ceph-mon[37097]: pgmap v1100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:05 smithi171 ceph-mon[37789]: pgmap v1101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1030.573474 secs 2024-07-26T08:58:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:05 smithi044 ceph-mon[32404]: pgmap v1101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1030.573474 secs 2024-07-26T08:58:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:05 smithi116 ceph-mon[37097]: pgmap v1101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1030.573474 secs 2024-07-26T08:58:08.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:07 smithi044 ceph-mon[32404]: pgmap v1102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:07 smithi171 ceph-mon[37789]: pgmap v1102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:07 smithi116 ceph-mon[37097]: pgmap v1102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:10.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:09 smithi116 ceph-mon[37097]: pgmap v1103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:09 smithi171 ceph-mon[37789]: pgmap v1103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:09 smithi044 ceph-mon[32404]: pgmap v1103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:10.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:10.409+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:58:10.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:10.409+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:58:10.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:10.409+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1035.573680 secs 2024-07-26T08:58:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1035.573680 secs 2024-07-26T08:58:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1035.573680 secs 2024-07-26T08:58:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:11 smithi171 ceph-mon[37789]: pgmap v1104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:11 smithi044 ceph-mon[32404]: pgmap v1104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:11 smithi116 ceph-mon[37097]: pgmap v1104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:13 smithi171 ceph-mon[37789]: pgmap v1105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:13 smithi044 ceph-mon[32404]: pgmap v1105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:13 smithi116 ceph-mon[37097]: pgmap v1105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:16.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:15 smithi044 ceph-mon[32404]: pgmap v1106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:16.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1040.573777 secs 2024-07-26T08:58:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:15 smithi171 ceph-mon[37789]: pgmap v1106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1040.573777 secs 2024-07-26T08:58:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:15 smithi116 ceph-mon[37097]: pgmap v1106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1040.573777 secs 2024-07-26T08:58:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:17 smithi171 ceph-mon[37789]: pgmap v1107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:17 smithi044 ceph-mon[32404]: pgmap v1107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:17 smithi116 ceph-mon[37097]: pgmap v1107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:19 smithi171 ceph-mon[37789]: pgmap v1108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:19 smithi044 ceph-mon[32404]: pgmap v1108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:19 smithi116 ceph-mon[37097]: pgmap v1108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1045.573944 secs 2024-07-26T08:58:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1045.573944 secs 2024-07-26T08:58:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1045.573944 secs 2024-07-26T08:58:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:21 smithi044 ceph-mon[32404]: pgmap v1109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:21 smithi116 ceph-mon[37097]: pgmap v1109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:21 smithi171 ceph-mon[37789]: pgmap v1109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:23 smithi044 ceph-mon[32404]: pgmap v1110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:23 smithi116 ceph-mon[37097]: pgmap v1110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:23 smithi171 ceph-mon[37789]: pgmap v1110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:25.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:25.417+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:58:25.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:25.417+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:58:25.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:25.417+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:25 smithi044 ceph-mon[32404]: pgmap v1111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1050.574104 secs 2024-07-26T08:58:26.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:25 smithi116 ceph-mon[37097]: pgmap v1111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:26.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1050.574104 secs 2024-07-26T08:58:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:25 smithi171 ceph-mon[37789]: pgmap v1111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1050.574104 secs 2024-07-26T08:58:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:27 smithi044 ceph-mon[32404]: pgmap v1112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:27 smithi116 ceph-mon[37097]: pgmap v1112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:27 smithi171 ceph-mon[37789]: pgmap v1112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:28 smithi044 ceph-mon[32404]: pgmap v1113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:28 smithi116 ceph-mon[37097]: pgmap v1113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:28 smithi171 ceph-mon[37789]: pgmap v1113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1055.574264 secs 2024-07-26T08:58:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1055.574264 secs 2024-07-26T08:58:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1055.574264 secs 2024-07-26T08:58:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:31 smithi171 ceph-mon[37789]: pgmap v1114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:31 smithi044 ceph-mon[32404]: pgmap v1114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:31 smithi116 ceph-mon[37097]: pgmap v1114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:33 smithi044 ceph-mon[32404]: pgmap v1115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:33 smithi116 ceph-mon[37097]: pgmap v1115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:33 smithi171 ceph-mon[37789]: pgmap v1115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:35 smithi044 ceph-mon[32404]: pgmap v1116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1060.574409 secs 2024-07-26T08:58:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:35 smithi116 ceph-mon[37097]: pgmap v1116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1060.574409 secs 2024-07-26T08:58:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:35 smithi171 ceph-mon[37789]: pgmap v1116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:58:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1060.574409 secs 2024-07-26T08:58:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:37 smithi044 ceph-mon[32404]: pgmap v1117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:37 smithi116 ceph-mon[37097]: pgmap v1117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:37 smithi171 ceph-mon[37789]: pgmap v1117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T08:58:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:39 smithi044 ceph-mon[32404]: pgmap v1118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:39 smithi116 ceph-mon[37097]: pgmap v1118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:39 smithi171 ceph-mon[37789]: pgmap v1118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T08:58:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:40.421+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:58:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:40.421+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:58:40.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:40.421+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:58:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1065.574518 secs 2024-07-26T08:58:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:58:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1065.574518 secs 2024-07-26T08:58:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:58:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1065.574518 secs 2024-07-26T08:58:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:41 smithi044 ceph-mon[32404]: pgmap v1119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:58:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:41 smithi044 ceph-mon[32404]: pgmap v1120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:41 smithi116 ceph-mon[37097]: pgmap v1119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:58:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:41 smithi116 ceph-mon[37097]: pgmap v1120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:41 smithi171 ceph-mon[37789]: pgmap v1119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T08:58:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:41 smithi171 ceph-mon[37789]: pgmap v1120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:43 smithi116 ceph-mon[37097]: pgmap v1121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:43 smithi171 ceph-mon[37789]: pgmap v1121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:43 smithi044 ceph-mon[32404]: pgmap v1121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:45 smithi171 ceph-mon[37789]: pgmap v1122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1070.574675 secs 2024-07-26T08:58:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:45 smithi044 ceph-mon[32404]: pgmap v1122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1070.574675 secs 2024-07-26T08:58:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:45 smithi116 ceph-mon[37097]: pgmap v1122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:58:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1070.574675 secs 2024-07-26T08:58:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:47 smithi171 ceph-mon[37789]: pgmap v1123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:58:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:47 smithi044 ceph-mon[32404]: pgmap v1123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:58:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:47 smithi116 ceph-mon[37097]: pgmap v1123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T08:58:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:49 smithi171 ceph-mon[37789]: pgmap v1124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:58:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:49 smithi044 ceph-mon[32404]: pgmap v1124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:58:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:49 smithi116 ceph-mon[37097]: pgmap v1124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T08:58:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1075.574832 secs 2024-07-26T08:58:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1075.574832 secs 2024-07-26T08:58:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1075.574832 secs 2024-07-26T08:58:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:51 smithi171 ceph-mon[37789]: pgmap v1125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:58:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:51 smithi044 ceph-mon[32404]: pgmap v1125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:58:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:51 smithi116 ceph-mon[37097]: pgmap v1125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:58:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:53 smithi171 ceph-mon[37789]: pgmap v1126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:53 smithi044 ceph-mon[32404]: pgmap v1126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:53 smithi116 ceph-mon[37097]: pgmap v1126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:55.425+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:58:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:55.425+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:58:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:58:55.425+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:55 smithi171 ceph-mon[37789]: pgmap v1127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1080.574932 secs 2024-07-26T08:58:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:55 smithi044 ceph-mon[32404]: pgmap v1127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1080.574932 secs 2024-07-26T08:58:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:55 smithi116 ceph-mon[37097]: pgmap v1127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:58:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1080.574932 secs 2024-07-26T08:58:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:57 smithi171 ceph-mon[37789]: pgmap v1128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:57 smithi044 ceph-mon[32404]: pgmap v1128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:58:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:57 smithi116 ceph-mon[37097]: pgmap v1128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:58:59 smithi171 ceph-mon[37789]: pgmap v1129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:58:59 smithi044 ceph-mon[32404]: pgmap v1129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:58:59 smithi116 ceph-mon[37097]: pgmap v1129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1085.575016 secs 2024-07-26T08:59:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1085.575016 secs 2024-07-26T08:59:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1085.575016 secs 2024-07-26T08:59:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:01 smithi044 ceph-mon[32404]: pgmap v1130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:01 smithi116 ceph-mon[37097]: pgmap v1130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:01 smithi171 ceph-mon[37789]: pgmap v1130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:03 smithi044 ceph-mon[32404]: pgmap v1131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:03 smithi116 ceph-mon[37097]: pgmap v1131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:03 smithi171 ceph-mon[37789]: pgmap v1131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:05 smithi044 ceph-mon[32404]: pgmap v1132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1090.575154 secs 2024-07-26T08:59:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:05 smithi116 ceph-mon[37097]: pgmap v1132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1090.575154 secs 2024-07-26T08:59:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:05 smithi171 ceph-mon[37789]: pgmap v1132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1090.575154 secs 2024-07-26T08:59:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:07 smithi044 ceph-mon[32404]: pgmap v1133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:07 smithi116 ceph-mon[37097]: pgmap v1133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:07 smithi171 ceph-mon[37789]: pgmap v1133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:09 smithi044 ceph-mon[32404]: pgmap v1134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:09 smithi116 ceph-mon[37097]: pgmap v1134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:09 smithi171 ceph-mon[37789]: pgmap v1134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:10.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:10.428+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:59:10.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:10.428+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:59:10.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:10.428+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1095.575294 secs 2024-07-26T08:59:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1095.575294 secs 2024-07-26T08:59:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1095.575294 secs 2024-07-26T08:59:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:11 smithi044 ceph-mon[32404]: pgmap v1135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:11 smithi116 ceph-mon[37097]: pgmap v1135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:11 smithi171 ceph-mon[37789]: pgmap v1135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:12 smithi044 ceph-mon[32404]: pgmap v1136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:12 smithi116 ceph-mon[37097]: pgmap v1136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:12 smithi171 ceph-mon[37789]: pgmap v1136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:14.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:14.853+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T08:59:14.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:14.853+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T08:59:14.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:14.856+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T08:59:14.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:14.857+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T08:59:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:14 smithi044 ceph-mon[32404]: pgmap v1137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:14 smithi116 ceph-mon[37097]: pgmap v1137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:14 smithi171 ceph-mon[37789]: pgmap v1137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1100.575472 secs 2024-07-26T08:59:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1100.575472 secs 2024-07-26T08:59:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1100.575472 secs 2024-07-26T08:59:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:17 smithi044 ceph-mon[32404]: pgmap v1138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:17 smithi116 ceph-mon[37097]: pgmap v1138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:17 smithi171 ceph-mon[37789]: pgmap v1138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:19.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:19 smithi044 ceph-mon[32404]: pgmap v1139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:19 smithi116 ceph-mon[37097]: pgmap v1139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:19 smithi171 ceph-mon[37789]: pgmap v1139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:21 smithi171 ceph-mon[37789]: pgmap v1140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1105.575607 secs 2024-07-26T08:59:21.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:21 smithi044 ceph-mon[32404]: pgmap v1140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:21.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1105.575607 secs 2024-07-26T08:59:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:21 smithi116 ceph-mon[37097]: pgmap v1140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1105.575607 secs 2024-07-26T08:59:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:23 smithi171 ceph-mon[37789]: pgmap v1141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:23 smithi044 ceph-mon[32404]: pgmap v1141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:23 smithi116 ceph-mon[37097]: pgmap v1141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:25 smithi171 ceph-mon[37789]: pgmap v1142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:25.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:25.435+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:59:25.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:25.435+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:59:25.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:25.435+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:25.438 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:25 smithi044 ceph-mon[32404]: pgmap v1142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:25 smithi116 ceph-mon[37097]: pgmap v1142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1110.575721 secs 2024-07-26T08:59:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1110.575721 secs 2024-07-26T08:59:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1110.575721 secs 2024-07-26T08:59:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:27 smithi171 ceph-mon[37789]: pgmap v1143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:27 smithi044 ceph-mon[32404]: pgmap v1143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:27 smithi116 ceph-mon[37097]: pgmap v1143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:29 smithi171 ceph-mon[37789]: pgmap v1144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:29.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:29 smithi044 ceph-mon[32404]: pgmap v1144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:29 smithi116 ceph-mon[37097]: pgmap v1144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:31 smithi171 ceph-mon[37789]: pgmap v1145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1115.575876 secs 2024-07-26T08:59:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:31 smithi044 ceph-mon[32404]: pgmap v1145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1115.575876 secs 2024-07-26T08:59:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:31 smithi116 ceph-mon[37097]: pgmap v1145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1115.575876 secs 2024-07-26T08:59:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:33 smithi171 ceph-mon[37789]: pgmap v1146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:33 smithi044 ceph-mon[32404]: pgmap v1146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:33 smithi116 ceph-mon[37097]: pgmap v1146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:35 smithi171 ceph-mon[37789]: pgmap v1147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:35 smithi044 ceph-mon[32404]: pgmap v1147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:35 smithi116 ceph-mon[37097]: pgmap v1147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1120.575977 secs 2024-07-26T08:59:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1120.575977 secs 2024-07-26T08:59:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1120.575977 secs 2024-07-26T08:59:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:37 smithi044 ceph-mon[32404]: pgmap v1148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:37 smithi116 ceph-mon[37097]: pgmap v1148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:37 smithi171 ceph-mon[37789]: pgmap v1148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:39 smithi044 ceph-mon[32404]: pgmap v1149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:59:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:39 smithi116 ceph-mon[37097]: pgmap v1149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:59:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:39 smithi171 ceph-mon[37789]: pgmap v1149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T08:59:40.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:40.438+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:59:40.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:40.438+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:59:40.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:40.438+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:41 smithi044 ceph-mon[32404]: pgmap v1150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1125.576052 secs 2024-07-26T08:59:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:41 smithi116 ceph-mon[37097]: pgmap v1150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1125.576052 secs 2024-07-26T08:59:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:41 smithi171 ceph-mon[37789]: pgmap v1150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T08:59:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1125.576052 secs 2024-07-26T08:59:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:42 smithi044 ceph-mon[32404]: pgmap v1151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:59:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:42 smithi044 ceph-mon[32404]: pgmap v1152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T08:59:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:59:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:42 smithi116 ceph-mon[37097]: pgmap v1151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:59:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:42 smithi116 ceph-mon[37097]: pgmap v1152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T08:59:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:59:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:42 smithi171 ceph-mon[37789]: pgmap v1151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T08:59:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:42 smithi171 ceph-mon[37789]: pgmap v1152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T08:59:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T08:59:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:44 smithi171 ceph-mon[37789]: pgmap v1153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:44 smithi044 ceph-mon[32404]: pgmap v1153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:44 smithi116 ceph-mon[37097]: pgmap v1153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:46 smithi171 ceph-mon[37789]: pgmap v1154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1130.576209 secs 2024-07-26T08:59:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:46 smithi044 ceph-mon[32404]: pgmap v1154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1130.576209 secs 2024-07-26T08:59:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:46 smithi116 ceph-mon[37097]: pgmap v1154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T08:59:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1130.576209 secs 2024-07-26T08:59:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:48 smithi171 ceph-mon[37789]: pgmap v1155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:59:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:48 smithi044 ceph-mon[32404]: pgmap v1155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:59:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:48 smithi116 ceph-mon[37097]: pgmap v1155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:59:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:50 smithi044 ceph-mon[32404]: pgmap v1156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:59:50.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:50 smithi171 ceph-mon[37789]: pgmap v1156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:59:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:50 smithi116 ceph-mon[37097]: pgmap v1156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T08:59:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1135.576357 secs 2024-07-26T08:59:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1135.576357 secs 2024-07-26T08:59:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1135.576357 secs 2024-07-26T08:59:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:52 smithi171 ceph-mon[37789]: pgmap v1157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:59:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:52 smithi044 ceph-mon[32404]: pgmap v1157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:59:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:52 smithi116 ceph-mon[37097]: pgmap v1157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T08:59:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:54 smithi171 ceph-mon[37789]: pgmap v1158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:59:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:54 smithi044 ceph-mon[32404]: pgmap v1158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:59:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:54 smithi116 ceph-mon[37097]: pgmap v1158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T08:59:55.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:55.446+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T08:59:55.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:55.446+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T08:59:55.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T08:59:55.446+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:56 smithi171 ceph-mon[37789]: pgmap v1159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1140.576467 secs 2024-07-26T08:59:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:56 smithi044 ceph-mon[32404]: pgmap v1159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1140.576467 secs 2024-07-26T08:59:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:56 smithi116 ceph-mon[37097]: pgmap v1159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T08:59:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1140.576467 secs 2024-07-26T08:59:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 08:59:58 smithi171 ceph-mon[37789]: pgmap v1160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 08:59:58 smithi044 ceph-mon[32404]: pgmap v1160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T08:59:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 08:59:58 smithi116 ceph-mon[37097]: pgmap v1160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: pgmap v1161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:00:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T09:00:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: daemon mds.j on smithi044 is in error state 2024-07-26T09:00:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: daemon mds.a on smithi044 is in error state 2024-07-26T09:00:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: daemon mds.b on smithi116 is in error state 2024-07-26T09:00:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: daemon mds.f on smithi171 is in error state 2024-07-26T09:00:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T09:00:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:00 smithi171 ceph-mon[37789]: mds.g(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T09:00:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: pgmap v1161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:00:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T09:00:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: daemon mds.j on smithi044 is in error state 2024-07-26T09:00:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: daemon mds.a on smithi044 is in error state 2024-07-26T09:00:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: daemon mds.b on smithi116 is in error state 2024-07-26T09:00:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: daemon mds.f on smithi171 is in error state 2024-07-26T09:00:00.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T09:00:00.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:00 smithi044 ceph-mon[32404]: mds.g(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T09:00:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: pgmap v1161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:00:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T09:00:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: daemon mds.j on smithi044 is in error state 2024-07-26T09:00:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: daemon mds.a on smithi044 is in error state 2024-07-26T09:00:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: daemon mds.b on smithi116 is in error state 2024-07-26T09:00:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: daemon mds.f on smithi171 is in error state 2024-07-26T09:00:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T09:00:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:00 smithi116 ceph-mon[37097]: mds.g(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T09:00:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1145.576681 secs 2024-07-26T09:00:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1145.576681 secs 2024-07-26T09:00:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1145.576681 secs 2024-07-26T09:00:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:02 smithi044 ceph-mon[32404]: pgmap v1162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:00:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:02 smithi116 ceph-mon[37097]: pgmap v1162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:00:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:02 smithi171 ceph-mon[37789]: pgmap v1162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:00:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:04 smithi044 ceph-mon[32404]: pgmap v1163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:04 smithi116 ceph-mon[37097]: pgmap v1163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:04 smithi171 ceph-mon[37789]: pgmap v1163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1150.576841 secs 2024-07-26T09:00:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:06 smithi044 ceph-mon[32404]: pgmap v1164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1150.576841 secs 2024-07-26T09:00:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:06 smithi116 ceph-mon[37097]: pgmap v1164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1150.576841 secs 2024-07-26T09:00:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:06 smithi171 ceph-mon[37789]: pgmap v1164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:08 smithi044 ceph-mon[32404]: pgmap v1165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:08 smithi116 ceph-mon[37097]: pgmap v1165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:08 smithi171 ceph-mon[37789]: pgmap v1165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:10.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:10.450+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:00:10.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:10.450+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:10.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:10.450+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:00:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:10 smithi044 ceph-mon[32404]: pgmap v1166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:00:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:10 smithi116 ceph-mon[37097]: pgmap v1166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:00:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:10 smithi171 ceph-mon[37789]: pgmap v1166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:00:11.675 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1155.576998 secs 2024-07-26T09:00:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1155.576998 secs 2024-07-26T09:00:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1155.576998 secs 2024-07-26T09:00:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:12 smithi044 ceph-mon[32404]: pgmap v1167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:00:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:12 smithi116 ceph-mon[37097]: pgmap v1167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:00:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:12 smithi171 ceph-mon[37789]: pgmap v1167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:00:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:14 smithi044 ceph-mon[32404]: pgmap v1168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:00:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:14 smithi116 ceph-mon[37097]: pgmap v1168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:00:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:14 smithi171 ceph-mon[37789]: pgmap v1168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:00:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1160.577134 secs 2024-07-26T09:00:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:16 smithi044 ceph-mon[32404]: pgmap v1169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:00:16.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1160.577134 secs 2024-07-26T09:00:16.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:16 smithi116 ceph-mon[37097]: pgmap v1169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:00:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1160.577134 secs 2024-07-26T09:00:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:16 smithi171 ceph-mon[37789]: pgmap v1169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:00:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:18 smithi044 ceph-mon[32404]: pgmap v1170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:18 smithi116 ceph-mon[37097]: pgmap v1170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:18 smithi171 ceph-mon[37789]: pgmap v1170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:00:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:20 smithi044 ceph-mon[32404]: pgmap v1171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:20 smithi116 ceph-mon[37097]: pgmap v1171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:20 smithi171 ceph-mon[37789]: pgmap v1171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1165.577276 secs 2024-07-26T09:00:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1165.577276 secs 2024-07-26T09:00:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1165.577276 secs 2024-07-26T09:00:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:22 smithi044 ceph-mon[32404]: pgmap v1172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:22 smithi116 ceph-mon[37097]: pgmap v1172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:22 smithi171 ceph-mon[37789]: pgmap v1172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:24 smithi044 ceph-mon[32404]: pgmap v1173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:24 smithi116 ceph-mon[37097]: pgmap v1173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:24 smithi171 ceph-mon[37789]: pgmap v1173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:25.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:25.457+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:00:25.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:25.457+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:00:25.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:25.457+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1170.577435 secs 2024-07-26T09:00:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:26 smithi044 ceph-mon[32404]: pgmap v1174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1170.577435 secs 2024-07-26T09:00:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:26 smithi116 ceph-mon[37097]: pgmap v1174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1170.577435 secs 2024-07-26T09:00:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:26 smithi171 ceph-mon[37789]: pgmap v1174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:28 smithi044 ceph-mon[32404]: pgmap v1175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:28 smithi116 ceph-mon[37097]: pgmap v1175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:28 smithi171 ceph-mon[37789]: pgmap v1175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:30 smithi044 ceph-mon[32404]: pgmap v1176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:30 smithi116 ceph-mon[37097]: pgmap v1176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:30 smithi171 ceph-mon[37789]: pgmap v1176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1175.577583 secs 2024-07-26T09:00:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1175.577583 secs 2024-07-26T09:00:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1175.577583 secs 2024-07-26T09:00:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:32 smithi044 ceph-mon[32404]: pgmap v1177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:32 smithi116 ceph-mon[37097]: pgmap v1177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:32.826 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:32 smithi171 ceph-mon[37789]: pgmap v1177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:34 smithi044 ceph-mon[32404]: pgmap v1178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:34.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:34 smithi116 ceph-mon[37097]: pgmap v1178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:34 smithi171 ceph-mon[37789]: pgmap v1178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:00:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1180.577744 secs 2024-07-26T09:00:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:36 smithi044 ceph-mon[32404]: pgmap v1179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1180.577744 secs 2024-07-26T09:00:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:36 smithi116 ceph-mon[37097]: pgmap v1179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1180.577744 secs 2024-07-26T09:00:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:36 smithi171 ceph-mon[37789]: pgmap v1179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:38 smithi044 ceph-mon[32404]: pgmap v1180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:38 smithi116 ceph-mon[37097]: pgmap v1180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:38 smithi171 ceph-mon[37789]: pgmap v1180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:40.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:40.460+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:00:40.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:40.460+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:00:40.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:40.460+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:40 smithi044 ceph-mon[32404]: pgmap v1181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:40 smithi116 ceph-mon[37097]: pgmap v1181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:40 smithi171 ceph-mon[37789]: pgmap v1181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:00:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1185.577900 secs 2024-07-26T09:00:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1185.577900 secs 2024-07-26T09:00:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1185.577900 secs 2024-07-26T09:00:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:42 smithi044 ceph-mon[32404]: pgmap v1182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:00:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:42 smithi116 ceph-mon[37097]: pgmap v1182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:00:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:42 smithi171 ceph-mon[37789]: pgmap v1182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:00:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:43 smithi044 ceph-mon[32404]: pgmap v1183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:00:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:43 smithi044 ceph-mon[32404]: pgmap v1184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:43 smithi116 ceph-mon[37097]: pgmap v1183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:00:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:43 smithi116 ceph-mon[37097]: pgmap v1184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:43 smithi171 ceph-mon[37789]: pgmap v1183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:00:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:43 smithi171 ceph-mon[37789]: pgmap v1184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:45 smithi044 ceph-mon[32404]: pgmap v1185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1190.578067 secs 2024-07-26T09:00:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:45 smithi116 ceph-mon[37097]: pgmap v1185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1190.578067 secs 2024-07-26T09:00:45.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:45 smithi171 ceph-mon[37789]: pgmap v1185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:45.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1190.578067 secs 2024-07-26T09:00:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:47 smithi044 ceph-mon[32404]: pgmap v1186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:47 smithi116 ceph-mon[37097]: pgmap v1186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:47 smithi171 ceph-mon[37789]: pgmap v1186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:49 smithi044 ceph-mon[32404]: pgmap v1187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:49 smithi116 ceph-mon[37097]: pgmap v1187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:49 smithi171 ceph-mon[37789]: pgmap v1187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1195.578196 secs 2024-07-26T09:00:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1195.578196 secs 2024-07-26T09:00:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1195.578196 secs 2024-07-26T09:00:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:51 smithi171 ceph-mon[37789]: pgmap v1188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:52.002 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:51 smithi044 ceph-mon[32404]: pgmap v1188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:51 smithi116 ceph-mon[37097]: pgmap v1188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:00:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:53 smithi171 ceph-mon[37789]: pgmap v1189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:53 smithi044 ceph-mon[32404]: pgmap v1189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:53 smithi116 ceph-mon[37097]: pgmap v1189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:00:55.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:55.468+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:00:55.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:55.468+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:00:55.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:00:55.468+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:55 smithi044 ceph-mon[32404]: pgmap v1190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1200.578302 secs 2024-07-26T09:00:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:55 smithi171 ceph-mon[37789]: pgmap v1190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1200.578302 secs 2024-07-26T09:00:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:55 smithi116 ceph-mon[37097]: pgmap v1190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:00:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1200.578302 secs 2024-07-26T09:00:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:57 smithi171 ceph-mon[37789]: pgmap v1191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:57 smithi044 ceph-mon[32404]: pgmap v1191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:57 smithi116 ceph-mon[37097]: pgmap v1191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:00:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:00:59 smithi171 ceph-mon[37789]: pgmap v1192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:01:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:00:59 smithi044 ceph-mon[32404]: pgmap v1192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:01:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:00:59 smithi116 ceph-mon[37097]: pgmap v1192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:01:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1205.578436 secs 2024-07-26T09:01:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1205.578436 secs 2024-07-26T09:01:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1205.578436 secs 2024-07-26T09:01:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:01 smithi116 ceph-mon[37097]: pgmap v1193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:01 smithi171 ceph-mon[37789]: pgmap v1193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:01.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:01 smithi044 ceph-mon[32404]: pgmap v1193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:03 smithi171 ceph-mon[37789]: pgmap v1194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:03 smithi044 ceph-mon[32404]: pgmap v1194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:03 smithi116 ceph-mon[37097]: pgmap v1194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:05 smithi171 ceph-mon[37789]: pgmap v1195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1210.578580 secs 2024-07-26T09:01:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:05 smithi044 ceph-mon[32404]: pgmap v1195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1210.578580 secs 2024-07-26T09:01:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:05 smithi116 ceph-mon[37097]: pgmap v1195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1210.578580 secs 2024-07-26T09:01:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:07 smithi171 ceph-mon[37789]: pgmap v1196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:07.874 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:07 smithi044 ceph-mon[32404]: pgmap v1196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:07 smithi116 ceph-mon[37097]: pgmap v1196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:09 smithi044 ceph-mon[32404]: pgmap v1197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:01:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:09 smithi116 ceph-mon[37097]: pgmap v1197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:01:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:09 smithi171 ceph-mon[37789]: pgmap v1197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:01:10.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:10.472+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:01:10.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:10.472+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:10.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:10.472+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:01:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1215.578734 secs 2024-07-26T09:01:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1215.578734 secs 2024-07-26T09:01:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1215.578734 secs 2024-07-26T09:01:11.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:11 smithi116 ceph-mon[37097]: pgmap v1198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:11 smithi044 ceph-mon[32404]: pgmap v1198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:11 smithi171 ceph-mon[37789]: pgmap v1198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:13 smithi044 ceph-mon[32404]: pgmap v1199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:13 smithi116 ceph-mon[37097]: pgmap v1199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:13 smithi171 ceph-mon[37789]: pgmap v1199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:15 smithi044 ceph-mon[32404]: pgmap v1200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1220.578886 secs 2024-07-26T09:01:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:15 smithi116 ceph-mon[37097]: pgmap v1200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1220.578886 secs 2024-07-26T09:01:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:15 smithi171 ceph-mon[37789]: pgmap v1200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1220.578886 secs 2024-07-26T09:01:17.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:17 smithi171 ceph-mon[37789]: pgmap v1201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:17 smithi044 ceph-mon[32404]: pgmap v1201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:17 smithi116 ceph-mon[37097]: pgmap v1201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:19 smithi044 ceph-mon[32404]: pgmap v1202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:01:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:19 smithi116 ceph-mon[37097]: pgmap v1202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:01:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:19 smithi171 ceph-mon[37789]: pgmap v1202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:01:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1225.579028 secs 2024-07-26T09:01:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1225.579028 secs 2024-07-26T09:01:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1225.579028 secs 2024-07-26T09:01:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:21 smithi044 ceph-mon[32404]: pgmap v1203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:21 smithi116 ceph-mon[37097]: pgmap v1203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:21 smithi171 ceph-mon[37789]: pgmap v1203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:23 smithi044 ceph-mon[32404]: pgmap v1204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:23 smithi116 ceph-mon[37097]: pgmap v1204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:23 smithi171 ceph-mon[37789]: pgmap v1204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:01:25.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:25.480+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:01:25.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:25.480+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:25.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:25.480+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:01:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:25 smithi044 ceph-mon[32404]: pgmap v1205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1230.579188 secs 2024-07-26T09:01:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:25 smithi116 ceph-mon[37097]: pgmap v1205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1230.579188 secs 2024-07-26T09:01:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:25 smithi171 ceph-mon[37789]: pgmap v1205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:01:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1230.579188 secs 2024-07-26T09:01:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:27 smithi044 ceph-mon[32404]: pgmap v1206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:27 smithi116 ceph-mon[37097]: pgmap v1206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:27 smithi171 ceph-mon[37789]: pgmap v1206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:29 smithi044 ceph-mon[32404]: pgmap v1207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:01:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:29 smithi116 ceph-mon[37097]: pgmap v1207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:01:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:29 smithi171 ceph-mon[37789]: pgmap v1207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:01:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1235.579350 secs 2024-07-26T09:01:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1235.579350 secs 2024-07-26T09:01:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1235.579350 secs 2024-07-26T09:01:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:31 smithi044 ceph-mon[32404]: pgmap v1208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:31 smithi116 ceph-mon[37097]: pgmap v1208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:31 smithi171 ceph-mon[37789]: pgmap v1208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:33 smithi044 ceph-mon[32404]: pgmap v1209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:33 smithi116 ceph-mon[37097]: pgmap v1209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:33 smithi171 ceph-mon[37789]: pgmap v1209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:35 smithi044 ceph-mon[32404]: pgmap v1210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1240.579482 secs 2024-07-26T09:01:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:35 smithi116 ceph-mon[37097]: pgmap v1210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1240.579482 secs 2024-07-26T09:01:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:35 smithi171 ceph-mon[37789]: pgmap v1210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:01:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1240.579482 secs 2024-07-26T09:01:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:37 smithi044 ceph-mon[32404]: pgmap v1211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:01:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:37 smithi116 ceph-mon[37097]: pgmap v1211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:01:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:37 smithi171 ceph-mon[37789]: pgmap v1211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:01:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:39 smithi044 ceph-mon[32404]: pgmap v1212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:39 smithi116 ceph-mon[37097]: pgmap v1212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:39 smithi171 ceph-mon[37789]: pgmap v1212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:40.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:40.483+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:01:40.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:40.483+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:40.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:40.483+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:01:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1245.579658 secs 2024-07-26T09:01:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1245.579658 secs 2024-07-26T09:01:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1245.579658 secs 2024-07-26T09:01:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:41 smithi044 ceph-mon[32404]: pgmap v1213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:41 smithi116 ceph-mon[37097]: pgmap v1213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:41 smithi171 ceph-mon[37789]: pgmap v1213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:01:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:43.818 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:43 smithi116 ceph-mon[37097]: pgmap v1214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:43 smithi044 ceph-mon[32404]: pgmap v1214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:43 smithi171 ceph-mon[37789]: pgmap v1214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:01:45.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:45 smithi116 ceph-mon[37097]: pgmap v1215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:45.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1250.579756 secs 2024-07-26T09:01:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:45 smithi044 ceph-mon[32404]: pgmap v1215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1250.579756 secs 2024-07-26T09:01:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:45 smithi171 ceph-mon[37789]: pgmap v1215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:01:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1250.579756 secs 2024-07-26T09:01:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:47 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:47 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:47 smithi044 ceph-mon[32404]: pgmap v1216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:47 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:47 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:47 smithi116 ceph-mon[37097]: pgmap v1216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:47 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:47 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:47 smithi171 ceph-mon[37789]: pgmap v1216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:01:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: pgmap v1217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:49.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:49 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: pgmap v1217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:01:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:49 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: pgmap v1217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:01:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:49 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:50 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:01:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:50 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:01:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:50 smithi171 ceph-mon[37789]: pgmap v1218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:01:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:50 smithi171 ceph-mon[37789]: pgmap v1219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1255.579919 secs 2024-07-26T09:01:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:50 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:01:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:50 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:01:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:50 smithi044 ceph-mon[32404]: pgmap v1218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:01:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:50 smithi044 ceph-mon[32404]: pgmap v1219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1255.579919 secs 2024-07-26T09:01:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:50 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:01:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:50 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:01:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:50 smithi116 ceph-mon[37097]: pgmap v1218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:01:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:50 smithi116 ceph-mon[37097]: pgmap v1219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:01:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1255.579919 secs 2024-07-26T09:01:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:52 smithi171 ceph-mon[37789]: pgmap v1220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:01:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:52 smithi044 ceph-mon[32404]: pgmap v1220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:01:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:52 smithi116 ceph-mon[37097]: pgmap v1220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:01:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:54 smithi171 ceph-mon[37789]: pgmap v1221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:54 smithi044 ceph-mon[32404]: pgmap v1221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:54 smithi116 ceph-mon[37097]: pgmap v1221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:01:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:55.490+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:01:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:55.490+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:01:55.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:01:55.490+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1260.580069 secs 2024-07-26T09:01:55.997 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1260.580069 secs 2024-07-26T09:01:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1260.580069 secs 2024-07-26T09:01:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:56 smithi116 ceph-mon[37097]: pgmap v1222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:01:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:56 smithi171 ceph-mon[37789]: pgmap v1222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:01:56.942 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:56 smithi044 ceph-mon[32404]: pgmap v1222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:01:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:01:58 smithi171 ceph-mon[37789]: pgmap v1223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:01:58 smithi044 ceph-mon[32404]: pgmap v1223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:01:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:01:58 smithi116 ceph-mon[37097]: pgmap v1223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:00 smithi171 ceph-mon[37789]: pgmap v1224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:02:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1265.580222 secs 2024-07-26T09:02:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:00 smithi044 ceph-mon[32404]: pgmap v1224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:02:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1265.580222 secs 2024-07-26T09:02:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:00 smithi116 ceph-mon[37097]: pgmap v1224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:02:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1265.580222 secs 2024-07-26T09:02:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:02 smithi171 ceph-mon[37789]: pgmap v1225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:02 smithi044 ceph-mon[32404]: pgmap v1225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:02 smithi116 ceph-mon[37097]: pgmap v1225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:04 smithi044 ceph-mon[32404]: pgmap v1226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:04 smithi116 ceph-mon[37097]: pgmap v1226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:04 smithi171 ceph-mon[37789]: pgmap v1226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1270.580351 secs 2024-07-26T09:02:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1270.580351 secs 2024-07-26T09:02:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1270.580351 secs 2024-07-26T09:02:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:06 smithi044 ceph-mon[32404]: pgmap v1227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:06 smithi116 ceph-mon[37097]: pgmap v1227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:06 smithi171 ceph-mon[37789]: pgmap v1227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:08 smithi044 ceph-mon[32404]: pgmap v1228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:08 smithi116 ceph-mon[37097]: pgmap v1228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:08 smithi171 ceph-mon[37789]: pgmap v1228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:10.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:10.494+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:02:10.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:10.494+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:10.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:10.494+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:02:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:10 smithi044 ceph-mon[32404]: pgmap v1229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1275.580490 secs 2024-07-26T09:02:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:10 smithi116 ceph-mon[37097]: pgmap v1229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1275.580490 secs 2024-07-26T09:02:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:10 smithi171 ceph-mon[37789]: pgmap v1229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1275.580490 secs 2024-07-26T09:02:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:12 smithi044 ceph-mon[32404]: pgmap v1230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:12 smithi116 ceph-mon[37097]: pgmap v1230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:12 smithi171 ceph-mon[37789]: pgmap v1230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:14.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:14.885+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:02:14.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:14 smithi044 ceph-mon[32404]: pgmap v1231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:14.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:14.885+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:02:14.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:14.885+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:02:14.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:14.886+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:02:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:14 smithi116 ceph-mon[37097]: pgmap v1231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:14 smithi171 ceph-mon[37789]: pgmap v1231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1280.580667 secs 2024-07-26T09:02:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1280.580667 secs 2024-07-26T09:02:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1280.580667 secs 2024-07-26T09:02:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:16 smithi044 ceph-mon[32404]: pgmap v1232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:16 smithi116 ceph-mon[37097]: pgmap v1232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:16 smithi171 ceph-mon[37789]: pgmap v1232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:18 smithi044 ceph-mon[32404]: pgmap v1233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:18 smithi116 ceph-mon[37097]: pgmap v1233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:18 smithi171 ceph-mon[37789]: pgmap v1233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:20 smithi044 ceph-mon[32404]: pgmap v1234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1285.580759 secs 2024-07-26T09:02:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:20 smithi116 ceph-mon[37097]: pgmap v1234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1285.580759 secs 2024-07-26T09:02:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:20 smithi171 ceph-mon[37789]: pgmap v1234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1285.580759 secs 2024-07-26T09:02:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:22 smithi044 ceph-mon[32404]: pgmap v1235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:22 smithi116 ceph-mon[37097]: pgmap v1235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:22 smithi171 ceph-mon[37789]: pgmap v1235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:24 smithi044 ceph-mon[32404]: pgmap v1236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:24 smithi116 ceph-mon[37097]: pgmap v1236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:24 smithi171 ceph-mon[37789]: pgmap v1236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:25.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:25.503+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:02:25.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:25.503+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:02:25.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:25.503+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1290.580919 secs 2024-07-26T09:02:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1290.580919 secs 2024-07-26T09:02:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1290.580919 secs 2024-07-26T09:02:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:26 smithi044 ceph-mon[32404]: pgmap v1237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:26 smithi116 ceph-mon[37097]: pgmap v1237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:26 smithi171 ceph-mon[37789]: pgmap v1237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:29.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:28 smithi044 ceph-mon[32404]: pgmap v1238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:28 smithi116 ceph-mon[37097]: pgmap v1238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:28 smithi171 ceph-mon[37789]: pgmap v1238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:30 smithi044 ceph-mon[32404]: pgmap v1239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1295.581045 secs 2024-07-26T09:02:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:30 smithi116 ceph-mon[37097]: pgmap v1239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1295.581045 secs 2024-07-26T09:02:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:30 smithi171 ceph-mon[37789]: pgmap v1239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1295.581045 secs 2024-07-26T09:02:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:32 smithi044 ceph-mon[32404]: pgmap v1240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:32 smithi116 ceph-mon[37097]: pgmap v1240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:32 smithi171 ceph-mon[37789]: pgmap v1240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:34 smithi044 ceph-mon[32404]: pgmap v1241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:34 smithi116 ceph-mon[37097]: pgmap v1241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:34 smithi171 ceph-mon[37789]: pgmap v1241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1300.581206 secs 2024-07-26T09:02:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1300.581206 secs 2024-07-26T09:02:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1300.581206 secs 2024-07-26T09:02:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:36 smithi044 ceph-mon[32404]: pgmap v1242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:36 smithi116 ceph-mon[37097]: pgmap v1242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:36 smithi171 ceph-mon[37789]: pgmap v1242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:38 smithi044 ceph-mon[32404]: pgmap v1243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:38 smithi116 ceph-mon[37097]: pgmap v1243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:38 smithi171 ceph-mon[37789]: pgmap v1243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:40.506+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:02:40.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:40.506+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:02:40.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:40.506+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:40 smithi044 ceph-mon[32404]: pgmap v1244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1305.581355 secs 2024-07-26T09:02:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:40 smithi116 ceph-mon[37097]: pgmap v1244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1305.581355 secs 2024-07-26T09:02:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:40 smithi171 ceph-mon[37789]: pgmap v1244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1305.581355 secs 2024-07-26T09:02:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:42 smithi044 ceph-mon[32404]: pgmap v1245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:42 smithi116 ceph-mon[37097]: pgmap v1245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:42 smithi171 ceph-mon[37789]: pgmap v1245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:44 smithi044 ceph-mon[32404]: pgmap v1246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:44 smithi116 ceph-mon[37097]: pgmap v1246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:44 smithi171 ceph-mon[37789]: pgmap v1246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:02:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1310.581516 secs 2024-07-26T09:02:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1310.581516 secs 2024-07-26T09:02:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1310.581516 secs 2024-07-26T09:02:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:46 smithi044 ceph-mon[32404]: pgmap v1247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:46 smithi116 ceph-mon[37097]: pgmap v1247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:46 smithi171 ceph-mon[37789]: pgmap v1247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:02:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:48 smithi116 ceph-mon[37097]: pgmap v1248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:48 smithi171 ceph-mon[37789]: pgmap v1248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:48 smithi044 ceph-mon[32404]: pgmap v1248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:51.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: pgmap v1249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:51.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:51.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1315.581648 secs 2024-07-26T09:02:51.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.047 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.047 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.047 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.047 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.047 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.047 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: pgmap v1249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1315.581648 secs 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: pgmap v1249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1315.581648 secs 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:51.971 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:51 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:02:51.971 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:51 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:02:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:51 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:02:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:51 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:02:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:51 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:02:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:51 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:02:52.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:52 smithi171 ceph-mon[37789]: pgmap v1250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:52 smithi044 ceph-mon[32404]: pgmap v1250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:52 smithi116 ceph-mon[37097]: pgmap v1250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: pgmap v1251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: pgmap v1252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: pgmap v1253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:02:54.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:02:54.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: pgmap v1251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:54.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:02:54.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:54.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:54.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: pgmap v1252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:02:54.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: pgmap v1253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:02:54.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:55.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: pgmap v1251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:02:55.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:02:55.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:55.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:55.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: pgmap v1252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:02:55.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: pgmap v1253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:02:55.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:02:55.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:55.514+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:02:55.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:55.514+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:02:55.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:02:55.514+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1320.581802 secs 2024-07-26T09:02:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1320.581802 secs 2024-07-26T09:02:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1320.581802 secs 2024-07-26T09:02:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:56 smithi171 ceph-mon[37789]: pgmap v1254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:56.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:56 smithi044 ceph-mon[32404]: pgmap v1254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:56 smithi116 ceph-mon[37097]: pgmap v1254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:02:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:02:58 smithi044 ceph-mon[32404]: pgmap v1255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:02:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:02:58 smithi116 ceph-mon[37097]: pgmap v1255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:02:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:02:58 smithi171 ceph-mon[37789]: pgmap v1255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:03:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:00 smithi044 ceph-mon[32404]: pgmap v1256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1325.581895 secs 2024-07-26T09:03:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:00 smithi116 ceph-mon[37097]: pgmap v1256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1325.581895 secs 2024-07-26T09:03:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:00 smithi171 ceph-mon[37789]: pgmap v1256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:03:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1325.581895 secs 2024-07-26T09:03:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:02 smithi044 ceph-mon[32404]: pgmap v1257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:03:03.020 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:02 smithi171 ceph-mon[37789]: pgmap v1257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:03:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:02 smithi116 ceph-mon[37097]: pgmap v1257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T09:03:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:04 smithi044 ceph-mon[32404]: pgmap v1258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:03:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:04 smithi116 ceph-mon[37097]: pgmap v1258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:03:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:04 smithi171 ceph-mon[37789]: pgmap v1258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:03:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1330.582003 secs 2024-07-26T09:03:06.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1330.582003 secs 2024-07-26T09:03:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1330.582003 secs 2024-07-26T09:03:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:06 smithi044 ceph-mon[32404]: pgmap v1259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:06 smithi116 ceph-mon[37097]: pgmap v1259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:06 smithi171 ceph-mon[37789]: pgmap v1259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:08 smithi044 ceph-mon[32404]: pgmap v1260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:08 smithi116 ceph-mon[37097]: pgmap v1260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:08 smithi171 ceph-mon[37789]: pgmap v1260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:10.517+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:03:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:10.518+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:10.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:10.518+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:03:10.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:10 smithi044 ceph-mon[32404]: pgmap v1261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:10.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1335.582416 secs 2024-07-26T09:03:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:10 smithi116 ceph-mon[37097]: pgmap v1261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1335.582416 secs 2024-07-26T09:03:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:10 smithi171 ceph-mon[37789]: pgmap v1261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1335.582416 secs 2024-07-26T09:03:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:12 smithi044 ceph-mon[32404]: pgmap v1262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:12 smithi116 ceph-mon[37097]: pgmap v1262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:12 smithi171 ceph-mon[37789]: pgmap v1262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:14 smithi044 ceph-mon[32404]: pgmap v1263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:14 smithi116 ceph-mon[37097]: pgmap v1263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:14 smithi171 ceph-mon[37789]: pgmap v1263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1340.582564 secs 2024-07-26T09:03:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1340.582564 secs 2024-07-26T09:03:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1340.582564 secs 2024-07-26T09:03:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:16 smithi044 ceph-mon[32404]: pgmap v1264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:16 smithi116 ceph-mon[37097]: pgmap v1264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:16 smithi171 ceph-mon[37789]: pgmap v1264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:18 smithi044 ceph-mon[32404]: pgmap v1265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:18 smithi116 ceph-mon[37097]: pgmap v1265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:18 smithi171 ceph-mon[37789]: pgmap v1265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:20 smithi044 ceph-mon[32404]: pgmap v1266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1345.582722 secs 2024-07-26T09:03:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:20 smithi116 ceph-mon[37097]: pgmap v1266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1345.582722 secs 2024-07-26T09:03:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:20 smithi171 ceph-mon[37789]: pgmap v1266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1345.582722 secs 2024-07-26T09:03:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:22 smithi044 ceph-mon[32404]: pgmap v1267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:22 smithi116 ceph-mon[37097]: pgmap v1267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:22 smithi171 ceph-mon[37789]: pgmap v1267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:24 smithi044 ceph-mon[32404]: pgmap v1268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:24 smithi116 ceph-mon[37097]: pgmap v1268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:24 smithi171 ceph-mon[37789]: pgmap v1268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:25.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:25.526+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:03:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:25.526+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:03:25.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:25.526+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1350.582868 secs 2024-07-26T09:03:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1350.582868 secs 2024-07-26T09:03:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1350.582868 secs 2024-07-26T09:03:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:26 smithi044 ceph-mon[32404]: pgmap v1269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:26 smithi116 ceph-mon[37097]: pgmap v1269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:26 smithi171 ceph-mon[37789]: pgmap v1269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:28 smithi044 ceph-mon[32404]: pgmap v1270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:28 smithi116 ceph-mon[37097]: pgmap v1270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:28 smithi171 ceph-mon[37789]: pgmap v1270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:30 smithi044 ceph-mon[32404]: pgmap v1271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1355.582961 secs 2024-07-26T09:03:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:30 smithi116 ceph-mon[37097]: pgmap v1271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1355.582961 secs 2024-07-26T09:03:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:30 smithi171 ceph-mon[37789]: pgmap v1271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1355.582961 secs 2024-07-26T09:03:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:32 smithi044 ceph-mon[32404]: pgmap v1272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:32 smithi116 ceph-mon[37097]: pgmap v1272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:32 smithi171 ceph-mon[37789]: pgmap v1272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:34 smithi044 ceph-mon[32404]: pgmap v1273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:34 smithi116 ceph-mon[37097]: pgmap v1273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:34 smithi171 ceph-mon[37789]: pgmap v1273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1360.583122 secs 2024-07-26T09:03:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1360.583122 secs 2024-07-26T09:03:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1360.583122 secs 2024-07-26T09:03:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:36 smithi171 ceph-mon[37789]: pgmap v1274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:36 smithi044 ceph-mon[32404]: pgmap v1274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:36 smithi116 ceph-mon[37097]: pgmap v1274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:03:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:38 smithi171 ceph-mon[37789]: pgmap v1275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:38 smithi044 ceph-mon[32404]: pgmap v1275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:38 smithi116 ceph-mon[37097]: pgmap v1275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:40.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:40.528+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:03:40.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:40.528+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:03:40.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:40.528+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:40 smithi171 ceph-mon[37789]: pgmap v1276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:03:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1365.583239 secs 2024-07-26T09:03:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:40 smithi044 ceph-mon[32404]: pgmap v1276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:03:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1365.583239 secs 2024-07-26T09:03:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:40 smithi116 ceph-mon[37097]: pgmap v1276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:03:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1365.583239 secs 2024-07-26T09:03:43.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:42 smithi116 ceph-mon[37097]: pgmap v1277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:42 smithi171 ceph-mon[37789]: pgmap v1277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:42 smithi044 ceph-mon[32404]: pgmap v1277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:44 smithi171 ceph-mon[37789]: pgmap v1278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:44 smithi044 ceph-mon[32404]: pgmap v1278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:44 smithi116 ceph-mon[37097]: pgmap v1278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1370.583396 secs 2024-07-26T09:03:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1370.583396 secs 2024-07-26T09:03:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1370.583396 secs 2024-07-26T09:03:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:46 smithi171 ceph-mon[37789]: pgmap v1279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:03:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:46 smithi044 ceph-mon[32404]: pgmap v1279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:03:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:46 smithi116 ceph-mon[37097]: pgmap v1279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:03:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:48 smithi171 ceph-mon[37789]: pgmap v1280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:48 smithi044 ceph-mon[32404]: pgmap v1280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:48 smithi116 ceph-mon[37097]: pgmap v1280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:03:51.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:50 smithi171 ceph-mon[37789]: pgmap v1281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:51.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1375.583552 secs 2024-07-26T09:03:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:50 smithi044 ceph-mon[32404]: pgmap v1281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1375.583552 secs 2024-07-26T09:03:51.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:50 smithi116 ceph-mon[37097]: pgmap v1281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:03:51.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1375.583552 secs 2024-07-26T09:03:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:52 smithi171 ceph-mon[37789]: pgmap v1282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:52 smithi044 ceph-mon[32404]: pgmap v1282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:52 smithi116 ceph-mon[37097]: pgmap v1282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:54 smithi044 ceph-mon[32404]: pgmap v1283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:54 smithi116 ceph-mon[37097]: pgmap v1283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:54 smithi171 ceph-mon[37789]: pgmap v1283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:03:55.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:55.536+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:03:55.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:55.536+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:03:55.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:03:55.536+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:55 smithi044 ceph-mon[32404]: pgmap v1284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:55 smithi044 ceph-mon[32404]: pgmap v1285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:03:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1380.583703 secs 2024-07-26T09:03:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:55 smithi116 ceph-mon[37097]: pgmap v1284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:55 smithi116 ceph-mon[37097]: pgmap v1285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:03:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1380.583703 secs 2024-07-26T09:03:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:55 smithi171 ceph-mon[37789]: pgmap v1284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:03:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:55 smithi171 ceph-mon[37789]: pgmap v1285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:03:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1380.583703 secs 2024-07-26T09:03:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:57 smithi044 ceph-mon[32404]: pgmap v1286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:57 smithi116 ceph-mon[37097]: pgmap v1286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:57 smithi171 ceph-mon[37789]: pgmap v1286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:59.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:03:58 smithi044 ceph-mon[32404]: pgmap v1287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:03:58 smithi116 ceph-mon[37097]: pgmap v1287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:03:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:03:58 smithi171 ceph-mon[37789]: pgmap v1287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:04:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1385.583831 secs 2024-07-26T09:04:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1385.583831 secs 2024-07-26T09:04:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1385.583831 secs 2024-07-26T09:04:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:01 smithi044 ceph-mon[32404]: pgmap v1288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:04:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:01 smithi116 ceph-mon[37097]: pgmap v1288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:04:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:01 smithi171 ceph-mon[37789]: pgmap v1288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:04:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:03 smithi044 ceph-mon[32404]: pgmap v1289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:04:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:03 smithi116 ceph-mon[37097]: pgmap v1289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:04:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:03 smithi171 ceph-mon[37789]: pgmap v1289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:04:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:04 smithi044 ceph-mon[32404]: pgmap v1290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:04 smithi116 ceph-mon[37097]: pgmap v1290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:04 smithi171 ceph-mon[37789]: pgmap v1290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1390.583927 secs 2024-07-26T09:04:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1390.583927 secs 2024-07-26T09:04:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1390.583927 secs 2024-07-26T09:04:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:06 smithi044 ceph-mon[32404]: pgmap v1291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:06 smithi116 ceph-mon[37097]: pgmap v1291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:06 smithi171 ceph-mon[37789]: pgmap v1291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:08 smithi044 ceph-mon[32404]: pgmap v1292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:08 smithi116 ceph-mon[37097]: pgmap v1292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:08 smithi171 ceph-mon[37789]: pgmap v1292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:10.538+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:10.538+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:10.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:10.538+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:04:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1395.584082 secs 2024-07-26T09:04:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1395.584082 secs 2024-07-26T09:04:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1395.584082 secs 2024-07-26T09:04:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:11 smithi044 ceph-mon[32404]: pgmap v1293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:11 smithi116 ceph-mon[37097]: pgmap v1293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:11 smithi171 ceph-mon[37789]: pgmap v1293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:12 smithi044 ceph-mon[32404]: pgmap v1294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:12 smithi116 ceph-mon[37097]: pgmap v1294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:12 smithi171 ceph-mon[37789]: pgmap v1294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:14 smithi044 ceph-mon[32404]: pgmap v1295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:14 smithi116 ceph-mon[37097]: pgmap v1295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:14 smithi171 ceph-mon[37789]: pgmap v1295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1400.584235 secs 2024-07-26T09:04:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1400.584235 secs 2024-07-26T09:04:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1400.584235 secs 2024-07-26T09:04:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:16 smithi044 ceph-mon[32404]: pgmap v1296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:16 smithi116 ceph-mon[37097]: pgmap v1296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:16 smithi171 ceph-mon[37789]: pgmap v1296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:18 smithi044 ceph-mon[32404]: pgmap v1297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:18 smithi116 ceph-mon[37097]: pgmap v1297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:18 smithi171 ceph-mon[37789]: pgmap v1297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1405.584371 secs 2024-07-26T09:04:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1405.584371 secs 2024-07-26T09:04:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1405.584371 secs 2024-07-26T09:04:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:21 smithi044 ceph-mon[32404]: pgmap v1298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:21 smithi116 ceph-mon[37097]: pgmap v1298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:21 smithi171 ceph-mon[37789]: pgmap v1298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:22 smithi044 ceph-mon[32404]: pgmap v1299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:22 smithi116 ceph-mon[37097]: pgmap v1299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:22 smithi171 ceph-mon[37789]: pgmap v1299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:24 smithi044 ceph-mon[32404]: pgmap v1300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:24 smithi116 ceph-mon[37097]: pgmap v1300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:24 smithi171 ceph-mon[37789]: pgmap v1300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:25.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:25.547+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:04:25.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:25.547+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:25.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:25.547+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:04:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1410.584536 secs 2024-07-26T09:04:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1410.584536 secs 2024-07-26T09:04:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1410.584536 secs 2024-07-26T09:04:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:26 smithi044 ceph-mon[32404]: pgmap v1301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:26 smithi116 ceph-mon[37097]: pgmap v1301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:26 smithi171 ceph-mon[37789]: pgmap v1301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:28 smithi044 ceph-mon[32404]: pgmap v1302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:28 smithi116 ceph-mon[37097]: pgmap v1302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:28 smithi171 ceph-mon[37789]: pgmap v1302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1415.584691 secs 2024-07-26T09:04:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1415.584691 secs 2024-07-26T09:04:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1415.584691 secs 2024-07-26T09:04:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:31 smithi044 ceph-mon[32404]: pgmap v1303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:31 smithi116 ceph-mon[37097]: pgmap v1303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:31 smithi171 ceph-mon[37789]: pgmap v1303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:32 smithi044 ceph-mon[32404]: pgmap v1304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:32 smithi116 ceph-mon[37097]: pgmap v1304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:32 smithi171 ceph-mon[37789]: pgmap v1304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:34 smithi044 ceph-mon[32404]: pgmap v1305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:34 smithi116 ceph-mon[37097]: pgmap v1305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:34 smithi171 ceph-mon[37789]: pgmap v1305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1420.584800 secs 2024-07-26T09:04:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1420.584800 secs 2024-07-26T09:04:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1420.584800 secs 2024-07-26T09:04:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:37 smithi044 ceph-mon[32404]: pgmap v1306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:37 smithi116 ceph-mon[37097]: pgmap v1306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:37 smithi171 ceph-mon[37789]: pgmap v1306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:38 smithi044 ceph-mon[32404]: pgmap v1307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:38 smithi116 ceph-mon[37097]: pgmap v1307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:38 smithi171 ceph-mon[37789]: pgmap v1307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:40.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:40.549+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:04:40.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:40.549+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:40.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:40.549+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:04:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1425.584956 secs 2024-07-26T09:04:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1425.584956 secs 2024-07-26T09:04:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1425.584956 secs 2024-07-26T09:04:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:41 smithi044 ceph-mon[32404]: pgmap v1308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:41 smithi116 ceph-mon[37097]: pgmap v1308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:41 smithi171 ceph-mon[37789]: pgmap v1308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:42 smithi044 ceph-mon[32404]: pgmap v1309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:42 smithi116 ceph-mon[37097]: pgmap v1309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:42 smithi171 ceph-mon[37789]: pgmap v1309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:44 smithi044 ceph-mon[32404]: pgmap v1310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:44 smithi116 ceph-mon[37097]: pgmap v1310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:44 smithi171 ceph-mon[37789]: pgmap v1310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1430.585101 secs 2024-07-26T09:04:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1430.585101 secs 2024-07-26T09:04:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1430.585101 secs 2024-07-26T09:04:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:47 smithi044 ceph-mon[32404]: pgmap v1311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:47 smithi116 ceph-mon[37097]: pgmap v1311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:47 smithi171 ceph-mon[37789]: pgmap v1311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:48 smithi044 ceph-mon[32404]: pgmap v1312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:48 smithi116 ceph-mon[37097]: pgmap v1312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:48 smithi171 ceph-mon[37789]: pgmap v1312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1435.585229 secs 2024-07-26T09:04:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1435.585229 secs 2024-07-26T09:04:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1435.585229 secs 2024-07-26T09:04:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:51 smithi044 ceph-mon[32404]: pgmap v1313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:51 smithi116 ceph-mon[37097]: pgmap v1313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:51 smithi171 ceph-mon[37789]: pgmap v1313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:52 smithi044 ceph-mon[32404]: pgmap v1314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:52 smithi116 ceph-mon[37097]: pgmap v1314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:52 smithi171 ceph-mon[37789]: pgmap v1314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:04:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:54 smithi044 ceph-mon[32404]: pgmap v1315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:54 smithi116 ceph-mon[37097]: pgmap v1315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:54 smithi171 ceph-mon[37789]: pgmap v1315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:04:55.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:55.556+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:04:55.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:55.556+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:04:55.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:04:55.556+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:55 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 1440.585339 secs 2024-07-26T09:04:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:55 smithi044 ceph-mon[32404]: slow request 1440.585338 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T09:04:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:55 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 1440.585339 secs 2024-07-26T09:04:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:55 smithi116 ceph-mon[37097]: slow request 1440.585338 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T09:04:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:55 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 1440.585339 secs 2024-07-26T09:04:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:55 smithi171 ceph-mon[37789]: slow request 1440.585338 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T09:04:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:57 smithi044 ceph-mon[32404]: pgmap v1316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:57 smithi044 ceph-mon[32404]: pgmap v1317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:57 smithi116 ceph-mon[37097]: pgmap v1316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:57 smithi116 ceph-mon[37097]: pgmap v1317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:04:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:57 smithi171 ceph-mon[37789]: pgmap v1316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:04:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:57 smithi171 ceph-mon[37789]: pgmap v1317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:04:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:04:59 smithi044 ceph-mon[32404]: pgmap v1318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:04:59 smithi116 ceph-mon[37097]: pgmap v1318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:04:59 smithi171 ceph-mon[37789]: pgmap v1318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:01.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1445.585497 secs 2024-07-26T09:05:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1445.585497 secs 2024-07-26T09:05:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1445.585497 secs 2024-07-26T09:05:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:01 smithi044 ceph-mon[32404]: pgmap v1319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:01 smithi116 ceph-mon[37097]: pgmap v1319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:01 smithi171 ceph-mon[37789]: pgmap v1319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:03 smithi044 ceph-mon[32404]: pgmap v1320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:03 smithi116 ceph-mon[37097]: pgmap v1320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:04.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:03 smithi171 ceph-mon[37789]: pgmap v1320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:05 smithi044 ceph-mon[32404]: pgmap v1321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1450.585665 secs 2024-07-26T09:05:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:05 smithi116 ceph-mon[37097]: pgmap v1321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1450.585665 secs 2024-07-26T09:05:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:05 smithi171 ceph-mon[37789]: pgmap v1321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:05:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1450.585665 secs 2024-07-26T09:05:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:06 smithi044 ceph-mon[32404]: pgmap v1322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:05:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:06 smithi116 ceph-mon[37097]: pgmap v1322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:05:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:06 smithi171 ceph-mon[37789]: pgmap v1322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:05:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:09 smithi044 ceph-mon[32404]: pgmap v1323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:09 smithi116 ceph-mon[37097]: pgmap v1323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:09 smithi171 ceph-mon[37789]: pgmap v1323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:10.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:10.559+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:05:10.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:10.559+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:10.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:10.559+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:05:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1455.585806 secs 2024-07-26T09:05:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1455.585806 secs 2024-07-26T09:05:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1455.585806 secs 2024-07-26T09:05:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:11 smithi044 ceph-mon[32404]: pgmap v1324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:05:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:11 smithi116 ceph-mon[37097]: pgmap v1324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:05:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:11 smithi171 ceph-mon[37789]: pgmap v1324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:05:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:12 smithi044 ceph-mon[32404]: pgmap v1325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:05:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:12 smithi116 ceph-mon[37097]: pgmap v1325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:05:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:12 smithi171 ceph-mon[37789]: pgmap v1325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:05:14.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:14.917+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:05:14.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:14.917+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:05:14.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:14.918+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:05:14.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:14.918+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:05:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:15 smithi044 ceph-mon[32404]: pgmap v1326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:05:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1460.585956 secs 2024-07-26T09:05:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:15 smithi116 ceph-mon[37097]: pgmap v1326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:05:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1460.585956 secs 2024-07-26T09:05:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:15 smithi171 ceph-mon[37789]: pgmap v1326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:05:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1460.585956 secs 2024-07-26T09:05:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:17 smithi044 ceph-mon[32404]: pgmap v1327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:17 smithi116 ceph-mon[37097]: pgmap v1327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:17 smithi171 ceph-mon[37789]: pgmap v1327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:05:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:19 smithi044 ceph-mon[32404]: pgmap v1328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:19 smithi116 ceph-mon[37097]: pgmap v1328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:19 smithi171 ceph-mon[37789]: pgmap v1328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1465.586039 secs 2024-07-26T09:05:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1465.586039 secs 2024-07-26T09:05:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1465.586039 secs 2024-07-26T09:05:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:21 smithi044 ceph-mon[32404]: pgmap v1329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:21 smithi116 ceph-mon[37097]: pgmap v1329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:21 smithi171 ceph-mon[37789]: pgmap v1329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:22 smithi044 ceph-mon[32404]: pgmap v1330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:22 smithi116 ceph-mon[37097]: pgmap v1330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:22 smithi171 ceph-mon[37789]: pgmap v1330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:25.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:25.567+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:05:25.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:25.567+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:25.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:25.567+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:05:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:25 smithi044 ceph-mon[32404]: pgmap v1331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1470.586203 secs 2024-07-26T09:05:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:25 smithi116 ceph-mon[37097]: pgmap v1331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1470.586203 secs 2024-07-26T09:05:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:25 smithi171 ceph-mon[37789]: pgmap v1331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1470.586203 secs 2024-07-26T09:05:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:27 smithi044 ceph-mon[32404]: pgmap v1332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:27 smithi116 ceph-mon[37097]: pgmap v1332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:27 smithi171 ceph-mon[37789]: pgmap v1332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:29 smithi044 ceph-mon[32404]: pgmap v1333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:29 smithi116 ceph-mon[37097]: pgmap v1333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:30.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:29 smithi171 ceph-mon[37789]: pgmap v1333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1475.586334 secs 2024-07-26T09:05:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1475.586334 secs 2024-07-26T09:05:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1475.586334 secs 2024-07-26T09:05:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:31 smithi044 ceph-mon[32404]: pgmap v1334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:31 smithi116 ceph-mon[37097]: pgmap v1334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:31 smithi171 ceph-mon[37789]: pgmap v1334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:32 smithi044 ceph-mon[32404]: pgmap v1335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:32 smithi116 ceph-mon[37097]: pgmap v1335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:32 smithi171 ceph-mon[37789]: pgmap v1335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:35 smithi044 ceph-mon[32404]: pgmap v1336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1480.586505 secs 2024-07-26T09:05:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:35 smithi116 ceph-mon[37097]: pgmap v1336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1480.586505 secs 2024-07-26T09:05:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:35 smithi171 ceph-mon[37789]: pgmap v1336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1480.586505 secs 2024-07-26T09:05:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:37 smithi044 ceph-mon[32404]: pgmap v1337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:37 smithi116 ceph-mon[37097]: pgmap v1337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:37 smithi171 ceph-mon[37789]: pgmap v1337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:39 smithi044 ceph-mon[32404]: pgmap v1338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:39 smithi116 ceph-mon[37097]: pgmap v1338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:39 smithi171 ceph-mon[37789]: pgmap v1338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:40.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:40.570+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:05:40.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:40.570+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:40.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:40.570+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:05:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1485.586668 secs 2024-07-26T09:05:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1485.586668 secs 2024-07-26T09:05:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1485.586668 secs 2024-07-26T09:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:41 smithi044 ceph-mon[32404]: pgmap v1339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:41 smithi116 ceph-mon[37097]: pgmap v1339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:41 smithi171 ceph-mon[37789]: pgmap v1339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:43 smithi044 ceph-mon[32404]: pgmap v1340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:43 smithi116 ceph-mon[37097]: pgmap v1340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:43 smithi171 ceph-mon[37789]: pgmap v1340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:45 smithi044 ceph-mon[32404]: pgmap v1341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:05:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1490.586835 secs 2024-07-26T09:05:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:45 smithi116 ceph-mon[37097]: pgmap v1341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:05:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1490.586835 secs 2024-07-26T09:05:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:45 smithi171 ceph-mon[37789]: pgmap v1341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:05:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1490.586835 secs 2024-07-26T09:05:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:46 smithi044 ceph-mon[32404]: pgmap v1342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:46 smithi116 ceph-mon[37097]: pgmap v1342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:46 smithi171 ceph-mon[37789]: pgmap v1342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:49 smithi044 ceph-mon[32404]: pgmap v1343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:49 smithi116 ceph-mon[37097]: pgmap v1343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:49 smithi171 ceph-mon[37789]: pgmap v1343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1495.586978 secs 2024-07-26T09:05:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1495.586978 secs 2024-07-26T09:05:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1495.586978 secs 2024-07-26T09:05:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:51 smithi044 ceph-mon[32404]: pgmap v1344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:05:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:51 smithi116 ceph-mon[37097]: pgmap v1344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:05:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:51 smithi171 ceph-mon[37789]: pgmap v1344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:05:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:52 smithi044 ceph-mon[32404]: pgmap v1345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:52 smithi116 ceph-mon[37097]: pgmap v1345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:52 smithi171 ceph-mon[37789]: pgmap v1345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:05:55.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:55.578+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:05:55.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:55.578+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:05:55.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:55.578+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:55 smithi044 ceph-mon[32404]: pgmap v1346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1500.587143 secs 2024-07-26T09:05:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:55 smithi116 ceph-mon[37097]: pgmap v1346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1500.587143 secs 2024-07-26T09:05:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:55 smithi171 ceph-mon[37789]: pgmap v1346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:05:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1500.587143 secs 2024-07-26T09:05:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:56 smithi044 ceph-mon[32404]: pgmap v1347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:56 smithi116 ceph-mon[37097]: pgmap v1347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:56 smithi171 ceph-mon[37789]: pgmap v1347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:58.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.254 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.254 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.254 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:05:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:05:58 smithi044 ceph-mon[32404]: pgmap v1348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:05:58 smithi116 ceph-mon[37097]: pgmap v1348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:05:58 smithi171 ceph-mon[37789]: pgmap v1348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:05:59.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:59.925+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:05:59.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:59.925+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:05:59.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:59.926+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:05:59.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:05:59.926+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:06:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1505.587218 secs 2024-07-26T09:06:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:01 smithi044 ceph-mon[32404]: pgmap v1349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1505.587218 secs 2024-07-26T09:06:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:01 smithi116 ceph-mon[37097]: pgmap v1349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1505.587218 secs 2024-07-26T09:06:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:01 smithi171 ceph-mon[37789]: pgmap v1349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:03.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:02 smithi171 ceph-mon[37789]: pgmap v1350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:06:03.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:02 smithi171 ceph-mon[37789]: pgmap v1351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:03.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:02 smithi044 ceph-mon[32404]: pgmap v1350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:06:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:02 smithi044 ceph-mon[32404]: pgmap v1351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:02 smithi116 ceph-mon[37097]: pgmap v1350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:06:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:02 smithi116 ceph-mon[37097]: pgmap v1351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:06:04.170 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:04 smithi044 ceph-mon[32404]: pgmap v1352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:04 smithi116 ceph-mon[37097]: pgmap v1352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:04 smithi171 ceph-mon[37789]: pgmap v1352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1510.587381 secs 2024-07-26T09:06:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1510.587381 secs 2024-07-26T09:06:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1510.587381 secs 2024-07-26T09:06:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:06 smithi044 ceph-mon[32404]: pgmap v1353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:06 smithi116 ceph-mon[37097]: pgmap v1353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:06 smithi171 ceph-mon[37789]: pgmap v1353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:08 smithi044 ceph-mon[32404]: pgmap v1354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:08 smithi116 ceph-mon[37097]: pgmap v1354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:08 smithi171 ceph-mon[37789]: pgmap v1354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:10 smithi044 ceph-mon[32404]: pgmap v1355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:10 smithi116 ceph-mon[37097]: pgmap v1355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:10 smithi171 ceph-mon[37789]: pgmap v1355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:06:10.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:10.581+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:06:10.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:10.581+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:10.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:10.581+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:06:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1515.587535 secs 2024-07-26T09:06:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1515.587535 secs 2024-07-26T09:06:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1515.587535 secs 2024-07-26T09:06:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:12 smithi171 ceph-mon[37789]: pgmap v1356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:06:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:12 smithi044 ceph-mon[32404]: pgmap v1356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:06:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:12 smithi116 ceph-mon[37097]: pgmap v1356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:06:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:14 smithi044 ceph-mon[32404]: pgmap v1357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:14 smithi116 ceph-mon[37097]: pgmap v1357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:14 smithi171 ceph-mon[37789]: pgmap v1357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1520.587700 secs 2024-07-26T09:06:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1520.587700 secs 2024-07-26T09:06:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1520.587700 secs 2024-07-26T09:06:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:16 smithi044 ceph-mon[32404]: pgmap v1358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:16 smithi116 ceph-mon[37097]: pgmap v1358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:16 smithi171 ceph-mon[37789]: pgmap v1358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:18 smithi044 ceph-mon[32404]: pgmap v1359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:18 smithi116 ceph-mon[37097]: pgmap v1359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:18 smithi171 ceph-mon[37789]: pgmap v1359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:20 smithi044 ceph-mon[32404]: pgmap v1360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:20 smithi116 ceph-mon[37097]: pgmap v1360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:20 smithi171 ceph-mon[37789]: pgmap v1360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1525.587856 secs 2024-07-26T09:06:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1525.587856 secs 2024-07-26T09:06:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1525.587856 secs 2024-07-26T09:06:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:22 smithi171 ceph-mon[37789]: pgmap v1361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:22 smithi044 ceph-mon[32404]: pgmap v1361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:22 smithi116 ceph-mon[37097]: pgmap v1361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:24 smithi044 ceph-mon[32404]: pgmap v1362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:24 smithi116 ceph-mon[37097]: pgmap v1362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:24 smithi171 ceph-mon[37789]: pgmap v1362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:25.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:25.589+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:06:25.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:25.589+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:25.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:25.589+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:06:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1530.588020 secs 2024-07-26T09:06:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1530.588020 secs 2024-07-26T09:06:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1530.588020 secs 2024-07-26T09:06:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:26 smithi044 ceph-mon[32404]: pgmap v1363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:26 smithi116 ceph-mon[37097]: pgmap v1363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:26 smithi171 ceph-mon[37789]: pgmap v1363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:28 smithi044 ceph-mon[32404]: pgmap v1364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:28 smithi116 ceph-mon[37097]: pgmap v1364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:28 smithi171 ceph-mon[37789]: pgmap v1364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:30 smithi044 ceph-mon[32404]: pgmap v1365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:30 smithi116 ceph-mon[37097]: pgmap v1365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:30 smithi171 ceph-mon[37789]: pgmap v1365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1535.588167 secs 2024-07-26T09:06:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1535.588167 secs 2024-07-26T09:06:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1535.588167 secs 2024-07-26T09:06:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:32 smithi171 ceph-mon[37789]: pgmap v1366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:32 smithi044 ceph-mon[32404]: pgmap v1366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:32 smithi116 ceph-mon[37097]: pgmap v1366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:34 smithi044 ceph-mon[32404]: pgmap v1367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:34 smithi116 ceph-mon[37097]: pgmap v1367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:34 smithi171 ceph-mon[37789]: pgmap v1367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1540.588326 secs 2024-07-26T09:06:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1540.588326 secs 2024-07-26T09:06:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1540.588326 secs 2024-07-26T09:06:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:36 smithi044 ceph-mon[32404]: pgmap v1368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:36 smithi116 ceph-mon[37097]: pgmap v1368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:36 smithi171 ceph-mon[37789]: pgmap v1368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:38 smithi044 ceph-mon[32404]: pgmap v1369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:38 smithi116 ceph-mon[37097]: pgmap v1369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:38 smithi171 ceph-mon[37789]: pgmap v1369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:40 smithi044 ceph-mon[32404]: pgmap v1370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:40 smithi116 ceph-mon[37097]: pgmap v1370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:40 smithi171 ceph-mon[37789]: pgmap v1370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:40.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:40.591+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:06:40.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:40.591+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:40.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:40.591+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:06:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1545.588465 secs 2024-07-26T09:06:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1545.588465 secs 2024-07-26T09:06:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1545.588465 secs 2024-07-26T09:06:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:42 smithi171 ceph-mon[37789]: pgmap v1371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:42 smithi044 ceph-mon[32404]: pgmap v1371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:42 smithi116 ceph-mon[37097]: pgmap v1371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:44 smithi116 ceph-mon[37097]: pgmap v1372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:44.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:44 smithi044 ceph-mon[32404]: pgmap v1372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:44 smithi171 ceph-mon[37789]: pgmap v1372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1550.588554 secs 2024-07-26T09:06:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1550.588554 secs 2024-07-26T09:06:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1550.588554 secs 2024-07-26T09:06:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:46 smithi171 ceph-mon[37789]: pgmap v1373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:46 smithi044 ceph-mon[32404]: pgmap v1373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:46 smithi116 ceph-mon[37097]: pgmap v1373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:48 smithi116 ceph-mon[37097]: pgmap v1374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:48 smithi171 ceph-mon[37789]: pgmap v1374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:48 smithi044 ceph-mon[32404]: pgmap v1374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:50 smithi044 ceph-mon[32404]: pgmap v1375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:50 smithi116 ceph-mon[37097]: pgmap v1375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:50 smithi171 ceph-mon[37789]: pgmap v1375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1555.588697 secs 2024-07-26T09:06:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1555.588697 secs 2024-07-26T09:06:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1555.588697 secs 2024-07-26T09:06:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:52 smithi171 ceph-mon[37789]: pgmap v1376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:52 smithi044 ceph-mon[32404]: pgmap v1376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:52 smithi116 ceph-mon[37097]: pgmap v1376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:54.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:54 smithi116 ceph-mon[37097]: pgmap v1377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:54 smithi171 ceph-mon[37789]: pgmap v1377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:54 smithi044 ceph-mon[32404]: pgmap v1377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:06:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:55.600+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:06:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:55.600+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:06:55.600+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:06:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1560.588802 secs 2024-07-26T09:06:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1560.588802 secs 2024-07-26T09:06:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1560.588802 secs 2024-07-26T09:06:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:56 smithi044 ceph-mon[32404]: pgmap v1378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:56 smithi116 ceph-mon[37097]: pgmap v1378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:56 smithi171 ceph-mon[37789]: pgmap v1378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:06:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:06:58 smithi116 ceph-mon[37097]: pgmap v1379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:06:58 smithi171 ceph-mon[37789]: pgmap v1379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:06:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:06:58 smithi044 ceph-mon[32404]: pgmap v1379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:00 smithi116 ceph-mon[37097]: pgmap v1380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:00 smithi171 ceph-mon[37789]: pgmap v1380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:00 smithi044 ceph-mon[32404]: pgmap v1380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1565.588888 secs 2024-07-26T09:07:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1565.588888 secs 2024-07-26T09:07:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1565.588888 secs 2024-07-26T09:07:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:02 smithi171 ceph-mon[37789]: pgmap v1381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:02 smithi044 ceph-mon[32404]: pgmap v1381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:02 smithi116 ceph-mon[37097]: pgmap v1381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:04 smithi171 ceph-mon[37789]: pgmap v1382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:07:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:04 smithi171 ceph-mon[37789]: pgmap v1383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:04.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:07:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:04 smithi044 ceph-mon[32404]: pgmap v1382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:07:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:04 smithi044 ceph-mon[32404]: pgmap v1383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:07:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:04 smithi116 ceph-mon[37097]: pgmap v1382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:07:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:04 smithi116 ceph-mon[37097]: pgmap v1383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:07:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:06 smithi044 ceph-mon[32404]: pgmap v1384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1570.589009 secs 2024-07-26T09:07:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:06 smithi116 ceph-mon[37097]: pgmap v1384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1570.589009 secs 2024-07-26T09:07:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:06 smithi171 ceph-mon[37789]: pgmap v1384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1570.589009 secs 2024-07-26T09:07:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:08 smithi044 ceph-mon[32404]: pgmap v1385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:07:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:08 smithi116 ceph-mon[37097]: pgmap v1385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:07:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:08 smithi171 ceph-mon[37789]: pgmap v1385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:07:10.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:10 smithi044 ceph-mon[32404]: pgmap v1386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:10 smithi116 ceph-mon[37097]: pgmap v1386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:10 smithi171 ceph-mon[37789]: pgmap v1386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:07:10.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:10.603+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:07:10.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:10.603+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:10.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:10.603+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:07:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1575.589170 secs 2024-07-26T09:07:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1575.589170 secs 2024-07-26T09:07:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1575.589170 secs 2024-07-26T09:07:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:12 smithi116 ceph-mon[37097]: pgmap v1387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:07:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:12 smithi171 ceph-mon[37789]: pgmap v1387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:07:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:12 smithi044 ceph-mon[32404]: pgmap v1387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:07:14.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:14 smithi044 ceph-mon[32404]: pgmap v1388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:07:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:14 smithi171 ceph-mon[37789]: pgmap v1388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:07:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:14 smithi116 ceph-mon[37097]: pgmap v1388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:07:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:16 smithi171 ceph-mon[37789]: pgmap v1389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1580.589319 secs 2024-07-26T09:07:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:16 smithi044 ceph-mon[32404]: pgmap v1389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1580.589319 secs 2024-07-26T09:07:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:16 smithi116 ceph-mon[37097]: pgmap v1389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1580.589319 secs 2024-07-26T09:07:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:18 smithi171 ceph-mon[37789]: pgmap v1390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:18 smithi044 ceph-mon[32404]: pgmap v1390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:18 smithi116 ceph-mon[37097]: pgmap v1390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:20 smithi171 ceph-mon[37789]: pgmap v1391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:20.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:20 smithi044 ceph-mon[32404]: pgmap v1391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:20 smithi116 ceph-mon[37097]: pgmap v1391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1585.589465 secs 2024-07-26T09:07:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1585.589465 secs 2024-07-26T09:07:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1585.589465 secs 2024-07-26T09:07:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:22 smithi171 ceph-mon[37789]: pgmap v1392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:22 smithi044 ceph-mon[32404]: pgmap v1392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:22 smithi116 ceph-mon[37097]: pgmap v1392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:24 smithi171 ceph-mon[37789]: pgmap v1393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:24 smithi044 ceph-mon[32404]: pgmap v1393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:24 smithi116 ceph-mon[37097]: pgmap v1393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:25.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:25.612+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:07:25.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:25.612+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:07:25.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:25.612+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:26 smithi171 ceph-mon[37789]: pgmap v1394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1590.589607 secs 2024-07-26T09:07:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:26 smithi044 ceph-mon[32404]: pgmap v1394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1590.589607 secs 2024-07-26T09:07:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:26 smithi116 ceph-mon[37097]: pgmap v1394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1590.589607 secs 2024-07-26T09:07:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:28 smithi171 ceph-mon[37789]: pgmap v1395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:28 smithi044 ceph-mon[32404]: pgmap v1395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:28 smithi116 ceph-mon[37097]: pgmap v1395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:30 smithi171 ceph-mon[37789]: pgmap v1396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:30 smithi044 ceph-mon[32404]: pgmap v1396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:30 smithi116 ceph-mon[37097]: pgmap v1396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1595.589713 secs 2024-07-26T09:07:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1595.589713 secs 2024-07-26T09:07:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1595.589713 secs 2024-07-26T09:07:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:32 smithi044 ceph-mon[32404]: pgmap v1397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:32 smithi116 ceph-mon[37097]: pgmap v1397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:32 smithi171 ceph-mon[37789]: pgmap v1397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:34 smithi044 ceph-mon[32404]: pgmap v1398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:34 smithi116 ceph-mon[37097]: pgmap v1398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:34 smithi171 ceph-mon[37789]: pgmap v1398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:36 smithi044 ceph-mon[32404]: pgmap v1399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1600.589817 secs 2024-07-26T09:07:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:36 smithi116 ceph-mon[37097]: pgmap v1399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1600.589817 secs 2024-07-26T09:07:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:36 smithi171 ceph-mon[37789]: pgmap v1399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1600.589817 secs 2024-07-26T09:07:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:38 smithi044 ceph-mon[32404]: pgmap v1400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:38 smithi116 ceph-mon[37097]: pgmap v1400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:38 smithi171 ceph-mon[37789]: pgmap v1400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:40.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:40.616+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:07:40.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:40.616+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:40.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:40.616+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:07:40.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:40 smithi044 ceph-mon[32404]: pgmap v1401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:40 smithi116 ceph-mon[37097]: pgmap v1401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:40 smithi171 ceph-mon[37789]: pgmap v1401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1605.589910 secs 2024-07-26T09:07:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1605.589910 secs 2024-07-26T09:07:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1605.589910 secs 2024-07-26T09:07:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:42 smithi044 ceph-mon[32404]: pgmap v1402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:42 smithi116 ceph-mon[37097]: pgmap v1402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:42 smithi171 ceph-mon[37789]: pgmap v1402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:44 smithi044 ceph-mon[32404]: pgmap v1403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:44 smithi116 ceph-mon[37097]: pgmap v1403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:44 smithi171 ceph-mon[37789]: pgmap v1403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:46 smithi044 ceph-mon[32404]: pgmap v1404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1610.590063 secs 2024-07-26T09:07:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:46 smithi116 ceph-mon[37097]: pgmap v1404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1610.590063 secs 2024-07-26T09:07:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:46 smithi171 ceph-mon[37789]: pgmap v1404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1610.590063 secs 2024-07-26T09:07:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:48 smithi044 ceph-mon[32404]: pgmap v1405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:48 smithi116 ceph-mon[37097]: pgmap v1405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:48 smithi171 ceph-mon[37789]: pgmap v1405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:50 smithi044 ceph-mon[32404]: pgmap v1406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:50 smithi116 ceph-mon[37097]: pgmap v1406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:50 smithi171 ceph-mon[37789]: pgmap v1406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:07:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1615.590206 secs 2024-07-26T09:07:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1615.590206 secs 2024-07-26T09:07:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1615.590206 secs 2024-07-26T09:07:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:52 smithi044 ceph-mon[32404]: pgmap v1407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:52 smithi116 ceph-mon[37097]: pgmap v1407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:52 smithi171 ceph-mon[37789]: pgmap v1407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:54 smithi044 ceph-mon[32404]: pgmap v1408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:54 smithi116 ceph-mon[37097]: pgmap v1408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:54 smithi171 ceph-mon[37789]: pgmap v1408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:07:55.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:55.622+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:07:55.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:55.622+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:55.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:07:55.622+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:07:56.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:56 smithi044 ceph-mon[32404]: pgmap v1409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:56.770 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1620.590367 secs 2024-07-26T09:07:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:56 smithi116 ceph-mon[37097]: pgmap v1409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1620.590367 secs 2024-07-26T09:07:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:56 smithi171 ceph-mon[37789]: pgmap v1409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:07:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1620.590367 secs 2024-07-26T09:07:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:07:58 smithi044 ceph-mon[32404]: pgmap v1410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:07:58 smithi116 ceph-mon[37097]: pgmap v1410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:07:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:07:58 smithi171 ceph-mon[37789]: pgmap v1410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:00 smithi044 ceph-mon[32404]: pgmap v1411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:00 smithi116 ceph-mon[37097]: pgmap v1411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:00 smithi171 ceph-mon[37789]: pgmap v1411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1625.590509 secs 2024-07-26T09:08:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1625.590509 secs 2024-07-26T09:08:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1625.590509 secs 2024-07-26T09:08:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:02 smithi044 ceph-mon[32404]: pgmap v1412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:02 smithi116 ceph-mon[37097]: pgmap v1412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:02 smithi171 ceph-mon[37789]: pgmap v1412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:04 smithi044 ceph-mon[32404]: pgmap v1413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:08:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:08:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:04 smithi116 ceph-mon[37097]: pgmap v1413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:08:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:08:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:04 smithi171 ceph-mon[37789]: pgmap v1413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:08:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:08:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:05 smithi044 ceph-mon[32404]: pgmap v1414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:08:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:05 smithi044 ceph-mon[32404]: pgmap v1415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:05 smithi116 ceph-mon[37097]: pgmap v1414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:08:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:05 smithi116 ceph-mon[37097]: pgmap v1415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:05 smithi171 ceph-mon[37789]: pgmap v1414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:08:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:05 smithi171 ceph-mon[37789]: pgmap v1415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1630.590679 secs 2024-07-26T09:08:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1630.590679 secs 2024-07-26T09:08:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1630.590679 secs 2024-07-26T09:08:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:07 smithi044 ceph-mon[32404]: pgmap v1416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:08:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:07 smithi116 ceph-mon[37097]: pgmap v1416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:08:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:07 smithi171 ceph-mon[37789]: pgmap v1416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:08:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:09 smithi044 ceph-mon[32404]: pgmap v1417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:09 smithi116 ceph-mon[37097]: pgmap v1417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:09 smithi171 ceph-mon[37789]: pgmap v1417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:10.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:10.626+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:08:10.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:10.626+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:10.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:10.626+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:08:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1635.590844 secs 2024-07-26T09:08:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:11 smithi116 ceph-mon[37097]: pgmap v1418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1635.590844 secs 2024-07-26T09:08:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:11 smithi171 ceph-mon[37789]: pgmap v1418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1635.590844 secs 2024-07-26T09:08:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:11 smithi044 ceph-mon[32404]: pgmap v1418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:08:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:13 smithi171 ceph-mon[37789]: pgmap v1419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:08:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:13 smithi044 ceph-mon[32404]: pgmap v1419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:08:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:13 smithi116 ceph-mon[37097]: pgmap v1419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:08:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:14.949+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:08:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:14.949+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:08:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:14.950+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:08:14.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:14.950+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:08:15.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:15 smithi116 ceph-mon[37097]: pgmap v1420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:15 smithi171 ceph-mon[37789]: pgmap v1420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:15 smithi044 ceph-mon[32404]: pgmap v1420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:08:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1640.590969 secs 2024-07-26T09:08:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1640.590969 secs 2024-07-26T09:08:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1640.590969 secs 2024-07-26T09:08:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:17 smithi171 ceph-mon[37789]: pgmap v1421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:17 smithi044 ceph-mon[32404]: pgmap v1421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:17 smithi116 ceph-mon[37097]: pgmap v1421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:19 smithi171 ceph-mon[37789]: pgmap v1422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:19 smithi044 ceph-mon[32404]: pgmap v1422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:19 smithi116 ceph-mon[37097]: pgmap v1422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1645.591127 secs 2024-07-26T09:08:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:21 smithi171 ceph-mon[37789]: pgmap v1423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1645.591127 secs 2024-07-26T09:08:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:21 smithi044 ceph-mon[32404]: pgmap v1423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1645.591127 secs 2024-07-26T09:08:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:21 smithi116 ceph-mon[37097]: pgmap v1423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:23 smithi171 ceph-mon[37789]: pgmap v1424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:23 smithi044 ceph-mon[32404]: pgmap v1424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:23 smithi116 ceph-mon[37097]: pgmap v1424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:25.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:25.634+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:08:25.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:25.634+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:08:25.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:25.634+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:25 smithi171 ceph-mon[37789]: pgmap v1425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:25 smithi044 ceph-mon[32404]: pgmap v1425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:25 smithi116 ceph-mon[37097]: pgmap v1425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1650.591298 secs 2024-07-26T09:08:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1650.591298 secs 2024-07-26T09:08:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1650.591298 secs 2024-07-26T09:08:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:27 smithi044 ceph-mon[32404]: pgmap v1426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:27 smithi116 ceph-mon[37097]: pgmap v1426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:27 smithi171 ceph-mon[37789]: pgmap v1426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:29 smithi044 ceph-mon[32404]: pgmap v1427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:29 smithi116 ceph-mon[37097]: pgmap v1427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:29 smithi171 ceph-mon[37789]: pgmap v1427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1655.591454 secs 2024-07-26T09:08:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1655.591454 secs 2024-07-26T09:08:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1655.591454 secs 2024-07-26T09:08:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:31 smithi044 ceph-mon[32404]: pgmap v1428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:31 smithi116 ceph-mon[37097]: pgmap v1428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:31 smithi171 ceph-mon[37789]: pgmap v1428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:33 smithi044 ceph-mon[32404]: pgmap v1429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:33 smithi116 ceph-mon[37097]: pgmap v1429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:33 smithi171 ceph-mon[37789]: pgmap v1429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:35 smithi044 ceph-mon[32404]: pgmap v1430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1660.591609 secs 2024-07-26T09:08:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:35 smithi116 ceph-mon[37097]: pgmap v1430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1660.591609 secs 2024-07-26T09:08:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:35 smithi171 ceph-mon[37789]: pgmap v1430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1660.591609 secs 2024-07-26T09:08:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:37 smithi044 ceph-mon[32404]: pgmap v1431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:37 smithi116 ceph-mon[37097]: pgmap v1431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:37 smithi171 ceph-mon[37789]: pgmap v1431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:39 smithi044 ceph-mon[32404]: pgmap v1432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:39 smithi116 ceph-mon[37097]: pgmap v1432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:39 smithi171 ceph-mon[37789]: pgmap v1432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:40.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:40.637+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:08:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:40.637+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:08:40.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:40.637+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1665.591749 secs 2024-07-26T09:08:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1665.591749 secs 2024-07-26T09:08:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1665.591749 secs 2024-07-26T09:08:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:41 smithi044 ceph-mon[32404]: pgmap v1433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:41 smithi116 ceph-mon[37097]: pgmap v1433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:41 smithi171 ceph-mon[37789]: pgmap v1433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:43 smithi044 ceph-mon[32404]: pgmap v1434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:43 smithi116 ceph-mon[37097]: pgmap v1434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:43 smithi171 ceph-mon[37789]: pgmap v1434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:45 smithi044 ceph-mon[32404]: pgmap v1435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1670.591899 secs 2024-07-26T09:08:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:45 smithi116 ceph-mon[37097]: pgmap v1435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1670.591899 secs 2024-07-26T09:08:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:45 smithi171 ceph-mon[37789]: pgmap v1435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1670.591899 secs 2024-07-26T09:08:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:47 smithi044 ceph-mon[32404]: pgmap v1436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:47 smithi116 ceph-mon[37097]: pgmap v1436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:47 smithi171 ceph-mon[37789]: pgmap v1436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:49 smithi044 ceph-mon[32404]: pgmap v1437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:49 smithi116 ceph-mon[37097]: pgmap v1437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:49 smithi171 ceph-mon[37789]: pgmap v1437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:08:51.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1675.591993 secs 2024-07-26T09:08:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1675.591993 secs 2024-07-26T09:08:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1675.591993 secs 2024-07-26T09:08:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:51 smithi044 ceph-mon[32404]: pgmap v1438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:51 smithi116 ceph-mon[37097]: pgmap v1438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:51 smithi171 ceph-mon[37789]: pgmap v1438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:08:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:53 smithi044 ceph-mon[32404]: pgmap v1439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:53 smithi116 ceph-mon[37097]: pgmap v1439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:53 smithi171 ceph-mon[37789]: pgmap v1439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:08:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:55.644+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:08:55.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:55.644+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:08:55.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:08:55.644+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:55 smithi044 ceph-mon[32404]: pgmap v1440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1680.592084 secs 2024-07-26T09:08:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:55 smithi116 ceph-mon[37097]: pgmap v1440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1680.592084 secs 2024-07-26T09:08:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:55 smithi171 ceph-mon[37789]: pgmap v1440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:08:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1680.592084 secs 2024-07-26T09:08:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:57 smithi044 ceph-mon[32404]: pgmap v1441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:57 smithi116 ceph-mon[37097]: pgmap v1441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:08:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:57 smithi171 ceph-mon[37789]: pgmap v1441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:09:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:08:59 smithi044 ceph-mon[32404]: pgmap v1442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:09:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:08:59 smithi116 ceph-mon[37097]: pgmap v1442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:09:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:08:59 smithi171 ceph-mon[37789]: pgmap v1442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:09:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1685.592213 secs 2024-07-26T09:09:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1685.592213 secs 2024-07-26T09:09:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1685.592213 secs 2024-07-26T09:09:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:01 smithi044 ceph-mon[32404]: pgmap v1443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:01 smithi116 ceph-mon[37097]: pgmap v1443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:01 smithi171 ceph-mon[37789]: pgmap v1443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:03 smithi044 ceph-mon[32404]: pgmap v1444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:03 smithi116 ceph-mon[37097]: pgmap v1444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:03 smithi171 ceph-mon[37789]: pgmap v1444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:05 smithi044 ceph-mon[32404]: pgmap v1445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1690.592313 secs 2024-07-26T09:09:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:09:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:05 smithi116 ceph-mon[37097]: pgmap v1445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1690.592313 secs 2024-07-26T09:09:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:09:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:05 smithi171 ceph-mon[37789]: pgmap v1445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1690.592313 secs 2024-07-26T09:09:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:09:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:06 smithi044 ceph-mon[32404]: pgmap v1446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:09:07.302 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:06 smithi044 ceph-mon[32404]: pgmap v1447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:09:07.303 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:06 smithi116 ceph-mon[37097]: pgmap v1446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:09:07.303 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:06 smithi116 ceph-mon[37097]: pgmap v1447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:09:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:06 smithi171 ceph-mon[37789]: pgmap v1446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:09:07.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:06 smithi171 ceph-mon[37789]: pgmap v1447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:09:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:08 smithi044 ceph-mon[32404]: pgmap v1448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:09:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:08 smithi116 ceph-mon[37097]: pgmap v1448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:09:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:08 smithi171 ceph-mon[37789]: pgmap v1448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:09:10.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:10.648+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:09:10.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:10.648+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:09:10.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:10.648+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:10 smithi044 ceph-mon[32404]: pgmap v1449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1695.592468 secs 2024-07-26T09:09:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:10 smithi116 ceph-mon[37097]: pgmap v1449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1695.592468 secs 2024-07-26T09:09:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:10 smithi171 ceph-mon[37789]: pgmap v1449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1695.592468 secs 2024-07-26T09:09:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:12 smithi044 ceph-mon[32404]: pgmap v1450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:12 smithi116 ceph-mon[37097]: pgmap v1450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:12 smithi171 ceph-mon[37789]: pgmap v1450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:14 smithi044 ceph-mon[32404]: pgmap v1451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:14 smithi116 ceph-mon[37097]: pgmap v1451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:14 smithi171 ceph-mon[37789]: pgmap v1451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:09:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1700.592658 secs 2024-07-26T09:09:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1700.592658 secs 2024-07-26T09:09:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1700.592658 secs 2024-07-26T09:09:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:16 smithi044 ceph-mon[32404]: pgmap v1452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:09:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:16 smithi116 ceph-mon[37097]: pgmap v1452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:09:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:16 smithi171 ceph-mon[37789]: pgmap v1452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:09:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:18 smithi044 ceph-mon[32404]: pgmap v1453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:18 smithi116 ceph-mon[37097]: pgmap v1453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:18 smithi171 ceph-mon[37789]: pgmap v1453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:20 smithi044 ceph-mon[32404]: pgmap v1454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1705.592805 secs 2024-07-26T09:09:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:20 smithi116 ceph-mon[37097]: pgmap v1454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1705.592805 secs 2024-07-26T09:09:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:20 smithi171 ceph-mon[37789]: pgmap v1454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1705.592805 secs 2024-07-26T09:09:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:22 smithi044 ceph-mon[32404]: pgmap v1455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:22 smithi116 ceph-mon[37097]: pgmap v1455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:22 smithi171 ceph-mon[37789]: pgmap v1455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:24 smithi044 ceph-mon[32404]: pgmap v1456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:24 smithi116 ceph-mon[37097]: pgmap v1456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:24 smithi171 ceph-mon[37789]: pgmap v1456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:25.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:25.656+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:09:25.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:25.656+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:25.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:25.656+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:09:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1710.592896 secs 2024-07-26T09:09:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1710.592896 secs 2024-07-26T09:09:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1710.592896 secs 2024-07-26T09:09:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:27 smithi044 ceph-mon[32404]: pgmap v1457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:09:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:27 smithi116 ceph-mon[37097]: pgmap v1457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:09:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:27 smithi171 ceph-mon[37789]: pgmap v1457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:09:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:28 smithi116 ceph-mon[37097]: pgmap v1458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:28 smithi171 ceph-mon[37789]: pgmap v1458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:28 smithi044 ceph-mon[32404]: pgmap v1458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:30 smithi171 ceph-mon[37789]: pgmap v1459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:09:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1715.593052 secs 2024-07-26T09:09:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:30 smithi044 ceph-mon[32404]: pgmap v1459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:09:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1715.593052 secs 2024-07-26T09:09:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:30 smithi116 ceph-mon[37097]: pgmap v1459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:09:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1715.593052 secs 2024-07-26T09:09:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:32 smithi044 ceph-mon[32404]: pgmap v1460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:32 smithi116 ceph-mon[37097]: pgmap v1460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:32 smithi171 ceph-mon[37789]: pgmap v1460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:09:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:34 smithi044 ceph-mon[32404]: pgmap v1461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:34 smithi116 ceph-mon[37097]: pgmap v1461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:34 smithi171 ceph-mon[37789]: pgmap v1461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1720.593219 secs 2024-07-26T09:09:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1720.593219 secs 2024-07-26T09:09:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1720.593219 secs 2024-07-26T09:09:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:36 smithi044 ceph-mon[32404]: pgmap v1462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:36 smithi116 ceph-mon[37097]: pgmap v1462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:36 smithi171 ceph-mon[37789]: pgmap v1462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:38 smithi044 ceph-mon[32404]: pgmap v1463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:38 smithi116 ceph-mon[37097]: pgmap v1463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:38 smithi171 ceph-mon[37789]: pgmap v1463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:40.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:40.659+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:09:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:40.659+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:09:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:40.659+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:40 smithi044 ceph-mon[32404]: pgmap v1464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1725.593380 secs 2024-07-26T09:09:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:40 smithi116 ceph-mon[37097]: pgmap v1464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1725.593380 secs 2024-07-26T09:09:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:40 smithi171 ceph-mon[37789]: pgmap v1464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1725.593380 secs 2024-07-26T09:09:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:42 smithi044 ceph-mon[32404]: pgmap v1465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:42 smithi116 ceph-mon[37097]: pgmap v1465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:42 smithi171 ceph-mon[37789]: pgmap v1465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:44 smithi044 ceph-mon[32404]: pgmap v1466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:44 smithi116 ceph-mon[37097]: pgmap v1466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:44 smithi171 ceph-mon[37789]: pgmap v1466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1730.593548 secs 2024-07-26T09:09:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1730.593548 secs 2024-07-26T09:09:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1730.593548 secs 2024-07-26T09:09:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:46 smithi044 ceph-mon[32404]: pgmap v1467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:46 smithi116 ceph-mon[37097]: pgmap v1467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:46 smithi171 ceph-mon[37789]: pgmap v1467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:48 smithi044 ceph-mon[32404]: pgmap v1468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:48 smithi116 ceph-mon[37097]: pgmap v1468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:48 smithi171 ceph-mon[37789]: pgmap v1468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:50 smithi044 ceph-mon[32404]: pgmap v1469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1735.593706 secs 2024-07-26T09:09:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:50 smithi116 ceph-mon[37097]: pgmap v1469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1735.593706 secs 2024-07-26T09:09:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:50 smithi171 ceph-mon[37789]: pgmap v1469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:09:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1735.593706 secs 2024-07-26T09:09:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:52 smithi044 ceph-mon[32404]: pgmap v1470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:52 smithi116 ceph-mon[37097]: pgmap v1470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:52 smithi171 ceph-mon[37789]: pgmap v1470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:54 smithi044 ceph-mon[32404]: pgmap v1471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:54 smithi116 ceph-mon[37097]: pgmap v1471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:54 smithi171 ceph-mon[37789]: pgmap v1471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:55.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:55.662+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:09:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:55.662+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:09:55.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:09:55.662+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1740.593868 secs 2024-07-26T09:09:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1740.593868 secs 2024-07-26T09:09:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1740.593868 secs 2024-07-26T09:09:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:56 smithi044 ceph-mon[32404]: pgmap v1472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:56 smithi116 ceph-mon[37097]: pgmap v1472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:56 smithi171 ceph-mon[37789]: pgmap v1472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:09:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:09:58 smithi044 ceph-mon[32404]: pgmap v1473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:09:58 smithi116 ceph-mon[37097]: pgmap v1473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:09:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:09:58 smithi171 ceph-mon[37789]: pgmap v1473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:00 smithi044 ceph-mon[32404]: pgmap v1474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:10:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1745.594005 secs 2024-07-26T09:10:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:00 smithi116 ceph-mon[37097]: pgmap v1474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:10:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1745.594005 secs 2024-07-26T09:10:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:00 smithi171 ceph-mon[37789]: pgmap v1474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:10:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:10:01.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1745.594005 secs 2024-07-26T09:10:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:02 smithi044 ceph-mon[32404]: pgmap v1475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:02 smithi116 ceph-mon[37097]: pgmap v1475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:02 smithi171 ceph-mon[37789]: pgmap v1475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:04 smithi044 ceph-mon[32404]: pgmap v1476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:04 smithi116 ceph-mon[37097]: pgmap v1476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:04 smithi171 ceph-mon[37789]: pgmap v1476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1750.594150 secs 2024-07-26T09:10:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1750.594150 secs 2024-07-26T09:10:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1750.594150 secs 2024-07-26T09:10:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:06 smithi044 ceph-mon[32404]: pgmap v1477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:06 smithi116 ceph-mon[37097]: pgmap v1477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:06 smithi171 ceph-mon[37789]: pgmap v1477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:10:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:10:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:10:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:08 smithi044 ceph-mon[32404]: pgmap v1478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:10:09.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:08 smithi044 ceph-mon[32404]: pgmap v1479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:08 smithi116 ceph-mon[37097]: pgmap v1478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:10:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:08 smithi116 ceph-mon[37097]: pgmap v1479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:08 smithi171 ceph-mon[37789]: pgmap v1478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:10:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:08 smithi171 ceph-mon[37789]: pgmap v1479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:10.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:10.664+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:10:10.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:10.664+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:10:10.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:10.664+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:10 smithi044 ceph-mon[32404]: pgmap v1480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1755.594288 secs 2024-07-26T09:10:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:10 smithi116 ceph-mon[37097]: pgmap v1480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1755.594288 secs 2024-07-26T09:10:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:10 smithi171 ceph-mon[37789]: pgmap v1480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1755.594288 secs 2024-07-26T09:10:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:12 smithi044 ceph-mon[32404]: pgmap v1481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:12 smithi116 ceph-mon[37097]: pgmap v1481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:12 smithi171 ceph-mon[37789]: pgmap v1481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:15 smithi044 ceph-mon[32404]: pgmap v1482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:15 smithi116 ceph-mon[37097]: pgmap v1482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:15 smithi171 ceph-mon[37789]: pgmap v1482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1760.594436 secs 2024-07-26T09:10:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1760.594436 secs 2024-07-26T09:10:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1760.594436 secs 2024-07-26T09:10:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:17 smithi116 ceph-mon[37097]: pgmap v1483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:17 smithi171 ceph-mon[37789]: pgmap v1483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:17 smithi044 ceph-mon[32404]: pgmap v1483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:10:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:18 smithi171 ceph-mon[37789]: pgmap v1484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:10:18.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:18 smithi044 ceph-mon[32404]: pgmap v1484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:10:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:18 smithi116 ceph-mon[37097]: pgmap v1484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:10:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:20 smithi171 ceph-mon[37789]: pgmap v1485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:20 smithi044 ceph-mon[32404]: pgmap v1485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:20 smithi116 ceph-mon[37097]: pgmap v1485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1765.594590 secs 2024-07-26T09:10:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1765.594590 secs 2024-07-26T09:10:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1765.594590 secs 2024-07-26T09:10:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:22 smithi171 ceph-mon[37789]: pgmap v1486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:22 smithi044 ceph-mon[32404]: pgmap v1486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:22 smithi116 ceph-mon[37097]: pgmap v1486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:24 smithi171 ceph-mon[37789]: pgmap v1487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:24 smithi044 ceph-mon[32404]: pgmap v1487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:24 smithi116 ceph-mon[37097]: pgmap v1487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:25.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:25.672+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:10:25.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:25.672+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:10:25.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:25.672+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:26 smithi171 ceph-mon[37789]: pgmap v1488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1770.594736 secs 2024-07-26T09:10:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:26 smithi044 ceph-mon[32404]: pgmap v1488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1770.594736 secs 2024-07-26T09:10:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:26 smithi116 ceph-mon[37097]: pgmap v1488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1770.594736 secs 2024-07-26T09:10:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:28 smithi044 ceph-mon[32404]: pgmap v1489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:28 smithi171 ceph-mon[37789]: pgmap v1489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:28 smithi116 ceph-mon[37097]: pgmap v1489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:30 smithi171 ceph-mon[37789]: pgmap v1490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:30 smithi044 ceph-mon[32404]: pgmap v1490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:30 smithi116 ceph-mon[37097]: pgmap v1490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1775.594872 secs 2024-07-26T09:10:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1775.594872 secs 2024-07-26T09:10:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1775.594872 secs 2024-07-26T09:10:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:32 smithi044 ceph-mon[32404]: pgmap v1491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:32 smithi116 ceph-mon[37097]: pgmap v1491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:32 smithi171 ceph-mon[37789]: pgmap v1491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:34 smithi044 ceph-mon[32404]: pgmap v1492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:34 smithi116 ceph-mon[37097]: pgmap v1492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:34 smithi171 ceph-mon[37789]: pgmap v1492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:36 smithi044 ceph-mon[32404]: pgmap v1493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1780.595026 secs 2024-07-26T09:10:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:36 smithi116 ceph-mon[37097]: pgmap v1493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1780.595026 secs 2024-07-26T09:10:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:36 smithi171 ceph-mon[37789]: pgmap v1493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1780.595026 secs 2024-07-26T09:10:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:38 smithi044 ceph-mon[32404]: pgmap v1494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:38 smithi116 ceph-mon[37097]: pgmap v1494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:38 smithi171 ceph-mon[37789]: pgmap v1494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:40 smithi044 ceph-mon[32404]: pgmap v1495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:40 smithi116 ceph-mon[37097]: pgmap v1495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:40 smithi171 ceph-mon[37789]: pgmap v1495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:40.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:40.675+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:10:40.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:40.675+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:10:40.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:40.675+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1785.595169 secs 2024-07-26T09:10:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1785.595169 secs 2024-07-26T09:10:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1785.595169 secs 2024-07-26T09:10:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:42 smithi044 ceph-mon[32404]: pgmap v1496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:42 smithi116 ceph-mon[37097]: pgmap v1496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:42 smithi171 ceph-mon[37789]: pgmap v1496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:44.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:44 smithi044 ceph-mon[32404]: pgmap v1497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:44 smithi116 ceph-mon[37097]: pgmap v1497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:44 smithi171 ceph-mon[37789]: pgmap v1497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:46 smithi044 ceph-mon[32404]: pgmap v1498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1790.595295 secs 2024-07-26T09:10:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:46 smithi116 ceph-mon[37097]: pgmap v1498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1790.595295 secs 2024-07-26T09:10:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:46 smithi171 ceph-mon[37789]: pgmap v1498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1790.595295 secs 2024-07-26T09:10:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:48 smithi044 ceph-mon[32404]: pgmap v1499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:48 smithi116 ceph-mon[37097]: pgmap v1499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:48 smithi171 ceph-mon[37789]: pgmap v1499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:50 smithi044 ceph-mon[32404]: pgmap v1500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:50 smithi116 ceph-mon[37097]: pgmap v1500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:50 smithi171 ceph-mon[37789]: pgmap v1500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:51.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1795.595460 secs 2024-07-26T09:10:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1795.595460 secs 2024-07-26T09:10:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1795.595460 secs 2024-07-26T09:10:52.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:52 smithi171 ceph-mon[37789]: pgmap v1501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:52 smithi044 ceph-mon[32404]: pgmap v1501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:52 smithi116 ceph-mon[37097]: pgmap v1501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:54.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:54 smithi044 ceph-mon[32404]: pgmap v1502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:54 smithi116 ceph-mon[37097]: pgmap v1502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:54.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:54 smithi171 ceph-mon[37789]: pgmap v1502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:10:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:55.682+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:10:55.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:55.682+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:10:55.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:10:55.682+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:56 smithi044 ceph-mon[32404]: pgmap v1503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1800.595661 secs 2024-07-26T09:10:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:56 smithi116 ceph-mon[37097]: pgmap v1503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1800.595661 secs 2024-07-26T09:10:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:56 smithi171 ceph-mon[37789]: pgmap v1503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:10:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1800.595661 secs 2024-07-26T09:10:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:10:58 smithi044 ceph-mon[32404]: pgmap v1504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:10:58 smithi116 ceph-mon[37097]: pgmap v1504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:10:58 smithi171 ceph-mon[37789]: pgmap v1504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:00 smithi044 ceph-mon[32404]: pgmap v1505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:00 smithi116 ceph-mon[37097]: pgmap v1505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:00 smithi171 ceph-mon[37789]: pgmap v1505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1805.595817 secs 2024-07-26T09:11:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1805.595817 secs 2024-07-26T09:11:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1805.595817 secs 2024-07-26T09:11:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:02 smithi044 ceph-mon[32404]: pgmap v1506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:02 smithi116 ceph-mon[37097]: pgmap v1506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:02 smithi171 ceph-mon[37789]: pgmap v1506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:04 smithi044 ceph-mon[32404]: pgmap v1507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:04 smithi116 ceph-mon[37097]: pgmap v1507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:04 smithi171 ceph-mon[37789]: pgmap v1507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:06 smithi044 ceph-mon[32404]: pgmap v1508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1810.595983 secs 2024-07-26T09:11:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:06 smithi116 ceph-mon[37097]: pgmap v1508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1810.595983 secs 2024-07-26T09:11:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:06 smithi171 ceph-mon[37789]: pgmap v1508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1810.595983 secs 2024-07-26T09:11:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:08 smithi044 ceph-mon[32404]: pgmap v1509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:08 smithi116 ceph-mon[37097]: pgmap v1509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:08 smithi171 ceph-mon[37789]: pgmap v1509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:09 smithi044 ceph-mon[32404]: pgmap v1510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:11:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:09 smithi044 ceph-mon[32404]: pgmap v1511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:11:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:09 smithi116 ceph-mon[37097]: pgmap v1510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:11:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:09 smithi116 ceph-mon[37097]: pgmap v1511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:11:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:09.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:09 smithi171 ceph-mon[37789]: pgmap v1510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:11:09.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:09 smithi171 ceph-mon[37789]: pgmap v1511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:09.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:11:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:10.685+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:11:10.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:10.686+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:11:10.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:10.686+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1815.596139 secs 2024-07-26T09:11:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:11 smithi171 ceph-mon[37789]: pgmap v1512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1815.596139 secs 2024-07-26T09:11:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:11 smithi044 ceph-mon[32404]: pgmap v1512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1815.596139 secs 2024-07-26T09:11:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:11 smithi116 ceph-mon[37097]: pgmap v1512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:13 smithi171 ceph-mon[37789]: pgmap v1513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:13 smithi044 ceph-mon[32404]: pgmap v1513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:13 smithi116 ceph-mon[37097]: pgmap v1513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T09:11:14.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:11:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf58005ef0 tx=0x7fcf5802a3f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:11:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcf58005ef0 tx=0x7fcf5802a3f0 comp rx=0 tx=0).stop 2024-07-26T09:11:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T09:11:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:11:14.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:11:14.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.680+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:11:14.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.681+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:11:14.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.684+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540c1eb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802a430 tx=0x7fcf5804d000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:11:14.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.982+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:11:14.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.982+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:11:14.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.983+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:11:14.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:14.983+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:11:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:15 smithi171 ceph-mon[37789]: pgmap v1514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:15 smithi044 ceph-mon[32404]: pgmap v1514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:15 smithi116 ceph-mon[37097]: pgmap v1514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:11:16.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1820.596305 secs 2024-07-26T09:11:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1820.596305 secs 2024-07-26T09:11:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1820.596305 secs 2024-07-26T09:11:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:17 smithi044 ceph-mon[32404]: pgmap v1515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:17 smithi116 ceph-mon[37097]: pgmap v1515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:17 smithi171 ceph-mon[37789]: pgmap v1515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:19 smithi044 ceph-mon[32404]: pgmap v1516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:11:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:19 smithi116 ceph-mon[37097]: pgmap v1516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:11:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:19 smithi171 ceph-mon[37789]: pgmap v1516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:11:21.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1825.596465 secs 2024-07-26T09:11:21.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:21 smithi171 ceph-mon[37789]: pgmap v1517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1825.596465 secs 2024-07-26T09:11:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:21 smithi044 ceph-mon[32404]: pgmap v1517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1825.596465 secs 2024-07-26T09:11:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:21 smithi116 ceph-mon[37097]: pgmap v1517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:23 smithi044 ceph-mon[32404]: pgmap v1518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:23 smithi116 ceph-mon[37097]: pgmap v1518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:23 smithi171 ceph-mon[37789]: pgmap v1518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:25.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:25.694+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:11:25.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:25.695+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:11:25.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:25.695+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:25.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:25 smithi044 ceph-mon[32404]: pgmap v1519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:25 smithi116 ceph-mon[37097]: pgmap v1519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:25 smithi171 ceph-mon[37789]: pgmap v1519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1830.596666 secs 2024-07-26T09:11:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1830.596666 secs 2024-07-26T09:11:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1830.596666 secs 2024-07-26T09:11:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:27 smithi044 ceph-mon[32404]: pgmap v1520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:27 smithi116 ceph-mon[37097]: pgmap v1520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:27 smithi171 ceph-mon[37789]: pgmap v1520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:29 smithi044 ceph-mon[32404]: pgmap v1521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:29 smithi116 ceph-mon[37097]: pgmap v1521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:29 smithi171 ceph-mon[37789]: pgmap v1521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1835.596760 secs 2024-07-26T09:11:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:31 smithi044 ceph-mon[32404]: pgmap v1522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1835.596760 secs 2024-07-26T09:11:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:31 smithi116 ceph-mon[37097]: pgmap v1522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1835.596760 secs 2024-07-26T09:11:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:31 smithi171 ceph-mon[37789]: pgmap v1522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:33 smithi044 ceph-mon[32404]: pgmap v1523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:33 smithi116 ceph-mon[37097]: pgmap v1523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:33 smithi171 ceph-mon[37789]: pgmap v1523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:35 smithi044 ceph-mon[32404]: pgmap v1524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:35 smithi116 ceph-mon[37097]: pgmap v1524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:35 smithi171 ceph-mon[37789]: pgmap v1524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1840.596869 secs 2024-07-26T09:11:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1840.596869 secs 2024-07-26T09:11:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1840.596869 secs 2024-07-26T09:11:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:37 smithi044 ceph-mon[32404]: pgmap v1525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:37 smithi116 ceph-mon[37097]: pgmap v1525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:37.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:37 smithi171 ceph-mon[37789]: pgmap v1525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:39 smithi044 ceph-mon[32404]: pgmap v1526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:39 smithi116 ceph-mon[37097]: pgmap v1526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:39 smithi171 ceph-mon[37789]: pgmap v1526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:40.697+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:11:40.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:40.697+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:11:40.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:40.697+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1845.596957 secs 2024-07-26T09:11:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:41 smithi044 ceph-mon[32404]: pgmap v1527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1845.596957 secs 2024-07-26T09:11:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:41 smithi116 ceph-mon[37097]: pgmap v1527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1845.596957 secs 2024-07-26T09:11:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:41 smithi171 ceph-mon[37789]: pgmap v1527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:43 smithi044 ceph-mon[32404]: pgmap v1528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:43 smithi116 ceph-mon[37097]: pgmap v1528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:43 smithi171 ceph-mon[37789]: pgmap v1528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:45 smithi044 ceph-mon[32404]: pgmap v1529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:45 smithi116 ceph-mon[37097]: pgmap v1529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:45 smithi171 ceph-mon[37789]: pgmap v1529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1850.597117 secs 2024-07-26T09:11:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1850.597117 secs 2024-07-26T09:11:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1850.597117 secs 2024-07-26T09:11:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:47 smithi044 ceph-mon[32404]: pgmap v1530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:47 smithi116 ceph-mon[37097]: pgmap v1530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:47 smithi171 ceph-mon[37789]: pgmap v1530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:49 smithi044 ceph-mon[32404]: pgmap v1531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:49 smithi116 ceph-mon[37097]: pgmap v1531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:49 smithi171 ceph-mon[37789]: pgmap v1531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1855.597243 secs 2024-07-26T09:11:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:51 smithi044 ceph-mon[32404]: pgmap v1532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1855.597243 secs 2024-07-26T09:11:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:51 smithi116 ceph-mon[37097]: pgmap v1532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1855.597243 secs 2024-07-26T09:11:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:51 smithi171 ceph-mon[37789]: pgmap v1532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:53 smithi044 ceph-mon[32404]: pgmap v1533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:53 smithi116 ceph-mon[37097]: pgmap v1533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:53 smithi171 ceph-mon[37789]: pgmap v1533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:11:55.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:55.706+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:11:55.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:55.706+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:55.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:11:55.706+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:11:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:55 smithi044 ceph-mon[32404]: pgmap v1534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:55 smithi116 ceph-mon[37097]: pgmap v1534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:55 smithi171 ceph-mon[37789]: pgmap v1534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:11:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1860.597405 secs 2024-07-26T09:11:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1860.597405 secs 2024-07-26T09:11:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1860.597405 secs 2024-07-26T09:11:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:57 smithi044 ceph-mon[32404]: pgmap v1535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:57 smithi116 ceph-mon[37097]: pgmap v1535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:57 smithi171 ceph-mon[37789]: pgmap v1535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:11:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:11:59 smithi044 ceph-mon[32404]: pgmap v1536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:11:59 smithi116 ceph-mon[37097]: pgmap v1536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:11:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:11:59 smithi171 ceph-mon[37789]: pgmap v1536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:12:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1865.597552 secs 2024-07-26T09:12:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:01 smithi044 ceph-mon[32404]: pgmap v1537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1865.597552 secs 2024-07-26T09:12:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:01 smithi116 ceph-mon[37097]: pgmap v1537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1865.597552 secs 2024-07-26T09:12:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:01 smithi171 ceph-mon[37789]: pgmap v1537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:03 smithi044 ceph-mon[32404]: pgmap v1538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:03 smithi116 ceph-mon[37097]: pgmap v1538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:03 smithi171 ceph-mon[37789]: pgmap v1538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:05 smithi116 ceph-mon[37097]: pgmap v1539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:05 smithi171 ceph-mon[37789]: pgmap v1539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:05 smithi044 ceph-mon[32404]: pgmap v1539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1870.597702 secs 2024-07-26T09:12:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1870.597702 secs 2024-07-26T09:12:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1870.597702 secs 2024-07-26T09:12:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:07 smithi171 ceph-mon[37789]: pgmap v1540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:07 smithi044 ceph-mon[32404]: pgmap v1540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:07 smithi116 ceph-mon[37097]: pgmap v1540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: pgmap v1541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: pgmap v1541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:12:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: pgmap v1541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:12:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:10.709+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:12:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:10.709+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:10.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:10.709+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:12:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:12:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:12:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: pgmap v1542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:12:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: pgmap v1543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:12:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:12:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:12:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: pgmap v1542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:12:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: pgmap v1543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:12:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:12:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:12:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: pgmap v1542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:12:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: pgmap v1543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:11.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:12:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1875.597795 secs 2024-07-26T09:12:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1875.597795 secs 2024-07-26T09:12:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1875.597795 secs 2024-07-26T09:12:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:12 smithi171 ceph-mon[37789]: pgmap v1544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:12:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:12 smithi044 ceph-mon[32404]: pgmap v1544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:12:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:12 smithi116 ceph-mon[37097]: pgmap v1544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:12:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:14 smithi171 ceph-mon[37789]: pgmap v1545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:14 smithi044 ceph-mon[32404]: pgmap v1545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:14 smithi116 ceph-mon[37097]: pgmap v1545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1880.597889 secs 2024-07-26T09:12:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:16 smithi171 ceph-mon[37789]: pgmap v1546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1880.597889 secs 2024-07-26T09:12:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:16 smithi044 ceph-mon[32404]: pgmap v1546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1880.597889 secs 2024-07-26T09:12:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:16 smithi116 ceph-mon[37097]: pgmap v1546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:18 smithi171 ceph-mon[37789]: pgmap v1547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:12:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:18 smithi044 ceph-mon[32404]: pgmap v1547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:12:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:18 smithi116 ceph-mon[37097]: pgmap v1547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:12:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:20 smithi044 ceph-mon[32404]: pgmap v1548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:20 smithi116 ceph-mon[37097]: pgmap v1548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:20 smithi171 ceph-mon[37789]: pgmap v1548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:21.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1885.598029 secs 2024-07-26T09:12:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1885.598029 secs 2024-07-26T09:12:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1885.598029 secs 2024-07-26T09:12:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:22 smithi044 ceph-mon[32404]: pgmap v1549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:22 smithi116 ceph-mon[37097]: pgmap v1549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:22 smithi171 ceph-mon[37789]: pgmap v1549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:24 smithi044 ceph-mon[32404]: pgmap v1550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:24 smithi116 ceph-mon[37097]: pgmap v1550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:24 smithi171 ceph-mon[37789]: pgmap v1550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:25.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:25.718+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:12:25.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:25.718+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:25.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:25.718+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:12:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1890.598190 secs 2024-07-26T09:12:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:26 smithi044 ceph-mon[32404]: pgmap v1551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1890.598190 secs 2024-07-26T09:12:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:26 smithi116 ceph-mon[37097]: pgmap v1551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1890.598190 secs 2024-07-26T09:12:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:26 smithi171 ceph-mon[37789]: pgmap v1551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:28 smithi044 ceph-mon[32404]: pgmap v1552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:28 smithi116 ceph-mon[37097]: pgmap v1552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:28 smithi171 ceph-mon[37789]: pgmap v1552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:30 smithi044 ceph-mon[32404]: pgmap v1553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:30 smithi116 ceph-mon[37097]: pgmap v1553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:30 smithi171 ceph-mon[37789]: pgmap v1553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:12:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1895.598308 secs 2024-07-26T09:12:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1895.598308 secs 2024-07-26T09:12:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1895.598308 secs 2024-07-26T09:12:32.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:32 smithi044 ceph-mon[32404]: pgmap v1554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:32 smithi116 ceph-mon[37097]: pgmap v1554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:32 smithi171 ceph-mon[37789]: pgmap v1554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:34 smithi044 ceph-mon[32404]: pgmap v1555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:34 smithi116 ceph-mon[37097]: pgmap v1555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:34 smithi171 ceph-mon[37789]: pgmap v1555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1900.598471 secs 2024-07-26T09:12:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1900.598471 secs 2024-07-26T09:12:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1900.598471 secs 2024-07-26T09:12:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:36 smithi044 ceph-mon[32404]: pgmap v1556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:36 smithi116 ceph-mon[37097]: pgmap v1556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:36 smithi171 ceph-mon[37789]: pgmap v1556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:38 smithi044 ceph-mon[32404]: pgmap v1557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:38 smithi116 ceph-mon[37097]: pgmap v1557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:38 smithi171 ceph-mon[37789]: pgmap v1557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:12:40.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:40.721+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:12:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:40.721+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:40.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:40.721+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:12:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:40 smithi044 ceph-mon[32404]: pgmap v1558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:12:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1905.598662 secs 2024-07-26T09:12:41.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:40 smithi116 ceph-mon[37097]: pgmap v1558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:12:41.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1905.598662 secs 2024-07-26T09:12:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:40 smithi171 ceph-mon[37789]: pgmap v1558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:12:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1905.598662 secs 2024-07-26T09:12:43.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:42 smithi044 ceph-mon[32404]: pgmap v1559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:12:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:42 smithi116 ceph-mon[37097]: pgmap v1559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:12:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:42 smithi171 ceph-mon[37789]: pgmap v1559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:12:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:44 smithi044 ceph-mon[32404]: pgmap v1560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:44 smithi116 ceph-mon[37097]: pgmap v1560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:44 smithi171 ceph-mon[37789]: pgmap v1560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:12:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1910.598800 secs 2024-07-26T09:12:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1910.598800 secs 2024-07-26T09:12:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1910.598800 secs 2024-07-26T09:12:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:46 smithi044 ceph-mon[32404]: pgmap v1561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:12:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:46 smithi116 ceph-mon[37097]: pgmap v1561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:12:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:46 smithi171 ceph-mon[37789]: pgmap v1561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:12:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:48 smithi044 ceph-mon[32404]: pgmap v1562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:12:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:48 smithi116 ceph-mon[37097]: pgmap v1562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:12:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:48 smithi171 ceph-mon[37789]: pgmap v1562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:12:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:50 smithi044 ceph-mon[32404]: pgmap v1563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1915.598942 secs 2024-07-26T09:12:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:50 smithi116 ceph-mon[37097]: pgmap v1563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1915.598942 secs 2024-07-26T09:12:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:50 smithi171 ceph-mon[37789]: pgmap v1563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:12:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1915.598942 secs 2024-07-26T09:12:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:52 smithi044 ceph-mon[32404]: pgmap v1564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:52 smithi116 ceph-mon[37097]: pgmap v1564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:52 smithi171 ceph-mon[37789]: pgmap v1564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:12:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:54 smithi044 ceph-mon[32404]: pgmap v1565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:54 smithi116 ceph-mon[37097]: pgmap v1565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:55.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:54 smithi171 ceph-mon[37789]: pgmap v1565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:12:55.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:55.729+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:12:55.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:55.729+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:12:55.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:12:55.729+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1920.599061 secs 2024-07-26T09:12:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1920.599061 secs 2024-07-26T09:12:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1920.599061 secs 2024-07-26T09:12:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:56 smithi044 ceph-mon[32404]: pgmap v1566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:56 smithi116 ceph-mon[37097]: pgmap v1566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:56 smithi171 ceph-mon[37789]: pgmap v1566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:12:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:12:58 smithi044 ceph-mon[32404]: pgmap v1567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:12:58 smithi116 ceph-mon[37097]: pgmap v1567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:12:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:12:58 smithi171 ceph-mon[37789]: pgmap v1567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:00 smithi044 ceph-mon[32404]: pgmap v1568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1925.599201 secs 2024-07-26T09:13:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:00 smithi116 ceph-mon[37097]: pgmap v1568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1925.599201 secs 2024-07-26T09:13:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:00 smithi171 ceph-mon[37789]: pgmap v1568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1925.599201 secs 2024-07-26T09:13:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:02 smithi044 ceph-mon[32404]: pgmap v1569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:02 smithi116 ceph-mon[37097]: pgmap v1569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:02 smithi171 ceph-mon[37789]: pgmap v1569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:13:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:04 smithi044 ceph-mon[32404]: pgmap v1570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:13:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:04 smithi116 ceph-mon[37097]: pgmap v1570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:13:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:04 smithi171 ceph-mon[37789]: pgmap v1570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:13:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1930.599362 secs 2024-07-26T09:13:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1930.599362 secs 2024-07-26T09:13:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1930.599362 secs 2024-07-26T09:13:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:06 smithi171 ceph-mon[37789]: pgmap v1571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:06 smithi044 ceph-mon[32404]: pgmap v1571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:06 smithi116 ceph-mon[37097]: pgmap v1571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:08 smithi171 ceph-mon[37789]: pgmap v1572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:08 smithi044 ceph-mon[32404]: pgmap v1572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:08 smithi116 ceph-mon[37097]: pgmap v1572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:10.731+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:13:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:10.732+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:10.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:10.732+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:13:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: pgmap v1573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1935.599433 secs 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: pgmap v1573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1935.599433 secs 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: pgmap v1573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1935.599433 secs 2024-07-26T09:13:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:11.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:13:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:13:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: pgmap v1574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: pgmap v1575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:13:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: pgmap v1574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: pgmap v1575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: pgmap v1574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: pgmap v1575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:13:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:13 smithi044 ceph-mon[32404]: pgmap v1576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:13 smithi116 ceph-mon[37097]: pgmap v1576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:13 smithi171 ceph-mon[37789]: pgmap v1576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:15 smithi044 ceph-mon[32404]: pgmap v1577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1940.599579 secs 2024-07-26T09:13:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:15 smithi116 ceph-mon[37097]: pgmap v1577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1940.599579 secs 2024-07-26T09:13:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:15 smithi171 ceph-mon[37789]: pgmap v1577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1940.599579 secs 2024-07-26T09:13:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:17 smithi044 ceph-mon[32404]: pgmap v1578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:17 smithi116 ceph-mon[37097]: pgmap v1578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:17 smithi171 ceph-mon[37789]: pgmap v1578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:19 smithi044 ceph-mon[32404]: pgmap v1579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:19 smithi116 ceph-mon[37097]: pgmap v1579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:19 smithi171 ceph-mon[37789]: pgmap v1579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:13:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1945.599731 secs 2024-07-26T09:13:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1945.599731 secs 2024-07-26T09:13:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1945.599731 secs 2024-07-26T09:13:22.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:21 smithi171 ceph-mon[37789]: pgmap v1580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:13:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:21 smithi044 ceph-mon[32404]: pgmap v1580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:13:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:21 smithi116 ceph-mon[37097]: pgmap v1580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:13:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:23 smithi044 ceph-mon[32404]: pgmap v1581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:23 smithi116 ceph-mon[37097]: pgmap v1581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:23 smithi171 ceph-mon[37789]: pgmap v1581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:25.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:25.736+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:13:25.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:25.736+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:25.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:25.736+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:13:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:25 smithi044 ceph-mon[32404]: pgmap v1582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1950.599884 secs 2024-07-26T09:13:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:25 smithi116 ceph-mon[37097]: pgmap v1582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1950.599884 secs 2024-07-26T09:13:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:25 smithi171 ceph-mon[37789]: pgmap v1582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1950.599884 secs 2024-07-26T09:13:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:27 smithi044 ceph-mon[32404]: pgmap v1583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:27 smithi116 ceph-mon[37097]: pgmap v1583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:27 smithi171 ceph-mon[37789]: pgmap v1583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:30.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:29 smithi044 ceph-mon[32404]: pgmap v1584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:29 smithi116 ceph-mon[37097]: pgmap v1584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:29 smithi171 ceph-mon[37789]: pgmap v1584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1955.600020 secs 2024-07-26T09:13:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1955.600020 secs 2024-07-26T09:13:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1955.600020 secs 2024-07-26T09:13:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:31 smithi044 ceph-mon[32404]: pgmap v1585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:31 smithi116 ceph-mon[37097]: pgmap v1585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:31 smithi171 ceph-mon[37789]: pgmap v1585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:33 smithi044 ceph-mon[32404]: pgmap v1586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:33 smithi116 ceph-mon[37097]: pgmap v1586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:33 smithi171 ceph-mon[37789]: pgmap v1586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:35 smithi044 ceph-mon[32404]: pgmap v1587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1960.600130 secs 2024-07-26T09:13:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:35 smithi116 ceph-mon[37097]: pgmap v1587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1960.600130 secs 2024-07-26T09:13:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:35 smithi171 ceph-mon[37789]: pgmap v1587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1960.600130 secs 2024-07-26T09:13:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:37 smithi044 ceph-mon[32404]: pgmap v1588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:37 smithi116 ceph-mon[37097]: pgmap v1588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:37 smithi171 ceph-mon[37789]: pgmap v1588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:39 smithi044 ceph-mon[32404]: pgmap v1589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:39 smithi116 ceph-mon[37097]: pgmap v1589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:39 smithi171 ceph-mon[37789]: pgmap v1589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:40.740+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:13:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:40.740+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:40.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:40.740+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:13:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1965.600273 secs 2024-07-26T09:13:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1965.600273 secs 2024-07-26T09:13:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1965.600273 secs 2024-07-26T09:13:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:41 smithi044 ceph-mon[32404]: pgmap v1590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:41 smithi116 ceph-mon[37097]: pgmap v1590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:41 smithi171 ceph-mon[37789]: pgmap v1590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:43 smithi044 ceph-mon[32404]: pgmap v1591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:43 smithi116 ceph-mon[37097]: pgmap v1591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:43 smithi171 ceph-mon[37789]: pgmap v1591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:45 smithi044 ceph-mon[32404]: pgmap v1592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1970.600441 secs 2024-07-26T09:13:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:45 smithi116 ceph-mon[37097]: pgmap v1592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1970.600441 secs 2024-07-26T09:13:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:45 smithi171 ceph-mon[37789]: pgmap v1592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1970.600441 secs 2024-07-26T09:13:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:47 smithi044 ceph-mon[32404]: pgmap v1593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:47 smithi116 ceph-mon[37097]: pgmap v1593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:47 smithi171 ceph-mon[37789]: pgmap v1593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:50 smithi044 ceph-mon[32404]: pgmap v1594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:50 smithi116 ceph-mon[37097]: pgmap v1594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:50 smithi171 ceph-mon[37789]: pgmap v1594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1975.600587 secs 2024-07-26T09:13:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1975.600587 secs 2024-07-26T09:13:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1975.600587 secs 2024-07-26T09:13:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:52 smithi171 ceph-mon[37789]: pgmap v1595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:52 smithi044 ceph-mon[32404]: pgmap v1595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:52 smithi116 ceph-mon[37097]: pgmap v1595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:13:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:54 smithi171 ceph-mon[37789]: pgmap v1596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:54 smithi044 ceph-mon[32404]: pgmap v1596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:54 smithi116 ceph-mon[37097]: pgmap v1596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:55.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:55.749+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:13:55.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:55.749+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:55.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:13:55.749+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:13:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:56 smithi171 ceph-mon[37789]: pgmap v1597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1980.600736 secs 2024-07-26T09:13:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:56 smithi044 ceph-mon[32404]: pgmap v1597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1980.600736 secs 2024-07-26T09:13:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:56 smithi116 ceph-mon[37097]: pgmap v1597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:13:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1980.600736 secs 2024-07-26T09:13:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:13:58 smithi171 ceph-mon[37789]: pgmap v1598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:13:58 smithi044 ceph-mon[32404]: pgmap v1598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:13:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:13:58 smithi116 ceph-mon[37097]: pgmap v1598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:00 smithi171 ceph-mon[37789]: pgmap v1599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:00 smithi044 ceph-mon[32404]: pgmap v1599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:00 smithi116 ceph-mon[37097]: pgmap v1599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1985.600901 secs 2024-07-26T09:14:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1985.600901 secs 2024-07-26T09:14:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1985.600901 secs 2024-07-26T09:14:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:02 smithi171 ceph-mon[37789]: pgmap v1600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:02 smithi044 ceph-mon[32404]: pgmap v1600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:02 smithi116 ceph-mon[37097]: pgmap v1600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:03 smithi044 ceph-mon[32404]: pgmap v1601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:03 smithi116 ceph-mon[37097]: pgmap v1601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:03 smithi171 ceph-mon[37789]: pgmap v1601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:05 smithi044 ceph-mon[32404]: pgmap v1602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1990.601061 secs 2024-07-26T09:14:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:05 smithi116 ceph-mon[37097]: pgmap v1602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1990.601061 secs 2024-07-26T09:14:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:05 smithi171 ceph-mon[37789]: pgmap v1602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:14:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1990.601061 secs 2024-07-26T09:14:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:07 smithi044 ceph-mon[32404]: pgmap v1603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:07 smithi116 ceph-mon[37097]: pgmap v1603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:07 smithi171 ceph-mon[37789]: pgmap v1603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:09 smithi044 ceph-mon[32404]: pgmap v1604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:09 smithi116 ceph-mon[37097]: pgmap v1604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:09 smithi171 ceph-mon[37789]: pgmap v1604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:10.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:10.751+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:14:10.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:10.751+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:10.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:10.751+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:14:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 1995.601188 secs 2024-07-26T09:14:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 1995.601188 secs 2024-07-26T09:14:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 1995.601188 secs 2024-07-26T09:14:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:11 smithi044 ceph-mon[32404]: pgmap v1605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:11 smithi116 ceph-mon[37097]: pgmap v1605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:11 smithi171 ceph-mon[37789]: pgmap v1605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:14:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:14:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:14:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:13 smithi044 ceph-mon[32404]: pgmap v1606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:14:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:13 smithi044 ceph-mon[32404]: pgmap v1607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:13 smithi116 ceph-mon[37097]: pgmap v1606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:14:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:13 smithi116 ceph-mon[37097]: pgmap v1607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:13 smithi171 ceph-mon[37789]: pgmap v1606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:14:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:13 smithi171 ceph-mon[37789]: pgmap v1607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.013+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:14:15.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.013+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:14:15.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.013+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:14:15.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.013+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:14:15.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.013+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:14:15.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.014+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:14:15.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.014+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:14:15.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:15.015+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:14:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:15 smithi044 ceph-mon[32404]: pgmap v1608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2000.601496 secs 2024-07-26T09:14:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:15 smithi116 ceph-mon[37097]: pgmap v1608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2000.601496 secs 2024-07-26T09:14:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:15 smithi171 ceph-mon[37789]: pgmap v1608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2000.601496 secs 2024-07-26T09:14:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:17 smithi044 ceph-mon[32404]: pgmap v1609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:17 smithi116 ceph-mon[37097]: pgmap v1609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:17 smithi171 ceph-mon[37789]: pgmap v1609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:19 smithi044 ceph-mon[32404]: pgmap v1610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:19 smithi116 ceph-mon[37097]: pgmap v1610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:19 smithi171 ceph-mon[37789]: pgmap v1610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2005.601690 secs 2024-07-26T09:14:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2005.601690 secs 2024-07-26T09:14:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2005.601690 secs 2024-07-26T09:14:22.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:21 smithi171 ceph-mon[37789]: pgmap v1611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:21 smithi044 ceph-mon[32404]: pgmap v1611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:21 smithi116 ceph-mon[37097]: pgmap v1611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:14:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:23 smithi044 ceph-mon[32404]: pgmap v1612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:23 smithi116 ceph-mon[37097]: pgmap v1612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:23 smithi171 ceph-mon[37789]: pgmap v1612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:14:25.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:25.758+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:14:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:25.758+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:25.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:25.758+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:14:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:25 smithi044 ceph-mon[32404]: pgmap v1613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2010.601843 secs 2024-07-26T09:14:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:25 smithi116 ceph-mon[37097]: pgmap v1613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2010.601843 secs 2024-07-26T09:14:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:25 smithi171 ceph-mon[37789]: pgmap v1613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2010.601843 secs 2024-07-26T09:14:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:28 smithi044 ceph-mon[32404]: pgmap v1614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:28 smithi116 ceph-mon[37097]: pgmap v1614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:28 smithi171 ceph-mon[37789]: pgmap v1614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:30 smithi044 ceph-mon[32404]: pgmap v1615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:30 smithi116 ceph-mon[37097]: pgmap v1615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:30 smithi171 ceph-mon[37789]: pgmap v1615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2015.601941 secs 2024-07-26T09:14:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2015.601941 secs 2024-07-26T09:14:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2015.601941 secs 2024-07-26T09:14:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:32 smithi171 ceph-mon[37789]: pgmap v1616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:32 smithi044 ceph-mon[32404]: pgmap v1616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:32 smithi116 ceph-mon[37097]: pgmap v1616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:34 smithi171 ceph-mon[37789]: pgmap v1617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:34 smithi044 ceph-mon[32404]: pgmap v1617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:34 smithi116 ceph-mon[37097]: pgmap v1617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:36 smithi171 ceph-mon[37789]: pgmap v1618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2020.602082 secs 2024-07-26T09:14:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:36 smithi044 ceph-mon[32404]: pgmap v1618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2020.602082 secs 2024-07-26T09:14:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:36 smithi116 ceph-mon[37097]: pgmap v1618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2020.602082 secs 2024-07-26T09:14:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:38 smithi171 ceph-mon[37789]: pgmap v1619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:38 smithi044 ceph-mon[32404]: pgmap v1619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:38 smithi116 ceph-mon[37097]: pgmap v1619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:40 smithi171 ceph-mon[37789]: pgmap v1620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:40 smithi044 ceph-mon[32404]: pgmap v1620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:40 smithi116 ceph-mon[37097]: pgmap v1620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:40.761+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:14:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:40.762+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:14:40.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:40.762+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2025.602235 secs 2024-07-26T09:14:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2025.602235 secs 2024-07-26T09:14:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2025.602235 secs 2024-07-26T09:14:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:42 smithi171 ceph-mon[37789]: pgmap v1621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:42 smithi044 ceph-mon[32404]: pgmap v1621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:42 smithi116 ceph-mon[37097]: pgmap v1621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:44 smithi171 ceph-mon[37789]: pgmap v1622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:44 smithi044 ceph-mon[32404]: pgmap v1622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:44 smithi116 ceph-mon[37097]: pgmap v1622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:45 smithi044 ceph-mon[32404]: pgmap v1623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:45 smithi116 ceph-mon[37097]: pgmap v1623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:45 smithi171 ceph-mon[37789]: pgmap v1623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:46.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2030.602392 secs 2024-07-26T09:14:46.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2030.602392 secs 2024-07-26T09:14:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2030.602392 secs 2024-07-26T09:14:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:48 smithi116 ceph-mon[37097]: pgmap v1624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:48 smithi171 ceph-mon[37789]: pgmap v1624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:48 smithi044 ceph-mon[32404]: pgmap v1624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:50 smithi171 ceph-mon[37789]: pgmap v1625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:50 smithi044 ceph-mon[32404]: pgmap v1625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:50 smithi116 ceph-mon[37097]: pgmap v1625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:14:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2035.602523 secs 2024-07-26T09:14:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2035.602523 secs 2024-07-26T09:14:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2035.602523 secs 2024-07-26T09:14:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:52 smithi171 ceph-mon[37789]: pgmap v1626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:52 smithi044 ceph-mon[32404]: pgmap v1626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:52 smithi116 ceph-mon[37097]: pgmap v1626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:14:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:54 smithi171 ceph-mon[37789]: pgmap v1627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:54 smithi044 ceph-mon[32404]: pgmap v1627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:54 smithi116 ceph-mon[37097]: pgmap v1627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:55.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:55.770+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:14:55.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:55.770+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:55.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:14:55.770+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:14:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:56 smithi171 ceph-mon[37789]: pgmap v1628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2040.602749 secs 2024-07-26T09:14:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:56 smithi044 ceph-mon[32404]: pgmap v1628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2040.602749 secs 2024-07-26T09:14:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:56 smithi116 ceph-mon[37097]: pgmap v1628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:14:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2040.602749 secs 2024-07-26T09:14:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:14:58 smithi171 ceph-mon[37789]: pgmap v1629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:14:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:14:58 smithi044 ceph-mon[32404]: pgmap v1629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:14:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:14:58 smithi116 ceph-mon[37097]: pgmap v1629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:00 smithi171 ceph-mon[37789]: pgmap v1630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:00 smithi044 ceph-mon[32404]: pgmap v1630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:00 smithi116 ceph-mon[37097]: pgmap v1630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2045.602910 secs 2024-07-26T09:15:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2045.602910 secs 2024-07-26T09:15:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2045.602910 secs 2024-07-26T09:15:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:02 smithi171 ceph-mon[37789]: pgmap v1631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:02 smithi044 ceph-mon[32404]: pgmap v1631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:02.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:02 smithi116 ceph-mon[37097]: pgmap v1631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:03 smithi044 ceph-mon[32404]: pgmap v1632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:03 smithi116 ceph-mon[37097]: pgmap v1632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:03 smithi171 ceph-mon[37789]: pgmap v1632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:05 smithi044 ceph-mon[32404]: pgmap v1633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:05 smithi116 ceph-mon[37097]: pgmap v1633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:05.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:05 smithi171 ceph-mon[37789]: pgmap v1633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2050.603063 secs 2024-07-26T09:15:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2050.603063 secs 2024-07-26T09:15:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2050.603063 secs 2024-07-26T09:15:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:07 smithi044 ceph-mon[32404]: pgmap v1634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:07 smithi116 ceph-mon[37097]: pgmap v1634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:07 smithi171 ceph-mon[37789]: pgmap v1634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:09 smithi044 ceph-mon[32404]: pgmap v1635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:15:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:09 smithi116 ceph-mon[37097]: pgmap v1635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:15:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:09 smithi171 ceph-mon[37789]: pgmap v1635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:15:10.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:10.773+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:15:10.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:10.773+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:10.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:10.773+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:15:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:11 smithi044 ceph-mon[32404]: pgmap v1636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:15:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2055.603221 secs 2024-07-26T09:15:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:11 smithi116 ceph-mon[37097]: pgmap v1636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:15:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2055.603221 secs 2024-07-26T09:15:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:11 smithi171 ceph-mon[37789]: pgmap v1636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:15:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2055.603221 secs 2024-07-26T09:15:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:13 smithi044 ceph-mon[32404]: pgmap v1637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:13 smithi116 ceph-mon[37097]: pgmap v1637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:13 smithi171 ceph-mon[37789]: pgmap v1637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:15:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:15:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:14 smithi044 ceph-mon[32404]: pgmap v1638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:14 smithi044 ceph-mon[32404]: pgmap v1639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:15:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:15:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:15:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:14 smithi116 ceph-mon[37097]: pgmap v1638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:14 smithi116 ceph-mon[37097]: pgmap v1639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:15:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:15:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:15:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:14 smithi171 ceph-mon[37789]: pgmap v1638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:15:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:14 smithi171 ceph-mon[37789]: pgmap v1639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:15:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:15:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2060.603371 secs 2024-07-26T09:15:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2060.603371 secs 2024-07-26T09:15:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2060.603371 secs 2024-07-26T09:15:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:16 smithi044 ceph-mon[32404]: pgmap v1640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:15:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:16 smithi116 ceph-mon[37097]: pgmap v1640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:15:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:16 smithi171 ceph-mon[37789]: pgmap v1640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:15:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:19 smithi171 ceph-mon[37789]: pgmap v1641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:19 smithi044 ceph-mon[32404]: pgmap v1641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:19 smithi116 ceph-mon[37097]: pgmap v1641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:21 smithi171 ceph-mon[37789]: pgmap v1642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2065.603518 secs 2024-07-26T09:15:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:21 smithi044 ceph-mon[32404]: pgmap v1642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2065.603518 secs 2024-07-26T09:15:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:21 smithi116 ceph-mon[37097]: pgmap v1642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2065.603518 secs 2024-07-26T09:15:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:23 smithi171 ceph-mon[37789]: pgmap v1643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:23 smithi044 ceph-mon[32404]: pgmap v1643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:23 smithi116 ceph-mon[37097]: pgmap v1643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:15:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:24 smithi044 ceph-mon[32404]: pgmap v1644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:15:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:24 smithi116 ceph-mon[37097]: pgmap v1644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:15:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:24 smithi171 ceph-mon[37789]: pgmap v1644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:15:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:25.782+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:15:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:25.782+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:25.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:25.782+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:15:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2070.603675 secs 2024-07-26T09:15:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2070.603675 secs 2024-07-26T09:15:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2070.603675 secs 2024-07-26T09:15:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:26 smithi044 ceph-mon[32404]: pgmap v1645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:26 smithi116 ceph-mon[37097]: pgmap v1645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:26 smithi171 ceph-mon[37789]: pgmap v1645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:29 smithi171 ceph-mon[37789]: pgmap v1646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:29 smithi044 ceph-mon[32404]: pgmap v1646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:29 smithi116 ceph-mon[37097]: pgmap v1646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:31 smithi171 ceph-mon[37789]: pgmap v1647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2075.603774 secs 2024-07-26T09:15:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:31 smithi044 ceph-mon[32404]: pgmap v1647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2075.603774 secs 2024-07-26T09:15:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:31 smithi116 ceph-mon[37097]: pgmap v1647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2075.603774 secs 2024-07-26T09:15:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:32 smithi044 ceph-mon[32404]: pgmap v1648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:32 smithi116 ceph-mon[37097]: pgmap v1648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:32 smithi171 ceph-mon[37789]: pgmap v1648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:35 smithi171 ceph-mon[37789]: pgmap v1649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:35 smithi044 ceph-mon[32404]: pgmap v1649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:35 smithi116 ceph-mon[37097]: pgmap v1649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2080.604556 secs 2024-07-26T09:15:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2080.604556 secs 2024-07-26T09:15:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2080.604556 secs 2024-07-26T09:15:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:37 smithi044 ceph-mon[32404]: pgmap v1650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:37 smithi116 ceph-mon[37097]: pgmap v1650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:37 smithi171 ceph-mon[37789]: pgmap v1650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:38 smithi044 ceph-mon[32404]: pgmap v1651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:38 smithi116 ceph-mon[37097]: pgmap v1651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:38 smithi171 ceph-mon[37789]: pgmap v1651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:40.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:40.785+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:15:40.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:40.785+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:15:40.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:40.785+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:41 smithi171 ceph-mon[37789]: pgmap v1652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2085.604721 secs 2024-07-26T09:15:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:41 smithi044 ceph-mon[32404]: pgmap v1652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2085.604721 secs 2024-07-26T09:15:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:41 smithi116 ceph-mon[37097]: pgmap v1652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2085.604721 secs 2024-07-26T09:15:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:42 smithi044 ceph-mon[32404]: pgmap v1653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:42 smithi116 ceph-mon[37097]: pgmap v1653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:42 smithi171 ceph-mon[37789]: pgmap v1653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:45 smithi171 ceph-mon[37789]: pgmap v1654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:45 smithi044 ceph-mon[32404]: pgmap v1654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:45 smithi116 ceph-mon[37097]: pgmap v1654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:46.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2090.604866 secs 2024-07-26T09:15:46.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:46 smithi044 ceph-mon[32404]: pgmap v1655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:46.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2090.604866 secs 2024-07-26T09:15:46.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:46 smithi116 ceph-mon[37097]: pgmap v1655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2090.604866 secs 2024-07-26T09:15:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:46 smithi171 ceph-mon[37789]: pgmap v1655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:49.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:49 smithi044 ceph-mon[32404]: pgmap v1656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:49 smithi116 ceph-mon[37097]: pgmap v1656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:49 smithi171 ceph-mon[37789]: pgmap v1656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:50 smithi044 ceph-mon[32404]: pgmap v1657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:50 smithi116 ceph-mon[37097]: pgmap v1657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:50 smithi171 ceph-mon[37789]: pgmap v1657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:15:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2095.605000 secs 2024-07-26T09:15:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2095.605000 secs 2024-07-26T09:15:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2095.605000 secs 2024-07-26T09:15:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:52 smithi044 ceph-mon[32404]: pgmap v1658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:52 smithi116 ceph-mon[37097]: pgmap v1658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:52 smithi171 ceph-mon[37789]: pgmap v1658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:54.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:54 smithi171 ceph-mon[37789]: pgmap v1659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:54 smithi044 ceph-mon[32404]: pgmap v1659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:54 smithi116 ceph-mon[37097]: pgmap v1659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:15:55.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:55.794+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:15:55.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:55.794+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:15:55.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:15:55.794+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2100.605132 secs 2024-07-26T09:15:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2100.605132 secs 2024-07-26T09:15:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2100.605132 secs 2024-07-26T09:15:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:56 smithi044 ceph-mon[32404]: pgmap v1660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:56 smithi116 ceph-mon[37097]: pgmap v1660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:56 smithi171 ceph-mon[37789]: pgmap v1660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:15:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:15:58 smithi044 ceph-mon[32404]: pgmap v1661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:15:58 smithi116 ceph-mon[37097]: pgmap v1661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:15:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:15:58 smithi171 ceph-mon[37789]: pgmap v1661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:00 smithi044 ceph-mon[32404]: pgmap v1662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:16:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:00 smithi116 ceph-mon[37097]: pgmap v1662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:16:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:00 smithi171 ceph-mon[37789]: pgmap v1662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:16:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2105.605279 secs 2024-07-26T09:16:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2105.605279 secs 2024-07-26T09:16:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2105.605279 secs 2024-07-26T09:16:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:02 smithi044 ceph-mon[32404]: pgmap v1663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:02 smithi116 ceph-mon[37097]: pgmap v1663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:02 smithi171 ceph-mon[37789]: pgmap v1663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:04 smithi044 ceph-mon[32404]: pgmap v1664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:04 smithi116 ceph-mon[37097]: pgmap v1664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:04 smithi171 ceph-mon[37789]: pgmap v1664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2110.605425 secs 2024-07-26T09:16:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2110.605425 secs 2024-07-26T09:16:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2110.605425 secs 2024-07-26T09:16:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:06 smithi044 ceph-mon[32404]: pgmap v1665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:16:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:06 smithi116 ceph-mon[37097]: pgmap v1665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:16:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:06 smithi171 ceph-mon[37789]: pgmap v1665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:16:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:08 smithi044 ceph-mon[32404]: pgmap v1666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:08 smithi116 ceph-mon[37097]: pgmap v1666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:08 smithi171 ceph-mon[37789]: pgmap v1666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:10 smithi044 ceph-mon[32404]: pgmap v1667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:10 smithi116 ceph-mon[37097]: pgmap v1667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:10 smithi171 ceph-mon[37789]: pgmap v1667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:10.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:10.797+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:16:10.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:10.797+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:16:10.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:10.797+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2115.605518 secs 2024-07-26T09:16:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2115.605518 secs 2024-07-26T09:16:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2115.605518 secs 2024-07-26T09:16:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:12 smithi044 ceph-mon[32404]: pgmap v1668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:12 smithi116 ceph-mon[37097]: pgmap v1668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:12 smithi171 ceph-mon[37789]: pgmap v1668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:14 smithi044 ceph-mon[32404]: pgmap v1669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:14 smithi116 ceph-mon[37097]: pgmap v1669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:14 smithi171 ceph-mon[37789]: pgmap v1669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:15.402 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:16:15.403 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.403 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.403 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.403 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:16:15.406 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.500 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:16:15.500 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.500 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.500 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:15.501 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2120.606135 secs 2024-07-26T09:16:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:16 smithi044 ceph-mon[32404]: pgmap v1670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2120.606135 secs 2024-07-26T09:16:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:16 smithi116 ceph-mon[37097]: pgmap v1670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2120.606135 secs 2024-07-26T09:16:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:16 smithi171 ceph-mon[37789]: pgmap v1670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:18 smithi044 ceph-mon[32404]: pgmap v1671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:18 smithi116 ceph-mon[37097]: pgmap v1671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:18 smithi171 ceph-mon[37789]: pgmap v1671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:19 smithi044 ceph-mon[32404]: pgmap v1672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:19 smithi044 ceph-mon[32404]: pgmap v1673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:19 smithi116 ceph-mon[37097]: pgmap v1672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:19 smithi116 ceph-mon[37097]: pgmap v1673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:19 smithi171 ceph-mon[37789]: pgmap v1672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:16:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:19 smithi171 ceph-mon[37789]: pgmap v1673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:16:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2125.606304 secs 2024-07-26T09:16:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2125.606304 secs 2024-07-26T09:16:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2125.606304 secs 2024-07-26T09:16:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:21 smithi044 ceph-mon[32404]: pgmap v1674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:21 smithi116 ceph-mon[37097]: pgmap v1674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:21 smithi171 ceph-mon[37789]: pgmap v1674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:24 smithi171 ceph-mon[37789]: pgmap v1675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:24 smithi044 ceph-mon[32404]: pgmap v1675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:24 smithi116 ceph-mon[37097]: pgmap v1675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:25.806+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:16:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:25.806+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:16:25.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:25.806+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:26 smithi044 ceph-mon[32404]: pgmap v1676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2130.606427 secs 2024-07-26T09:16:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:26 smithi116 ceph-mon[37097]: pgmap v1676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2130.606427 secs 2024-07-26T09:16:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:26 smithi171 ceph-mon[37789]: pgmap v1676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:16:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2130.606427 secs 2024-07-26T09:16:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:28 smithi044 ceph-mon[32404]: pgmap v1677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:28 smithi116 ceph-mon[37097]: pgmap v1677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:28 smithi171 ceph-mon[37789]: pgmap v1677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:29 smithi044 ceph-mon[32404]: pgmap v1678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:16:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:29 smithi116 ceph-mon[37097]: pgmap v1678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:16:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:29 smithi171 ceph-mon[37789]: pgmap v1678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:16:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2135.606596 secs 2024-07-26T09:16:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2135.606596 secs 2024-07-26T09:16:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2135.606596 secs 2024-07-26T09:16:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:31 smithi044 ceph-mon[32404]: pgmap v1679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:31 smithi116 ceph-mon[37097]: pgmap v1679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:31 smithi171 ceph-mon[37789]: pgmap v1679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:34 smithi044 ceph-mon[32404]: pgmap v1680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:34 smithi116 ceph-mon[37097]: pgmap v1680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:34 smithi171 ceph-mon[37789]: pgmap v1680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:36 smithi044 ceph-mon[32404]: pgmap v1681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2140.606760 secs 2024-07-26T09:16:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:36 smithi116 ceph-mon[37097]: pgmap v1681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2140.606760 secs 2024-07-26T09:16:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:36 smithi171 ceph-mon[37789]: pgmap v1681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2140.606760 secs 2024-07-26T09:16:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:37 smithi044 ceph-mon[32404]: pgmap v1682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:37 smithi116 ceph-mon[37097]: pgmap v1682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:37 smithi171 ceph-mon[37789]: pgmap v1682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:40 smithi044 ceph-mon[32404]: pgmap v1683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:40 smithi116 ceph-mon[37097]: pgmap v1683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:40 smithi171 ceph-mon[37789]: pgmap v1683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:40.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:40.809+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:16:40.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:40.809+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:40.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:40.809+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:16:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2145.606864 secs 2024-07-26T09:16:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2145.606864 secs 2024-07-26T09:16:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2145.606864 secs 2024-07-26T09:16:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:42 smithi044 ceph-mon[32404]: pgmap v1684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:42 smithi116 ceph-mon[37097]: pgmap v1684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:42 smithi171 ceph-mon[37789]: pgmap v1684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:43 smithi044 ceph-mon[32404]: pgmap v1685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:43 smithi116 ceph-mon[37097]: pgmap v1685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:43 smithi171 ceph-mon[37789]: pgmap v1685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:46 smithi044 ceph-mon[32404]: pgmap v1686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2150.607016 secs 2024-07-26T09:16:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:46 smithi116 ceph-mon[37097]: pgmap v1686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2150.607016 secs 2024-07-26T09:16:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:46 smithi171 ceph-mon[37789]: pgmap v1686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2150.607016 secs 2024-07-26T09:16:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:47 smithi044 ceph-mon[32404]: pgmap v1687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:47 smithi116 ceph-mon[37097]: pgmap v1687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:47 smithi171 ceph-mon[37789]: pgmap v1687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:49 smithi044 ceph-mon[32404]: pgmap v1688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:49 smithi116 ceph-mon[37097]: pgmap v1688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:49 smithi171 ceph-mon[37789]: pgmap v1688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2155.607261 secs 2024-07-26T09:16:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2155.607261 secs 2024-07-26T09:16:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2155.607261 secs 2024-07-26T09:16:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:51 smithi044 ceph-mon[32404]: pgmap v1689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:51 smithi116 ceph-mon[37097]: pgmap v1689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:51 smithi171 ceph-mon[37789]: pgmap v1689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:53 smithi044 ceph-mon[32404]: pgmap v1690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:53 smithi116 ceph-mon[37097]: pgmap v1690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:53 smithi171 ceph-mon[37789]: pgmap v1690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:16:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:55 smithi044 ceph-mon[32404]: pgmap v1691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:55 smithi116 ceph-mon[37097]: pgmap v1691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:55 smithi171 ceph-mon[37789]: pgmap v1691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:16:55.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:55.817+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:16:55.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:55.817+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:16:55.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:16:55.817+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2160.607983 secs 2024-07-26T09:16:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2160.607983 secs 2024-07-26T09:16:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2160.607983 secs 2024-07-26T09:16:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:57 smithi044 ceph-mon[32404]: pgmap v1692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:57 smithi116 ceph-mon[37097]: pgmap v1692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:57 smithi171 ceph-mon[37789]: pgmap v1692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:16:59 smithi044 ceph-mon[32404]: pgmap v1693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:16:59 smithi116 ceph-mon[37097]: pgmap v1693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:16:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:16:59 smithi171 ceph-mon[37789]: pgmap v1693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2165.608098 secs 2024-07-26T09:17:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2165.608098 secs 2024-07-26T09:17:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2165.608098 secs 2024-07-26T09:17:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:01 smithi044 ceph-mon[32404]: pgmap v1694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:01 smithi116 ceph-mon[37097]: pgmap v1694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:01 smithi171 ceph-mon[37789]: pgmap v1694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:03 smithi044 ceph-mon[32404]: pgmap v1695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:03 smithi116 ceph-mon[37097]: pgmap v1695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:03 smithi171 ceph-mon[37789]: pgmap v1695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:05 smithi044 ceph-mon[32404]: pgmap v1696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:05 smithi116 ceph-mon[37097]: pgmap v1696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:05 smithi171 ceph-mon[37789]: pgmap v1696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2170.608258 secs 2024-07-26T09:17:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2170.608258 secs 2024-07-26T09:17:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2170.608258 secs 2024-07-26T09:17:07.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:07 smithi044 ceph-mon[32404]: pgmap v1697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:07 smithi116 ceph-mon[37097]: pgmap v1697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:07 smithi171 ceph-mon[37789]: pgmap v1697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:09 smithi044 ceph-mon[32404]: pgmap v1698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:09 smithi116 ceph-mon[37097]: pgmap v1698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:09 smithi171 ceph-mon[37789]: pgmap v1698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:10.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:10.820+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:17:10.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:10.820+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:17:10.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:10.820+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2175.608421 secs 2024-07-26T09:17:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2175.608421 secs 2024-07-26T09:17:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2175.608421 secs 2024-07-26T09:17:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:11 smithi044 ceph-mon[32404]: pgmap v1699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:11 smithi116 ceph-mon[37097]: pgmap v1699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:11 smithi171 ceph-mon[37789]: pgmap v1699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:13 smithi044 ceph-mon[32404]: pgmap v1700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:13 smithi116 ceph-mon[37097]: pgmap v1700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:13 smithi171 ceph-mon[37789]: pgmap v1700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:15.045+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:17:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:15.045+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:17:15.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:15.045+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:17:15.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:15.046+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:17:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:15 smithi044 ceph-mon[32404]: pgmap v1701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:15 smithi116 ceph-mon[37097]: pgmap v1701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:15 smithi171 ceph-mon[37789]: pgmap v1701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:16.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2180.608588 secs 2024-07-26T09:17:16.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2180.608588 secs 2024-07-26T09:17:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2180.608588 secs 2024-07-26T09:17:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:17 smithi044 ceph-mon[32404]: pgmap v1702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:17 smithi116 ceph-mon[37097]: pgmap v1702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:17 smithi171 ceph-mon[37789]: pgmap v1702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:19 smithi044 ceph-mon[32404]: pgmap v1703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:19 smithi116 ceph-mon[37097]: pgmap v1703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:19 smithi171 ceph-mon[37789]: pgmap v1703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:21 smithi044 ceph-mon[32404]: pgmap v1704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:17:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:21 smithi044 ceph-mon[32404]: pgmap v1705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2185.608651 secs 2024-07-26T09:17:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:17:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:21 smithi116 ceph-mon[37097]: pgmap v1704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:17:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:21 smithi116 ceph-mon[37097]: pgmap v1705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2185.608651 secs 2024-07-26T09:17:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:17:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:21 smithi171 ceph-mon[37789]: pgmap v1704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:17:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:21 smithi171 ceph-mon[37789]: pgmap v1705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2185.608651 secs 2024-07-26T09:17:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:17:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:23 smithi171 ceph-mon[37789]: pgmap v1706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:23 smithi044 ceph-mon[32404]: pgmap v1706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:23 smithi116 ceph-mon[37097]: pgmap v1706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:25 smithi171 ceph-mon[37789]: pgmap v1707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:25 smithi044 ceph-mon[32404]: pgmap v1707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:25 smithi116 ceph-mon[37097]: pgmap v1707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:25.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:25.825+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:17:25.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:25.825+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:25.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:25.825+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:17:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2190.608794 secs 2024-07-26T09:17:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2190.608794 secs 2024-07-26T09:17:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2190.608794 secs 2024-07-26T09:17:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:27 smithi171 ceph-mon[37789]: pgmap v1708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:27 smithi044 ceph-mon[32404]: pgmap v1708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:27 smithi116 ceph-mon[37097]: pgmap v1708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:29 smithi044 ceph-mon[32404]: pgmap v1709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:29 smithi116 ceph-mon[37097]: pgmap v1709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:29 smithi171 ceph-mon[37789]: pgmap v1709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:17:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:31 smithi044 ceph-mon[32404]: pgmap v1710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:17:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2195.608910 secs 2024-07-26T09:17:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:31 smithi116 ceph-mon[37097]: pgmap v1710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:17:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2195.608910 secs 2024-07-26T09:17:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:31 smithi171 ceph-mon[37789]: pgmap v1710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:17:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2195.608910 secs 2024-07-26T09:17:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:33 smithi044 ceph-mon[32404]: pgmap v1711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:33 smithi116 ceph-mon[37097]: pgmap v1711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:33 smithi171 ceph-mon[37789]: pgmap v1711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:35 smithi044 ceph-mon[32404]: pgmap v1712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:35 smithi116 ceph-mon[37097]: pgmap v1712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:35 smithi171 ceph-mon[37789]: pgmap v1712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2200.609824 secs 2024-07-26T09:17:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2200.609824 secs 2024-07-26T09:17:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2200.609824 secs 2024-07-26T09:17:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:37 smithi044 ceph-mon[32404]: pgmap v1713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:37 smithi116 ceph-mon[37097]: pgmap v1713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:37 smithi171 ceph-mon[37789]: pgmap v1713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:39 smithi044 ceph-mon[32404]: pgmap v1714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:39 smithi116 ceph-mon[37097]: pgmap v1714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:39 smithi171 ceph-mon[37789]: pgmap v1714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:40.828+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:17:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:40.829+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:17:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:40.829+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2205.609967 secs 2024-07-26T09:17:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:41 smithi044 ceph-mon[32404]: pgmap v1715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2205.609967 secs 2024-07-26T09:17:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:41 smithi116 ceph-mon[37097]: pgmap v1715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2205.609967 secs 2024-07-26T09:17:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:41 smithi171 ceph-mon[37789]: pgmap v1715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:43.714 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:43 smithi044 ceph-mon[32404]: pgmap v1716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:43 smithi116 ceph-mon[37097]: pgmap v1716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:43 smithi171 ceph-mon[37789]: pgmap v1716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:45 smithi044 ceph-mon[32404]: pgmap v1717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:45 smithi116 ceph-mon[37097]: pgmap v1717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:45 smithi171 ceph-mon[37789]: pgmap v1717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:46.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2210.610133 secs 2024-07-26T09:17:46.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2210.610133 secs 2024-07-26T09:17:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2210.610133 secs 2024-07-26T09:17:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:48 smithi044 ceph-mon[32404]: pgmap v1718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:17:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:48 smithi116 ceph-mon[37097]: pgmap v1718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:17:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:48 smithi171 ceph-mon[37789]: pgmap v1718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:17:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:50 smithi044 ceph-mon[32404]: pgmap v1719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:50 smithi116 ceph-mon[37097]: pgmap v1719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:50 smithi171 ceph-mon[37789]: pgmap v1719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2215.610275 secs 2024-07-26T09:17:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2215.610275 secs 2024-07-26T09:17:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2215.610275 secs 2024-07-26T09:17:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:52 smithi044 ceph-mon[32404]: pgmap v1720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:52 smithi116 ceph-mon[37097]: pgmap v1720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:52 smithi171 ceph-mon[37789]: pgmap v1720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:53 smithi044 ceph-mon[32404]: pgmap v1721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:53 smithi116 ceph-mon[37097]: pgmap v1721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:53 smithi171 ceph-mon[37789]: pgmap v1721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:17:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:55 smithi171 ceph-mon[37789]: pgmap v1722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:55 smithi044 ceph-mon[32404]: pgmap v1722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:55 smithi116 ceph-mon[37097]: pgmap v1722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:17:55.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:55.837+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:17:55.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:55.838+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:17:55.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:17:55.838+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2220.610420 secs 2024-07-26T09:17:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2220.610420 secs 2024-07-26T09:17:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2220.610420 secs 2024-07-26T09:17:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:57 smithi044 ceph-mon[32404]: pgmap v1723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:57 smithi116 ceph-mon[37097]: pgmap v1723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:57 smithi171 ceph-mon[37789]: pgmap v1723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:17:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:17:59 smithi044 ceph-mon[32404]: pgmap v1724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:17:59 smithi116 ceph-mon[37097]: pgmap v1724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:17:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:17:59 smithi171 ceph-mon[37789]: pgmap v1724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2225.610580 secs 2024-07-26T09:18:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:01 smithi044 ceph-mon[32404]: pgmap v1725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2225.610580 secs 2024-07-26T09:18:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:01 smithi116 ceph-mon[37097]: pgmap v1725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2225.610580 secs 2024-07-26T09:18:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:01 smithi171 ceph-mon[37789]: pgmap v1725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:03 smithi044 ceph-mon[32404]: pgmap v1726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:03 smithi116 ceph-mon[37097]: pgmap v1726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:03 smithi171 ceph-mon[37789]: pgmap v1726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:05 smithi044 ceph-mon[32404]: pgmap v1727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:05 smithi116 ceph-mon[37097]: pgmap v1727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:05 smithi171 ceph-mon[37789]: pgmap v1727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2230.610716 secs 2024-07-26T09:18:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2230.610716 secs 2024-07-26T09:18:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2230.610716 secs 2024-07-26T09:18:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:07 smithi044 ceph-mon[32404]: pgmap v1728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:07 smithi116 ceph-mon[37097]: pgmap v1728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:07 smithi171 ceph-mon[37789]: pgmap v1728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:09 smithi044 ceph-mon[32404]: pgmap v1729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:09 smithi116 ceph-mon[37097]: pgmap v1729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:09 smithi171 ceph-mon[37789]: pgmap v1729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:10.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:10.842+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:18:10.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:10.842+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:18:10.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:10.842+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2235.610820 secs 2024-07-26T09:18:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:11 smithi044 ceph-mon[32404]: pgmap v1730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2235.610820 secs 2024-07-26T09:18:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:11 smithi116 ceph-mon[37097]: pgmap v1730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2235.610820 secs 2024-07-26T09:18:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:11 smithi171 ceph-mon[37789]: pgmap v1730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:13 smithi044 ceph-mon[32404]: pgmap v1731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:13 smithi116 ceph-mon[37097]: pgmap v1731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:13 smithi171 ceph-mon[37789]: pgmap v1731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:15 smithi044 ceph-mon[32404]: pgmap v1732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:15 smithi116 ceph-mon[37097]: pgmap v1732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:15 smithi171 ceph-mon[37789]: pgmap v1732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:16.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2240.611242 secs 2024-07-26T09:18:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2240.611242 secs 2024-07-26T09:18:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2240.611242 secs 2024-07-26T09:18:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:17 smithi044 ceph-mon[32404]: pgmap v1733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:17 smithi116 ceph-mon[37097]: pgmap v1733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:17 smithi171 ceph-mon[37789]: pgmap v1733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:19 smithi044 ceph-mon[32404]: pgmap v1734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:19 smithi116 ceph-mon[37097]: pgmap v1734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:19 smithi171 ceph-mon[37789]: pgmap v1734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:18:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2245.611384 secs 2024-07-26T09:18:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:21 smithi044 ceph-mon[32404]: pgmap v1735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2245.611384 secs 2024-07-26T09:18:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:21 smithi116 ceph-mon[37097]: pgmap v1735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2245.611384 secs 2024-07-26T09:18:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:21 smithi171 ceph-mon[37789]: pgmap v1735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:22 smithi044 ceph-mon[32404]: pgmap v1736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:18:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:22 smithi044 ceph-mon[32404]: pgmap v1737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:18:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:22 smithi116 ceph-mon[37097]: pgmap v1736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:18:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:22 smithi116 ceph-mon[37097]: pgmap v1737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:18:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:22 smithi171 ceph-mon[37789]: pgmap v1736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:18:22.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:22 smithi171 ceph-mon[37789]: pgmap v1737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:22.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:18:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:24 smithi171 ceph-mon[37789]: pgmap v1738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:24 smithi044 ceph-mon[32404]: pgmap v1738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:24 smithi116 ceph-mon[37097]: pgmap v1738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:25.847+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:18:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:25.847+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:25.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:25.847+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:18:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2250.611536 secs 2024-07-26T09:18:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:26 smithi171 ceph-mon[37789]: pgmap v1739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2250.611536 secs 2024-07-26T09:18:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:26 smithi044 ceph-mon[32404]: pgmap v1739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2250.611536 secs 2024-07-26T09:18:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:26 smithi116 ceph-mon[37097]: pgmap v1739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:28 smithi171 ceph-mon[37789]: pgmap v1740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:28 smithi044 ceph-mon[32404]: pgmap v1740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:28 smithi116 ceph-mon[37097]: pgmap v1740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:30 smithi171 ceph-mon[37789]: pgmap v1741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:30 smithi044 ceph-mon[32404]: pgmap v1741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:30 smithi116 ceph-mon[37097]: pgmap v1741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:18:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2255.611691 secs 2024-07-26T09:18:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2255.611691 secs 2024-07-26T09:18:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2255.611691 secs 2024-07-26T09:18:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:32 smithi044 ceph-mon[32404]: pgmap v1742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:18:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:32 smithi116 ceph-mon[37097]: pgmap v1742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:18:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:32 smithi171 ceph-mon[37789]: pgmap v1742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:18:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:34 smithi044 ceph-mon[32404]: pgmap v1743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:34 smithi116 ceph-mon[37097]: pgmap v1743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:34 smithi171 ceph-mon[37789]: pgmap v1743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2260.611837 secs 2024-07-26T09:18:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:36 smithi044 ceph-mon[32404]: pgmap v1744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2260.611837 secs 2024-07-26T09:18:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:36 smithi116 ceph-mon[37097]: pgmap v1744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2260.611837 secs 2024-07-26T09:18:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:36 smithi171 ceph-mon[37789]: pgmap v1744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:38 smithi044 ceph-mon[32404]: pgmap v1745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:38 smithi116 ceph-mon[37097]: pgmap v1745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:38 smithi171 ceph-mon[37789]: pgmap v1745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:40.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:40.850+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:18:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:40.850+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:40.850+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:18:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:40 smithi044 ceph-mon[32404]: pgmap v1746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:40 smithi116 ceph-mon[37097]: pgmap v1746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:40 smithi171 ceph-mon[37789]: pgmap v1746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2265.612003 secs 2024-07-26T09:18:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2265.612003 secs 2024-07-26T09:18:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2265.612003 secs 2024-07-26T09:18:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:42 smithi044 ceph-mon[32404]: pgmap v1747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:42 smithi116 ceph-mon[37097]: pgmap v1747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:42 smithi171 ceph-mon[37789]: pgmap v1747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:44 smithi044 ceph-mon[32404]: pgmap v1748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:44 smithi116 ceph-mon[37097]: pgmap v1748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:44 smithi171 ceph-mon[37789]: pgmap v1748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2270.612147 secs 2024-07-26T09:18:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:46 smithi044 ceph-mon[32404]: pgmap v1749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2270.612147 secs 2024-07-26T09:18:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:46 smithi116 ceph-mon[37097]: pgmap v1749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2270.612147 secs 2024-07-26T09:18:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:46 smithi171 ceph-mon[37789]: pgmap v1749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:48 smithi044 ceph-mon[32404]: pgmap v1750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:48 smithi116 ceph-mon[37097]: pgmap v1750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:48 smithi171 ceph-mon[37789]: pgmap v1750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:50 smithi044 ceph-mon[32404]: pgmap v1751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:50 smithi116 ceph-mon[37097]: pgmap v1751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:50 smithi171 ceph-mon[37789]: pgmap v1751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:18:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2275.612310 secs 2024-07-26T09:18:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2275.612310 secs 2024-07-26T09:18:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2275.612310 secs 2024-07-26T09:18:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:52 smithi044 ceph-mon[32404]: pgmap v1752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:18:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:52 smithi116 ceph-mon[37097]: pgmap v1752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:18:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:52 smithi171 ceph-mon[37789]: pgmap v1752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:18:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:54 smithi044 ceph-mon[32404]: pgmap v1753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:55.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:54 smithi116 ceph-mon[37097]: pgmap v1753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:54 smithi171 ceph-mon[37789]: pgmap v1753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:18:55.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:55.859+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:18:55.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:55.859+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:55.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:18:55.859+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:18:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2280.613279 secs 2024-07-26T09:18:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:56 smithi044 ceph-mon[32404]: pgmap v1754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2280.613279 secs 2024-07-26T09:18:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:56 smithi116 ceph-mon[37097]: pgmap v1754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2280.613279 secs 2024-07-26T09:18:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:56 smithi171 ceph-mon[37789]: pgmap v1754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:18:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:18:58 smithi044 ceph-mon[32404]: pgmap v1755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:18:58 smithi116 ceph-mon[37097]: pgmap v1755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:18:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:18:58 smithi171 ceph-mon[37789]: pgmap v1755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:00 smithi044 ceph-mon[32404]: pgmap v1756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:00 smithi116 ceph-mon[37097]: pgmap v1756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:00 smithi171 ceph-mon[37789]: pgmap v1756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2285.613442 secs 2024-07-26T09:19:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2285.613442 secs 2024-07-26T09:19:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2285.613442 secs 2024-07-26T09:19:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:02 smithi044 ceph-mon[32404]: pgmap v1757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:03.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:02 smithi116 ceph-mon[37097]: pgmap v1757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:02 smithi171 ceph-mon[37789]: pgmap v1757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:04 smithi044 ceph-mon[32404]: pgmap v1758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:04 smithi116 ceph-mon[37097]: pgmap v1758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:04 smithi171 ceph-mon[37789]: pgmap v1758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2290.613603 secs 2024-07-26T09:19:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2290.613603 secs 2024-07-26T09:19:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2290.613603 secs 2024-07-26T09:19:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:06 smithi044 ceph-mon[32404]: pgmap v1759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:06 smithi116 ceph-mon[37097]: pgmap v1759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:06 smithi171 ceph-mon[37789]: pgmap v1759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:08 smithi044 ceph-mon[32404]: pgmap v1760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:08 smithi116 ceph-mon[37097]: pgmap v1760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:08 smithi171 ceph-mon[37789]: pgmap v1760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:10.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:10.862+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:19:10.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:10.862+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:19:10.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:10.862+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:10 smithi044 ceph-mon[32404]: pgmap v1761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2295.613771 secs 2024-07-26T09:19:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:10 smithi116 ceph-mon[37097]: pgmap v1761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2295.613771 secs 2024-07-26T09:19:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:10 smithi171 ceph-mon[37789]: pgmap v1761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2295.613771 secs 2024-07-26T09:19:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:12 smithi044 ceph-mon[32404]: pgmap v1762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:12 smithi116 ceph-mon[37097]: pgmap v1762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:12 smithi171 ceph-mon[37789]: pgmap v1762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:14 smithi044 ceph-mon[32404]: pgmap v1763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:14 smithi116 ceph-mon[37097]: pgmap v1763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:14 smithi171 ceph-mon[37789]: pgmap v1763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:19:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2300.613881 secs 2024-07-26T09:19:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2300.613881 secs 2024-07-26T09:19:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2300.613881 secs 2024-07-26T09:19:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:16 smithi171 ceph-mon[37789]: pgmap v1764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:17.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:16 smithi044 ceph-mon[32404]: pgmap v1764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:16 smithi116 ceph-mon[37097]: pgmap v1764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:18 smithi171 ceph-mon[37789]: pgmap v1765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:19.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:18 smithi044 ceph-mon[32404]: pgmap v1765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:18 smithi116 ceph-mon[37097]: pgmap v1765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:20 smithi171 ceph-mon[37789]: pgmap v1766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:21.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2305.613988 secs 2024-07-26T09:19:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:20 smithi044 ceph-mon[32404]: pgmap v1766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2305.613988 secs 2024-07-26T09:19:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:20 smithi116 ceph-mon[37097]: pgmap v1766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2305.613988 secs 2024-07-26T09:19:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:22 smithi171 ceph-mon[37789]: pgmap v1767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:23.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:23.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:19:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:22 smithi044 ceph-mon[32404]: pgmap v1767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:23.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:23.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:23.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:19:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:22 smithi116 ceph-mon[37097]: pgmap v1767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:19:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:19:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:23 smithi171 ceph-mon[37789]: pgmap v1768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:19:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:23 smithi171 ceph-mon[37789]: pgmap v1769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:19:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:23 smithi044 ceph-mon[32404]: pgmap v1768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:19:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:23 smithi044 ceph-mon[32404]: pgmap v1769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:19:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:23 smithi116 ceph-mon[37097]: pgmap v1768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:19:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:23 smithi116 ceph-mon[37097]: pgmap v1769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:19:25.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:25.870+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:19:25.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:25.870+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:19:25.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:25.870+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:25 smithi171 ceph-mon[37789]: pgmap v1770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2310.614154 secs 2024-07-26T09:19:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:25 smithi044 ceph-mon[32404]: pgmap v1770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2310.614154 secs 2024-07-26T09:19:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:25 smithi116 ceph-mon[37097]: pgmap v1770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2310.614154 secs 2024-07-26T09:19:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:27 smithi171 ceph-mon[37789]: pgmap v1771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:27 smithi044 ceph-mon[32404]: pgmap v1771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:27 smithi116 ceph-mon[37097]: pgmap v1771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:29 smithi171 ceph-mon[37789]: pgmap v1772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:29 smithi044 ceph-mon[32404]: pgmap v1772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:30.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:29 smithi116 ceph-mon[37097]: pgmap v1772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:19:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2315.614285 secs 2024-07-26T09:19:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2315.614285 secs 2024-07-26T09:19:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2315.614285 secs 2024-07-26T09:19:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:31 smithi044 ceph-mon[32404]: pgmap v1773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:19:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:31 smithi116 ceph-mon[37097]: pgmap v1773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:19:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:31 smithi171 ceph-mon[37789]: pgmap v1773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:19:34.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:33 smithi044 ceph-mon[32404]: pgmap v1774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:19:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:33 smithi116 ceph-mon[37097]: pgmap v1774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:19:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:33 smithi171 ceph-mon[37789]: pgmap v1774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:19:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:35 smithi044 ceph-mon[32404]: pgmap v1775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2320.614586 secs 2024-07-26T09:19:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:35 smithi116 ceph-mon[37097]: pgmap v1775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2320.614586 secs 2024-07-26T09:19:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:35 smithi171 ceph-mon[37789]: pgmap v1775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2320.614586 secs 2024-07-26T09:19:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:37 smithi044 ceph-mon[32404]: pgmap v1776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:37 smithi116 ceph-mon[37097]: pgmap v1776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:37 smithi171 ceph-mon[37789]: pgmap v1776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:39 smithi044 ceph-mon[32404]: pgmap v1777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:39 smithi116 ceph-mon[37097]: pgmap v1777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:39 smithi171 ceph-mon[37789]: pgmap v1777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:40.873+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:19:40.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:40.873+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:19:40.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:40.873+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2325.614730 secs 2024-07-26T09:19:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2325.614730 secs 2024-07-26T09:19:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2325.614730 secs 2024-07-26T09:19:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:41 smithi044 ceph-mon[32404]: pgmap v1778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:41 smithi116 ceph-mon[37097]: pgmap v1778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:41 smithi171 ceph-mon[37789]: pgmap v1778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:43 smithi044 ceph-mon[32404]: pgmap v1779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:43 smithi116 ceph-mon[37097]: pgmap v1779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:43 smithi171 ceph-mon[37789]: pgmap v1779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:45 smithi044 ceph-mon[32404]: pgmap v1780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2330.614875 secs 2024-07-26T09:19:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:45 smithi116 ceph-mon[37097]: pgmap v1780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2330.614875 secs 2024-07-26T09:19:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:45 smithi171 ceph-mon[37789]: pgmap v1780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2330.614875 secs 2024-07-26T09:19:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:47 smithi044 ceph-mon[32404]: pgmap v1781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:47 smithi116 ceph-mon[37097]: pgmap v1781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:47 smithi171 ceph-mon[37789]: pgmap v1781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:50.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:49 smithi044 ceph-mon[32404]: pgmap v1782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:49 smithi116 ceph-mon[37097]: pgmap v1782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:49 smithi171 ceph-mon[37789]: pgmap v1782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2335.614978 secs 2024-07-26T09:19:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2335.614978 secs 2024-07-26T09:19:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2335.614978 secs 2024-07-26T09:19:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:51 smithi044 ceph-mon[32404]: pgmap v1783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:51 smithi116 ceph-mon[37097]: pgmap v1783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:51 smithi171 ceph-mon[37789]: pgmap v1783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:19:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:53 smithi044 ceph-mon[32404]: pgmap v1784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:53 smithi116 ceph-mon[37097]: pgmap v1784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:53 smithi171 ceph-mon[37789]: pgmap v1784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:55.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:55.882+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:19:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:55.882+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:19:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:19:55.882+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:55 smithi044 ceph-mon[32404]: pgmap v1785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2340.615138 secs 2024-07-26T09:19:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:55 smithi116 ceph-mon[37097]: pgmap v1785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2340.615138 secs 2024-07-26T09:19:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:55 smithi171 ceph-mon[37789]: pgmap v1785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:19:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2340.615138 secs 2024-07-26T09:19:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:57 smithi044 ceph-mon[32404]: pgmap v1786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:57 smithi116 ceph-mon[37097]: pgmap v1786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:19:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:57 smithi171 ceph-mon[37789]: pgmap v1786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:19:59 smithi044 ceph-mon[32404]: pgmap v1787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:19:59 smithi116 ceph-mon[37097]: pgmap v1787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:19:59 smithi171 ceph-mon[37789]: pgmap v1787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:20:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2345.615303 secs 2024-07-26T09:20:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:20:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2345.615303 secs 2024-07-26T09:20:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:20:01.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2345.615303 secs 2024-07-26T09:20:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:01 smithi044 ceph-mon[32404]: pgmap v1788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:01 smithi116 ceph-mon[37097]: pgmap v1788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:01 smithi171 ceph-mon[37789]: pgmap v1788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:03 smithi044 ceph-mon[32404]: pgmap v1789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:03 smithi116 ceph-mon[37097]: pgmap v1789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:03 smithi171 ceph-mon[37789]: pgmap v1789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:05 smithi044 ceph-mon[32404]: pgmap v1790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2350.615468 secs 2024-07-26T09:20:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:05 smithi116 ceph-mon[37097]: pgmap v1790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2350.615468 secs 2024-07-26T09:20:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:05 smithi171 ceph-mon[37789]: pgmap v1790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:20:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2350.615468 secs 2024-07-26T09:20:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:07 smithi044 ceph-mon[32404]: pgmap v1791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:07 smithi116 ceph-mon[37097]: pgmap v1791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:07 smithi171 ceph-mon[37789]: pgmap v1791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:09 smithi044 ceph-mon[32404]: pgmap v1792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:09 smithi116 ceph-mon[37097]: pgmap v1792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:09 smithi171 ceph-mon[37789]: pgmap v1792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:10.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:10.886+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:20:10.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:10.886+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:20:10.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:10.886+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2355.615608 secs 2024-07-26T09:20:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2355.615608 secs 2024-07-26T09:20:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2355.615608 secs 2024-07-26T09:20:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:11 smithi044 ceph-mon[32404]: pgmap v1793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:20:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:11 smithi116 ceph-mon[37097]: pgmap v1793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:20:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:11 smithi171 ceph-mon[37789]: pgmap v1793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:20:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:14 smithi044 ceph-mon[32404]: pgmap v1794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:14 smithi116 ceph-mon[37097]: pgmap v1794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:14 smithi171 ceph-mon[37789]: pgmap v1794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:15.077+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:20:15.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:15.077+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:20:15.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:15.077+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:20:15.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:15.078+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:20:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:16 smithi044 ceph-mon[32404]: pgmap v1795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2360.616557 secs 2024-07-26T09:20:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:16 smithi116 ceph-mon[37097]: pgmap v1795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2360.616557 secs 2024-07-26T09:20:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:16 smithi171 ceph-mon[37789]: pgmap v1795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:20:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2360.616557 secs 2024-07-26T09:20:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:18 smithi044 ceph-mon[32404]: pgmap v1796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:18 smithi116 ceph-mon[37097]: pgmap v1796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:18 smithi171 ceph-mon[37789]: pgmap v1796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:20:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:20 smithi044 ceph-mon[32404]: pgmap v1797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:20 smithi116 ceph-mon[37097]: pgmap v1797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:20 smithi171 ceph-mon[37789]: pgmap v1797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2365.616715 secs 2024-07-26T09:20:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2365.616715 secs 2024-07-26T09:20:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2365.616715 secs 2024-07-26T09:20:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:22 smithi171 ceph-mon[37789]: pgmap v1798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:22 smithi044 ceph-mon[32404]: pgmap v1798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:22 smithi116 ceph-mon[37097]: pgmap v1798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:24 smithi171 ceph-mon[37789]: pgmap v1799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:24 smithi044 ceph-mon[32404]: pgmap v1799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:24 smithi116 ceph-mon[37097]: pgmap v1799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:25.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:25 smithi171 ceph-mon[37789]: pgmap v1800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:20:25.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:25 smithi171 ceph-mon[37789]: pgmap v1801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:20:25.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:20:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:25 smithi044 ceph-mon[32404]: pgmap v1800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:20:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:25 smithi044 ceph-mon[32404]: pgmap v1801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:20:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:20:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:25 smithi116 ceph-mon[37097]: pgmap v1800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:20:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:25 smithi116 ceph-mon[37097]: pgmap v1801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:20:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:20:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:25.893+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:20:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:25.893+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:25.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:25.893+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:20:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2370.616885 secs 2024-07-26T09:20:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2370.616885 secs 2024-07-26T09:20:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2370.616885 secs 2024-07-26T09:20:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:27 smithi171 ceph-mon[37789]: pgmap v1802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:27 smithi044 ceph-mon[32404]: pgmap v1802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:27 smithi116 ceph-mon[37097]: pgmap v1802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:20:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:29 smithi171 ceph-mon[37789]: pgmap v1803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:20:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:29 smithi044 ceph-mon[32404]: pgmap v1803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:20:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:29 smithi116 ceph-mon[37097]: pgmap v1803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:20:31.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:31 smithi171 ceph-mon[37789]: pgmap v1804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:20:31.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2375.617033 secs 2024-07-26T09:20:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:31 smithi044 ceph-mon[32404]: pgmap v1804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:20:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2375.617033 secs 2024-07-26T09:20:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:31 smithi116 ceph-mon[37097]: pgmap v1804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:20:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2375.617033 secs 2024-07-26T09:20:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:33 smithi044 ceph-mon[32404]: pgmap v1805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:20:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:33 smithi116 ceph-mon[37097]: pgmap v1805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:20:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:33 smithi171 ceph-mon[37789]: pgmap v1805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:20:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:35 smithi044 ceph-mon[32404]: pgmap v1806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:20:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:35 smithi116 ceph-mon[37097]: pgmap v1806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:20:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:35 smithi171 ceph-mon[37789]: pgmap v1806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:20:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2380.617179 secs 2024-07-26T09:20:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2380.617179 secs 2024-07-26T09:20:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2380.617179 secs 2024-07-26T09:20:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:37 smithi044 ceph-mon[32404]: pgmap v1807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:20:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:37 smithi116 ceph-mon[37097]: pgmap v1807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:20:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:37 smithi171 ceph-mon[37789]: pgmap v1807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:20:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:39 smithi044 ceph-mon[32404]: pgmap v1808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:39 smithi116 ceph-mon[37097]: pgmap v1808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:39 smithi171 ceph-mon[37789]: pgmap v1808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:40.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:40.897+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:20:40.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:40.897+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:40.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:40.897+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:20:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:41 smithi044 ceph-mon[32404]: pgmap v1809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:20:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2385.617318 secs 2024-07-26T09:20:41.417 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:41 smithi171 ceph-mon[37789]: pgmap v1809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:20:41.418 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2385.617318 secs 2024-07-26T09:20:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:41 smithi116 ceph-mon[37097]: pgmap v1809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:20:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2385.617318 secs 2024-07-26T09:20:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:43 smithi044 ceph-mon[32404]: pgmap v1810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:43 smithi116 ceph-mon[37097]: pgmap v1810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:43 smithi171 ceph-mon[37789]: pgmap v1810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:45 smithi044 ceph-mon[32404]: pgmap v1811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:45 smithi116 ceph-mon[37097]: pgmap v1811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:45 smithi171 ceph-mon[37789]: pgmap v1811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2390.617466 secs 2024-07-26T09:20:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2390.617466 secs 2024-07-26T09:20:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2390.617466 secs 2024-07-26T09:20:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:47 smithi044 ceph-mon[32404]: pgmap v1812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:47 smithi116 ceph-mon[37097]: pgmap v1812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:47 smithi171 ceph-mon[37789]: pgmap v1812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:48.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:48 smithi044 ceph-mon[32404]: pgmap v1813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:48.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:48 smithi116 ceph-mon[37097]: pgmap v1813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:48 smithi171 ceph-mon[37789]: pgmap v1813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:20:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:51 smithi044 ceph-mon[32404]: pgmap v1814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2395.617581 secs 2024-07-26T09:20:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:51 smithi116 ceph-mon[37097]: pgmap v1814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2395.617581 secs 2024-07-26T09:20:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:51 smithi171 ceph-mon[37789]: pgmap v1814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2395.617581 secs 2024-07-26T09:20:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:53 smithi044 ceph-mon[32404]: pgmap v1815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:53 smithi116 ceph-mon[37097]: pgmap v1815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:53 smithi171 ceph-mon[37789]: pgmap v1815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:55 smithi044 ceph-mon[32404]: pgmap v1816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:55 smithi116 ceph-mon[37097]: pgmap v1816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:55 smithi171 ceph-mon[37789]: pgmap v1816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:20:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:55.905+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:20:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:55.906+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:20:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:20:55.905+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2400.617736 secs 2024-07-26T09:20:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:56 smithi044 ceph-mon[32404]: pgmap v1817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2400.617736 secs 2024-07-26T09:20:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:56 smithi116 ceph-mon[37097]: pgmap v1817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2400.617736 secs 2024-07-26T09:20:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:56 smithi171 ceph-mon[37789]: pgmap v1817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:20:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:20:59 smithi044 ceph-mon[32404]: pgmap v1818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:20:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:20:59 smithi116 ceph-mon[37097]: pgmap v1818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:20:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:20:59 smithi171 ceph-mon[37789]: pgmap v1818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:21:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:01 smithi044 ceph-mon[32404]: pgmap v1819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:21:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2405.617833 secs 2024-07-26T09:21:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:01 smithi116 ceph-mon[37097]: pgmap v1819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:21:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2405.617833 secs 2024-07-26T09:21:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:01 smithi171 ceph-mon[37789]: pgmap v1819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:21:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2405.617833 secs 2024-07-26T09:21:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:02 smithi044 ceph-mon[32404]: pgmap v1820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:21:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:02 smithi116 ceph-mon[37097]: pgmap v1820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:21:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:02 smithi171 ceph-mon[37789]: pgmap v1820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:21:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:05 smithi044 ceph-mon[32404]: pgmap v1821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:21:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:05 smithi116 ceph-mon[37097]: pgmap v1821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:21:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:05 smithi171 ceph-mon[37789]: pgmap v1821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:21:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2410.618000 secs 2024-07-26T09:21:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2410.618000 secs 2024-07-26T09:21:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2410.618000 secs 2024-07-26T09:21:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:07 smithi044 ceph-mon[32404]: pgmap v1822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:21:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:07 smithi116 ceph-mon[37097]: pgmap v1822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:21:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:07 smithi171 ceph-mon[37789]: pgmap v1822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:21:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:08 smithi044 ceph-mon[32404]: pgmap v1823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:21:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:08 smithi116 ceph-mon[37097]: pgmap v1823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:21:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:08 smithi171 ceph-mon[37789]: pgmap v1823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:21:10.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:10.909+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:21:10.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:10.909+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:10.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:10.909+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:21:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:11 smithi044 ceph-mon[32404]: pgmap v1824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2415.618150 secs 2024-07-26T09:21:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:11 smithi116 ceph-mon[37097]: pgmap v1824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2415.618150 secs 2024-07-26T09:21:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:11 smithi171 ceph-mon[37789]: pgmap v1824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2415.618150 secs 2024-07-26T09:21:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:12 smithi044 ceph-mon[32404]: pgmap v1825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:12 smithi116 ceph-mon[37097]: pgmap v1825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:12 smithi171 ceph-mon[37789]: pgmap v1825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:15 smithi044 ceph-mon[32404]: pgmap v1826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:15 smithi116 ceph-mon[37097]: pgmap v1826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:15 smithi171 ceph-mon[37789]: pgmap v1826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2420.618313 secs 2024-07-26T09:21:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:16 smithi044 ceph-mon[32404]: pgmap v1827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2420.618313 secs 2024-07-26T09:21:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:16 smithi116 ceph-mon[37097]: pgmap v1827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2420.618313 secs 2024-07-26T09:21:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:16 smithi171 ceph-mon[37789]: pgmap v1827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:19 smithi044 ceph-mon[32404]: pgmap v1828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:19 smithi116 ceph-mon[37097]: pgmap v1828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:19 smithi171 ceph-mon[37789]: pgmap v1828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:20 smithi044 ceph-mon[32404]: pgmap v1829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:20 smithi116 ceph-mon[37097]: pgmap v1829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:20 smithi171 ceph-mon[37789]: pgmap v1829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2425.618459 secs 2024-07-26T09:21:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2425.618459 secs 2024-07-26T09:21:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2425.618459 secs 2024-07-26T09:21:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:22 smithi044 ceph-mon[32404]: pgmap v1830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:22 smithi116 ceph-mon[37097]: pgmap v1830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:22 smithi171 ceph-mon[37789]: pgmap v1830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:24 smithi044 ceph-mon[32404]: pgmap v1831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:24 smithi116 ceph-mon[37097]: pgmap v1831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:24 smithi171 ceph-mon[37789]: pgmap v1831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:25.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:25.918+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:21:25.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:25.918+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:25.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:25.918+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:21:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2430.618590 secs 2024-07-26T09:21:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:26 smithi044 ceph-mon[32404]: pgmap v1832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:21:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:26 smithi044 ceph-mon[32404]: pgmap v1833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:21:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2430.618590 secs 2024-07-26T09:21:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:26 smithi116 ceph-mon[37097]: pgmap v1832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:21:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:26 smithi116 ceph-mon[37097]: pgmap v1833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:21:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2430.618590 secs 2024-07-26T09:21:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:26 smithi171 ceph-mon[37789]: pgmap v1832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:21:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:26 smithi171 ceph-mon[37789]: pgmap v1833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:21:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:28 smithi044 ceph-mon[32404]: pgmap v1834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:28 smithi116 ceph-mon[37097]: pgmap v1834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:28 smithi171 ceph-mon[37789]: pgmap v1834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:30 smithi044 ceph-mon[32404]: pgmap v1835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:30 smithi116 ceph-mon[37097]: pgmap v1835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:30 smithi171 ceph-mon[37789]: pgmap v1835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2435.618726 secs 2024-07-26T09:21:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2435.618726 secs 2024-07-26T09:21:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2435.618726 secs 2024-07-26T09:21:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:32 smithi044 ceph-mon[32404]: pgmap v1836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:32 smithi116 ceph-mon[37097]: pgmap v1836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:32 smithi171 ceph-mon[37789]: pgmap v1836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:34 smithi044 ceph-mon[32404]: pgmap v1837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:34 smithi116 ceph-mon[37097]: pgmap v1837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:34 smithi171 ceph-mon[37789]: pgmap v1837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:21:36.675 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2440.618850 secs 2024-07-26T09:21:36.675 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:36 smithi044 ceph-mon[32404]: pgmap v1838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2440.618850 secs 2024-07-26T09:21:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:36 smithi116 ceph-mon[37097]: pgmap v1838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2440.618850 secs 2024-07-26T09:21:36.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:36 smithi171 ceph-mon[37789]: pgmap v1838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:21:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:38 smithi044 ceph-mon[32404]: pgmap v1839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:38 smithi116 ceph-mon[37097]: pgmap v1839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:38 smithi171 ceph-mon[37789]: pgmap v1839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:40 smithi044 ceph-mon[32404]: pgmap v1840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:40 smithi116 ceph-mon[37097]: pgmap v1840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:40 smithi171 ceph-mon[37789]: pgmap v1840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:40.921+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:21:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:40.921+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:40.921+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:21:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2445.618995 secs 2024-07-26T09:21:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2445.618995 secs 2024-07-26T09:21:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2445.618995 secs 2024-07-26T09:21:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:42 smithi044 ceph-mon[32404]: pgmap v1841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:42 smithi116 ceph-mon[37097]: pgmap v1841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:42 smithi171 ceph-mon[37789]: pgmap v1841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:44 smithi044 ceph-mon[32404]: pgmap v1842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:44 smithi116 ceph-mon[37097]: pgmap v1842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:44 smithi171 ceph-mon[37789]: pgmap v1842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2450.619157 secs 2024-07-26T09:21:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:46 smithi044 ceph-mon[32404]: pgmap v1843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2450.619157 secs 2024-07-26T09:21:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:46 smithi116 ceph-mon[37097]: pgmap v1843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2450.619157 secs 2024-07-26T09:21:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:46 smithi171 ceph-mon[37789]: pgmap v1843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:48 smithi044 ceph-mon[32404]: pgmap v1844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:48 smithi116 ceph-mon[37097]: pgmap v1844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:48 smithi171 ceph-mon[37789]: pgmap v1844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:50 smithi044 ceph-mon[32404]: pgmap v1845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:50 smithi116 ceph-mon[37097]: pgmap v1845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:50 smithi171 ceph-mon[37789]: pgmap v1845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:21:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2455.619296 secs 2024-07-26T09:21:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2455.619296 secs 2024-07-26T09:21:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2455.619296 secs 2024-07-26T09:21:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:52 smithi044 ceph-mon[32404]: pgmap v1846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:52 smithi116 ceph-mon[37097]: pgmap v1846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:52 smithi171 ceph-mon[37789]: pgmap v1846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:54 smithi044 ceph-mon[32404]: pgmap v1847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:54 smithi116 ceph-mon[37097]: pgmap v1847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:54 smithi171 ceph-mon[37789]: pgmap v1847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:21:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:55.930+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:21:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:55.930+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:55.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:21:55.930+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:21:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2460.619440 secs 2024-07-26T09:21:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:56 smithi044 ceph-mon[32404]: pgmap v1848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2460.619440 secs 2024-07-26T09:21:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:56 smithi116 ceph-mon[37097]: pgmap v1848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2460.619440 secs 2024-07-26T09:21:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:56 smithi171 ceph-mon[37789]: pgmap v1848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:21:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:21:58 smithi044 ceph-mon[32404]: pgmap v1849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:21:58 smithi116 ceph-mon[37097]: pgmap v1849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:21:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:21:58 smithi171 ceph-mon[37789]: pgmap v1849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:00 smithi044 ceph-mon[32404]: pgmap v1850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:00 smithi116 ceph-mon[37097]: pgmap v1850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:00 smithi171 ceph-mon[37789]: pgmap v1850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2465.619611 secs 2024-07-26T09:22:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2465.619611 secs 2024-07-26T09:22:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2465.619611 secs 2024-07-26T09:22:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:02 smithi116 ceph-mon[37097]: pgmap v1851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:02 smithi171 ceph-mon[37789]: pgmap v1851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:02 smithi044 ceph-mon[32404]: pgmap v1851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:04 smithi171 ceph-mon[37789]: pgmap v1852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:04 smithi044 ceph-mon[32404]: pgmap v1852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:04 smithi116 ceph-mon[37097]: pgmap v1852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2470.619764 secs 2024-07-26T09:22:06.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:06 smithi171 ceph-mon[37789]: pgmap v1853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2470.619764 secs 2024-07-26T09:22:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:06 smithi044 ceph-mon[32404]: pgmap v1853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2470.619764 secs 2024-07-26T09:22:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:06 smithi116 ceph-mon[37097]: pgmap v1853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:08 smithi171 ceph-mon[37789]: pgmap v1854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:08 smithi044 ceph-mon[32404]: pgmap v1854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:08 smithi116 ceph-mon[37097]: pgmap v1854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:10 smithi171 ceph-mon[37789]: pgmap v1855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:10.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:10.933+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:22:10.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:10.933+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:10.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:10.933+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:22:10.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:10 smithi044 ceph-mon[32404]: pgmap v1855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:10 smithi116 ceph-mon[37097]: pgmap v1855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2475.619926 secs 2024-07-26T09:22:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2475.619926 secs 2024-07-26T09:22:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2475.619926 secs 2024-07-26T09:22:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:12 smithi171 ceph-mon[37789]: pgmap v1856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:12 smithi044 ceph-mon[32404]: pgmap v1856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:12 smithi116 ceph-mon[37097]: pgmap v1856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:14 smithi171 ceph-mon[37789]: pgmap v1857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:14 smithi044 ceph-mon[32404]: pgmap v1857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:14 smithi116 ceph-mon[37097]: pgmap v1857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2480.620033 secs 2024-07-26T09:22:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:16 smithi171 ceph-mon[37789]: pgmap v1858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2480.620033 secs 2024-07-26T09:22:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:16 smithi044 ceph-mon[32404]: pgmap v1858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2480.620033 secs 2024-07-26T09:22:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:16 smithi116 ceph-mon[37097]: pgmap v1858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:18 smithi171 ceph-mon[37789]: pgmap v1859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:18 smithi044 ceph-mon[32404]: pgmap v1859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:18 smithi116 ceph-mon[37097]: pgmap v1859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:20 smithi171 ceph-mon[37789]: pgmap v1860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:21.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:20 smithi044 ceph-mon[32404]: pgmap v1860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:20 smithi116 ceph-mon[37097]: pgmap v1860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2485.620194 secs 2024-07-26T09:22:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2485.620194 secs 2024-07-26T09:22:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2485.620194 secs 2024-07-26T09:22:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:22 smithi044 ceph-mon[32404]: pgmap v1861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:22 smithi116 ceph-mon[37097]: pgmap v1861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:22 smithi171 ceph-mon[37789]: pgmap v1861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:24 smithi044 ceph-mon[32404]: pgmap v1862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:24 smithi116 ceph-mon[37097]: pgmap v1862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:24 smithi171 ceph-mon[37789]: pgmap v1862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:22:25.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:25.941+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:22:25.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:25.941+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:22:25.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:25.941+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:25.944 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2490.620342 secs 2024-07-26T09:22:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:26 smithi044 ceph-mon[32404]: pgmap v1863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2490.620342 secs 2024-07-26T09:22:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:26 smithi116 ceph-mon[37097]: pgmap v1863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2490.620342 secs 2024-07-26T09:22:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:26 smithi171 ceph-mon[37789]: pgmap v1863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:22:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:22:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: pgmap v1864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: pgmap v1865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:22:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: pgmap v1864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: pgmap v1865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:22:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:22:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: pgmap v1864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: pgmap v1865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:22:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:22:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:29 smithi044 ceph-mon[32404]: pgmap v1866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:29 smithi116 ceph-mon[37097]: pgmap v1866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:29 smithi171 ceph-mon[37789]: pgmap v1866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:30.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:30.100+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:22:30.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:30.100+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:22:30.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:30.101+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:22:30.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:30.101+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:22:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2495.620505 secs 2024-07-26T09:22:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:31 smithi044 ceph-mon[32404]: pgmap v1867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2495.620505 secs 2024-07-26T09:22:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:31 smithi116 ceph-mon[37097]: pgmap v1867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2495.620505 secs 2024-07-26T09:22:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:31 smithi171 ceph-mon[37789]: pgmap v1867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:33 smithi044 ceph-mon[32404]: pgmap v1868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:22:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:33 smithi116 ceph-mon[37097]: pgmap v1868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:22:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:33 smithi171 ceph-mon[37789]: pgmap v1868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:22:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:35 smithi044 ceph-mon[32404]: pgmap v1869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:35 smithi116 ceph-mon[37097]: pgmap v1869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:35 smithi171 ceph-mon[37789]: pgmap v1869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:22:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2500.620697 secs 2024-07-26T09:22:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2500.620697 secs 2024-07-26T09:22:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2500.620697 secs 2024-07-26T09:22:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:37 smithi044 ceph-mon[32404]: pgmap v1870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:37 smithi116 ceph-mon[37097]: pgmap v1870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:37 smithi171 ceph-mon[37789]: pgmap v1870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:22:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:39 smithi044 ceph-mon[32404]: pgmap v1871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:39 smithi116 ceph-mon[37097]: pgmap v1871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:39 smithi171 ceph-mon[37789]: pgmap v1871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:40.944+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:22:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:40.944+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:40.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:40.944+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:22:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2505.620847 secs 2024-07-26T09:22:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:41 smithi044 ceph-mon[32404]: pgmap v1872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2505.620847 secs 2024-07-26T09:22:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:41 smithi116 ceph-mon[37097]: pgmap v1872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2505.620847 secs 2024-07-26T09:22:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:41 smithi171 ceph-mon[37789]: pgmap v1872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:43 smithi044 ceph-mon[32404]: pgmap v1873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:43 smithi116 ceph-mon[37097]: pgmap v1873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:43 smithi171 ceph-mon[37789]: pgmap v1873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:45 smithi044 ceph-mon[32404]: pgmap v1874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:45 smithi116 ceph-mon[37097]: pgmap v1874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:45 smithi171 ceph-mon[37789]: pgmap v1874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2510.620935 secs 2024-07-26T09:22:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2510.620935 secs 2024-07-26T09:22:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2510.620935 secs 2024-07-26T09:22:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:47 smithi044 ceph-mon[32404]: pgmap v1875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:47 smithi116 ceph-mon[37097]: pgmap v1875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:47 smithi171 ceph-mon[37789]: pgmap v1875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:49 smithi044 ceph-mon[32404]: pgmap v1876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:49 smithi116 ceph-mon[37097]: pgmap v1876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:49 smithi171 ceph-mon[37789]: pgmap v1876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:22:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2515.621066 secs 2024-07-26T09:22:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2515.621066 secs 2024-07-26T09:22:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2515.621066 secs 2024-07-26T09:22:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:51 smithi171 ceph-mon[37789]: pgmap v1877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:51 smithi044 ceph-mon[32404]: pgmap v1877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:51 smithi116 ceph-mon[37097]: pgmap v1877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:22:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:53 smithi171 ceph-mon[37789]: pgmap v1878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:53 smithi044 ceph-mon[32404]: pgmap v1878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:53 smithi116 ceph-mon[37097]: pgmap v1878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:22:55.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:55.952+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:22:55.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:55.953+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:22:55.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:22:55.953+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:55 smithi171 ceph-mon[37789]: pgmap v1879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2520.621201 secs 2024-07-26T09:22:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:55 smithi044 ceph-mon[32404]: pgmap v1879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2520.621201 secs 2024-07-26T09:22:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:55 smithi116 ceph-mon[37097]: pgmap v1879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:22:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2520.621201 secs 2024-07-26T09:22:58.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:57 smithi171 ceph-mon[37789]: pgmap v1880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:57 smithi044 ceph-mon[32404]: pgmap v1880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:22:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:57 smithi116 ceph-mon[37097]: pgmap v1880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:22:59 smithi171 ceph-mon[37789]: pgmap v1881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:22:59 smithi044 ceph-mon[32404]: pgmap v1881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:22:59 smithi116 ceph-mon[37097]: pgmap v1881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2525.621347 secs 2024-07-26T09:23:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2525.621347 secs 2024-07-26T09:23:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2525.621347 secs 2024-07-26T09:23:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:01 smithi171 ceph-mon[37789]: pgmap v1882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:01 smithi044 ceph-mon[32404]: pgmap v1882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:01 smithi116 ceph-mon[37097]: pgmap v1882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:03 smithi171 ceph-mon[37789]: pgmap v1883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:03 smithi044 ceph-mon[32404]: pgmap v1883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:03 smithi116 ceph-mon[37097]: pgmap v1883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:05 smithi171 ceph-mon[37789]: pgmap v1884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2530.621455 secs 2024-07-26T09:23:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:05 smithi044 ceph-mon[32404]: pgmap v1884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2530.621455 secs 2024-07-26T09:23:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:05 smithi116 ceph-mon[37097]: pgmap v1884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2530.621455 secs 2024-07-26T09:23:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:07 smithi171 ceph-mon[37789]: pgmap v1885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:07 smithi044 ceph-mon[32404]: pgmap v1885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:07 smithi116 ceph-mon[37097]: pgmap v1885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:09 smithi171 ceph-mon[37789]: pgmap v1886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:09 smithi044 ceph-mon[32404]: pgmap v1886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:09 smithi116 ceph-mon[37097]: pgmap v1886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:10.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:10.956+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:23:10.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:10.956+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:23:10.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:10.956+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2535.621612 secs 2024-07-26T09:23:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2535.621612 secs 2024-07-26T09:23:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2535.621612 secs 2024-07-26T09:23:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:11 smithi044 ceph-mon[32404]: pgmap v1887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:11 smithi116 ceph-mon[37097]: pgmap v1887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:11 smithi171 ceph-mon[37789]: pgmap v1887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:13 smithi044 ceph-mon[32404]: pgmap v1888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:13 smithi116 ceph-mon[37097]: pgmap v1888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:13 smithi171 ceph-mon[37789]: pgmap v1888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:15.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:15.109+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:23:15.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:15.109+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:23:15.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:15.109+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:23:15.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:15.109+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:23:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:15 smithi044 ceph-mon[32404]: pgmap v1889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2540.621777 secs 2024-07-26T09:23:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:15 smithi116 ceph-mon[37097]: pgmap v1889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2540.621777 secs 2024-07-26T09:23:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:15 smithi171 ceph-mon[37789]: pgmap v1889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2540.621777 secs 2024-07-26T09:23:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:17 smithi044 ceph-mon[32404]: pgmap v1890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:17 smithi116 ceph-mon[37097]: pgmap v1890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:17 smithi171 ceph-mon[37789]: pgmap v1890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:19 smithi044 ceph-mon[32404]: pgmap v1891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:19 smithi116 ceph-mon[37097]: pgmap v1891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:19 smithi171 ceph-mon[37789]: pgmap v1891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:23:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2545.621916 secs 2024-07-26T09:23:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2545.621916 secs 2024-07-26T09:23:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2545.621916 secs 2024-07-26T09:23:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:21 smithi044 ceph-mon[32404]: pgmap v1892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:21 smithi116 ceph-mon[37097]: pgmap v1892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:21 smithi171 ceph-mon[37789]: pgmap v1892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:23 smithi044 ceph-mon[32404]: pgmap v1893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:23 smithi116 ceph-mon[37097]: pgmap v1893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:23 smithi171 ceph-mon[37789]: pgmap v1893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:25.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:25.964+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:23:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:25.964+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:23:25.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:25.964+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:25 smithi044 ceph-mon[32404]: pgmap v1894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2550.622060 secs 2024-07-26T09:23:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:25 smithi116 ceph-mon[37097]: pgmap v1894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2550.622060 secs 2024-07-26T09:23:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:25 smithi171 ceph-mon[37789]: pgmap v1894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2550.622060 secs 2024-07-26T09:23:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: pgmap v1895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: pgmap v1895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: pgmap v1895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:28.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: pgmap v1896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: pgmap v1897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:23:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: pgmap v1896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: pgmap v1897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:23:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:23:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:23:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: pgmap v1896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: pgmap v1897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:23:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:23:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:30 smithi044 ceph-mon[32404]: pgmap v1898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:23:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2555.622190 secs 2024-07-26T09:23:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:30 smithi116 ceph-mon[37097]: pgmap v1898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:23:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2555.622190 secs 2024-07-26T09:23:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:30 smithi171 ceph-mon[37789]: pgmap v1898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:23:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2555.622190 secs 2024-07-26T09:23:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:32 smithi044 ceph-mon[32404]: pgmap v1899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:23:33.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:32 smithi116 ceph-mon[37097]: pgmap v1899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:23:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:32 smithi171 ceph-mon[37789]: pgmap v1899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:23:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:34 smithi044 ceph-mon[32404]: pgmap v1900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:35 smithi116 ceph-mon[37097]: pgmap v1900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:35 smithi171 ceph-mon[37789]: pgmap v1900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:23:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2560.622334 secs 2024-07-26T09:23:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2560.622334 secs 2024-07-26T09:23:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2560.622334 secs 2024-07-26T09:23:37.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:37 smithi044 ceph-mon[32404]: pgmap v1901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:23:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:37 smithi116 ceph-mon[37097]: pgmap v1901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:23:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:37 smithi171 ceph-mon[37789]: pgmap v1901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:23:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:39 smithi116 ceph-mon[37097]: pgmap v1902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:23:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:39 smithi171 ceph-mon[37789]: pgmap v1902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:23:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:39 smithi044 ceph-mon[32404]: pgmap v1902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:23:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:40.967+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:23:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:40.967+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:23:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:40.967+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:41 smithi044 ceph-mon[32404]: pgmap v1903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2565.622457 secs 2024-07-26T09:23:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:41 smithi171 ceph-mon[37789]: pgmap v1903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2565.622457 secs 2024-07-26T09:23:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:41 smithi116 ceph-mon[37097]: pgmap v1903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:23:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2565.622457 secs 2024-07-26T09:23:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:43 smithi171 ceph-mon[37789]: pgmap v1904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:43 smithi044 ceph-mon[32404]: pgmap v1904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:43 smithi116 ceph-mon[37097]: pgmap v1904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:45 smithi171 ceph-mon[37789]: pgmap v1905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:45 smithi044 ceph-mon[32404]: pgmap v1905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:45.573 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:45 smithi116 ceph-mon[37097]: pgmap v1905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:23:46.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2570.622594 secs 2024-07-26T09:23:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2570.622594 secs 2024-07-26T09:23:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2570.622594 secs 2024-07-26T09:23:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:47 smithi171 ceph-mon[37789]: pgmap v1906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:47 smithi044 ceph-mon[32404]: pgmap v1906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:47 smithi116 ceph-mon[37097]: pgmap v1906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:49 smithi171 ceph-mon[37789]: pgmap v1907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:49.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:49 smithi044 ceph-mon[32404]: pgmap v1907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:49 smithi116 ceph-mon[37097]: pgmap v1907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:51.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:51 smithi044 ceph-mon[32404]: pgmap v1908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:51.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2575.622731 secs 2024-07-26T09:23:51.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:51 smithi116 ceph-mon[37097]: pgmap v1908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:51.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2575.622731 secs 2024-07-26T09:23:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:51 smithi171 ceph-mon[37789]: pgmap v1908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:23:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2575.622731 secs 2024-07-26T09:23:53.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:53 smithi171 ceph-mon[37789]: pgmap v1909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:53 smithi044 ceph-mon[32404]: pgmap v1909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:53 smithi116 ceph-mon[37097]: pgmap v1909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:55 smithi171 ceph-mon[37789]: pgmap v1910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:55 smithi044 ceph-mon[32404]: pgmap v1910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:55 smithi116 ceph-mon[37097]: pgmap v1910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:55.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:55.975+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:23:55.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:55.975+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:23:55.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:23:55.975+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2580.622893 secs 2024-07-26T09:23:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2580.622893 secs 2024-07-26T09:23:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2580.622893 secs 2024-07-26T09:23:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:57 smithi044 ceph-mon[32404]: pgmap v1911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:57 smithi116 ceph-mon[37097]: pgmap v1911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:57 smithi171 ceph-mon[37789]: pgmap v1911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:23:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:23:59 smithi044 ceph-mon[32404]: pgmap v1912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:23:59 smithi116 ceph-mon[37097]: pgmap v1912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:23:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:23:59 smithi171 ceph-mon[37789]: pgmap v1912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:01.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:01 smithi171 ceph-mon[37789]: pgmap v1913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:01.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2585.623032 secs 2024-07-26T09:24:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:01 smithi044 ceph-mon[32404]: pgmap v1913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2585.623032 secs 2024-07-26T09:24:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:01 smithi116 ceph-mon[37097]: pgmap v1913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2585.623032 secs 2024-07-26T09:24:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:03 smithi044 ceph-mon[32404]: pgmap v1914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:03 smithi116 ceph-mon[37097]: pgmap v1914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:03 smithi171 ceph-mon[37789]: pgmap v1914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:05 smithi044 ceph-mon[32404]: pgmap v1915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:05 smithi116 ceph-mon[37097]: pgmap v1915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:05 smithi171 ceph-mon[37789]: pgmap v1915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2590.623142 secs 2024-07-26T09:24:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2590.623142 secs 2024-07-26T09:24:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2590.623142 secs 2024-07-26T09:24:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:07 smithi044 ceph-mon[32404]: pgmap v1916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:07 smithi116 ceph-mon[37097]: pgmap v1916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:07 smithi171 ceph-mon[37789]: pgmap v1916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:09 smithi044 ceph-mon[32404]: pgmap v1917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:09 smithi116 ceph-mon[37097]: pgmap v1917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:09 smithi171 ceph-mon[37789]: pgmap v1917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:10.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:10.979+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:24:10.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:10.979+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:24:10.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:10.979+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:11 smithi044 ceph-mon[32404]: pgmap v1918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2595.623261 secs 2024-07-26T09:24:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:11 smithi116 ceph-mon[37097]: pgmap v1918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2595.623261 secs 2024-07-26T09:24:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:11 smithi171 ceph-mon[37789]: pgmap v1918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2595.623261 secs 2024-07-26T09:24:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:13 smithi044 ceph-mon[32404]: pgmap v1919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:13 smithi116 ceph-mon[37097]: pgmap v1919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:13 smithi171 ceph-mon[37789]: pgmap v1919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:15 smithi044 ceph-mon[32404]: pgmap v1920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:15 smithi116 ceph-mon[37097]: pgmap v1920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:15 smithi171 ceph-mon[37789]: pgmap v1920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2600.623398 secs 2024-07-26T09:24:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2600.623398 secs 2024-07-26T09:24:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2600.623398 secs 2024-07-26T09:24:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:17 smithi044 ceph-mon[32404]: pgmap v1921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:17 smithi116 ceph-mon[37097]: pgmap v1921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:17 smithi171 ceph-mon[37789]: pgmap v1921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:19 smithi044 ceph-mon[32404]: pgmap v1922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:19 smithi116 ceph-mon[37097]: pgmap v1922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:19 smithi171 ceph-mon[37789]: pgmap v1922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:21 smithi044 ceph-mon[32404]: pgmap v1923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2605.623556 secs 2024-07-26T09:24:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:21 smithi116 ceph-mon[37097]: pgmap v1923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2605.623556 secs 2024-07-26T09:24:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:21 smithi171 ceph-mon[37789]: pgmap v1923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:24:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2605.623556 secs 2024-07-26T09:24:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:23 smithi044 ceph-mon[32404]: pgmap v1924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:23 smithi116 ceph-mon[37097]: pgmap v1924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:23 smithi171 ceph-mon[37789]: pgmap v1924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:25 smithi044 ceph-mon[32404]: pgmap v1925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:25 smithi116 ceph-mon[37097]: pgmap v1925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:25 smithi171 ceph-mon[37789]: pgmap v1925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:25.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:25.985+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:24:25.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:25.985+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:24:25.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:25.985+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2610.623699 secs 2024-07-26T09:24:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2610.623699 secs 2024-07-26T09:24:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2610.623699 secs 2024-07-26T09:24:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:27 smithi044 ceph-mon[32404]: pgmap v1926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:27 smithi116 ceph-mon[37097]: pgmap v1926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:27 smithi171 ceph-mon[37789]: pgmap v1926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:28 smithi044 ceph-mon[32404]: pgmap v1927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:28.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:28 smithi116 ceph-mon[37097]: pgmap v1927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:28 smithi171 ceph-mon[37789]: pgmap v1927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:24:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:24:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:24:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:30 smithi171 ceph-mon[37789]: pgmap v1928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:24:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:30 smithi171 ceph-mon[37789]: pgmap v1929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:30 smithi044 ceph-mon[32404]: pgmap v1928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:24:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:30 smithi044 ceph-mon[32404]: pgmap v1929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:30 smithi116 ceph-mon[37097]: pgmap v1928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:24:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:30 smithi116 ceph-mon[37097]: pgmap v1929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2615.623864 secs 2024-07-26T09:24:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2615.623864 secs 2024-07-26T09:24:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2615.623864 secs 2024-07-26T09:24:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:32 smithi171 ceph-mon[37789]: pgmap v1930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:32 smithi044 ceph-mon[32404]: pgmap v1930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:24:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:32 smithi116 ceph-mon[37097]: pgmap v1930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:24:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:34 smithi171 ceph-mon[37789]: pgmap v1931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:34 smithi044 ceph-mon[32404]: pgmap v1931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:34 smithi116 ceph-mon[37097]: pgmap v1931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:36 smithi171 ceph-mon[37789]: pgmap v1932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2620.624000 secs 2024-07-26T09:24:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:36 smithi044 ceph-mon[32404]: pgmap v1932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:36.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2620.624000 secs 2024-07-26T09:24:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:36 smithi116 ceph-mon[37097]: pgmap v1932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2620.624000 secs 2024-07-26T09:24:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:38 smithi171 ceph-mon[37789]: pgmap v1933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:38 smithi044 ceph-mon[32404]: pgmap v1933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:38 smithi116 ceph-mon[37097]: pgmap v1933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:24:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:40 smithi171 ceph-mon[37789]: pgmap v1934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:24:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:40 smithi044 ceph-mon[32404]: pgmap v1934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:24:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:40 smithi116 ceph-mon[37097]: pgmap v1934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:24:40.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:40.987+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:24:40.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:40.987+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:24:40.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:40.987+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2625.624166 secs 2024-07-26T09:24:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2625.624166 secs 2024-07-26T09:24:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2625.624166 secs 2024-07-26T09:24:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:42 smithi044 ceph-mon[32404]: pgmap v1935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:42 smithi116 ceph-mon[37097]: pgmap v1935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:42 smithi171 ceph-mon[37789]: pgmap v1935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:44 smithi044 ceph-mon[32404]: pgmap v1936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:24:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:44 smithi116 ceph-mon[37097]: pgmap v1936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:24:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:44 smithi171 ceph-mon[37789]: pgmap v1936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:24:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:46 smithi044 ceph-mon[32404]: pgmap v1937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2630.624319 secs 2024-07-26T09:24:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:46 smithi116 ceph-mon[37097]: pgmap v1937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2630.624319 secs 2024-07-26T09:24:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:46 smithi171 ceph-mon[37789]: pgmap v1937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2630.624319 secs 2024-07-26T09:24:48.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:48 smithi044 ceph-mon[32404]: pgmap v1938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:48 smithi116 ceph-mon[37097]: pgmap v1938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:48 smithi171 ceph-mon[37789]: pgmap v1938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:50 smithi044 ceph-mon[32404]: pgmap v1939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:50 smithi116 ceph-mon[37097]: pgmap v1939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:50 smithi171 ceph-mon[37789]: pgmap v1939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:51.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2635.624466 secs 2024-07-26T09:24:51.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2635.624466 secs 2024-07-26T09:24:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2635.624466 secs 2024-07-26T09:24:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:52 smithi044 ceph-mon[32404]: pgmap v1940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:52 smithi116 ceph-mon[37097]: pgmap v1940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:52 smithi171 ceph-mon[37789]: pgmap v1940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:54 smithi044 ceph-mon[32404]: pgmap v1941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:24:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:54 smithi116 ceph-mon[37097]: pgmap v1941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:24:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:54 smithi171 ceph-mon[37789]: pgmap v1941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:24:55.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:55.997+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:24:55.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:55.997+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:24:55.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:24:55.997+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:56 smithi044 ceph-mon[32404]: pgmap v1942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2640.624601 secs 2024-07-26T09:24:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:56 smithi116 ceph-mon[37097]: pgmap v1942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2640.624601 secs 2024-07-26T09:24:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:56 smithi171 ceph-mon[37789]: pgmap v1942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:24:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2640.624601 secs 2024-07-26T09:24:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:24:58 smithi044 ceph-mon[32404]: pgmap v1943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:24:58 smithi116 ceph-mon[37097]: pgmap v1943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:24:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:24:58 smithi171 ceph-mon[37789]: pgmap v1943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:00 smithi044 ceph-mon[32404]: pgmap v1944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:00 smithi116 ceph-mon[37097]: pgmap v1944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:00 smithi171 ceph-mon[37789]: pgmap v1944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:01.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2645.624767 secs 2024-07-26T09:25:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2645.624767 secs 2024-07-26T09:25:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2645.624767 secs 2024-07-26T09:25:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:02 smithi044 ceph-mon[32404]: pgmap v1945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:02 smithi116 ceph-mon[37097]: pgmap v1945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:02 smithi171 ceph-mon[37789]: pgmap v1945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:04 smithi044 ceph-mon[32404]: pgmap v1946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:04 smithi116 ceph-mon[37097]: pgmap v1946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:04 smithi171 ceph-mon[37789]: pgmap v1946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:06 smithi044 ceph-mon[32404]: pgmap v1947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2650.624926 secs 2024-07-26T09:25:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:06 smithi116 ceph-mon[37097]: pgmap v1947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2650.624926 secs 2024-07-26T09:25:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:06 smithi171 ceph-mon[37789]: pgmap v1947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2650.624926 secs 2024-07-26T09:25:08.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:08 smithi044 ceph-mon[32404]: pgmap v1948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:08 smithi116 ceph-mon[37097]: pgmap v1948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:08 smithi171 ceph-mon[37789]: pgmap v1948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:10 smithi044 ceph-mon[32404]: pgmap v1949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:10 smithi116 ceph-mon[37097]: pgmap v1949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:10 smithi171 ceph-mon[37789]: pgmap v1949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:11.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:11.000+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:25:11.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:11.001+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:25:11.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:11.001+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2655.625094 secs 2024-07-26T09:25:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2655.625094 secs 2024-07-26T09:25:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2655.625094 secs 2024-07-26T09:25:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:12 smithi044 ceph-mon[32404]: pgmap v1950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:12 smithi116 ceph-mon[37097]: pgmap v1950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:12 smithi171 ceph-mon[37789]: pgmap v1950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:14 smithi044 ceph-mon[32404]: pgmap v1951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:14 smithi116 ceph-mon[37097]: pgmap v1951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:14 smithi171 ceph-mon[37789]: pgmap v1951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:16 smithi044 ceph-mon[32404]: pgmap v1952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2660.625227 secs 2024-07-26T09:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:16 smithi116 ceph-mon[37097]: pgmap v1952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2660.625227 secs 2024-07-26T09:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:16 smithi171 ceph-mon[37789]: pgmap v1952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2660.625227 secs 2024-07-26T09:25:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:18 smithi044 ceph-mon[32404]: pgmap v1953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:18 smithi116 ceph-mon[37097]: pgmap v1953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:18 smithi171 ceph-mon[37789]: pgmap v1953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:20 smithi044 ceph-mon[32404]: pgmap v1954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:20 smithi116 ceph-mon[37097]: pgmap v1954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:20 smithi171 ceph-mon[37789]: pgmap v1954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2665.625388 secs 2024-07-26T09:25:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2665.625388 secs 2024-07-26T09:25:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2665.625388 secs 2024-07-26T09:25:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:22 smithi044 ceph-mon[32404]: pgmap v1955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:22 smithi116 ceph-mon[37097]: pgmap v1955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:22 smithi171 ceph-mon[37789]: pgmap v1955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:24 smithi044 ceph-mon[32404]: pgmap v1956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:24 smithi116 ceph-mon[37097]: pgmap v1956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:24 smithi171 ceph-mon[37789]: pgmap v1956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:26.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:26.003+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:25:26.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:26.003+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:26.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:26.003+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:25:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:26 smithi044 ceph-mon[32404]: pgmap v1957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2670.625534 secs 2024-07-26T09:25:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:26 smithi116 ceph-mon[37097]: pgmap v1957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2670.625534 secs 2024-07-26T09:25:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:26 smithi171 ceph-mon[37789]: pgmap v1957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2670.625534 secs 2024-07-26T09:25:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:28 smithi044 ceph-mon[32404]: pgmap v1958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:28 smithi116 ceph-mon[37097]: pgmap v1958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:28 smithi171 ceph-mon[37789]: pgmap v1958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:31 smithi171 ceph-mon[37789]: pgmap v1959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2675.625658 secs 2024-07-26T09:25:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:31 smithi044 ceph-mon[32404]: pgmap v1959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2675.625658 secs 2024-07-26T09:25:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:31 smithi116 ceph-mon[37097]: pgmap v1959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2675.625658 secs 2024-07-26T09:25:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:32 smithi171 ceph-mon[37789]: pgmap v1960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:25:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:32 smithi171 ceph-mon[37789]: pgmap v1961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:32 smithi044 ceph-mon[32404]: pgmap v1960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:25:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:32 smithi044 ceph-mon[32404]: pgmap v1961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:32 smithi116 ceph-mon[37097]: pgmap v1960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:32 smithi116 ceph-mon[37097]: pgmap v1961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:25:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:33 smithi116 ceph-mon[37097]: pgmap v1962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:33 smithi171 ceph-mon[37789]: pgmap v1962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:33 smithi044 ceph-mon[32404]: pgmap v1962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:35 smithi116 ceph-mon[37097]: pgmap v1963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:35 smithi171 ceph-mon[37789]: pgmap v1963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:35 smithi044 ceph-mon[32404]: pgmap v1963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2680.625725 secs 2024-07-26T09:25:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2680.625725 secs 2024-07-26T09:25:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2680.625725 secs 2024-07-26T09:25:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:37 smithi171 ceph-mon[37789]: pgmap v1964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:37 smithi044 ceph-mon[32404]: pgmap v1964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:37 smithi116 ceph-mon[37097]: pgmap v1964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:39 smithi116 ceph-mon[37097]: pgmap v1965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:39 smithi171 ceph-mon[37789]: pgmap v1965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:39 smithi044 ceph-mon[32404]: pgmap v1965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:25:41.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:41.006+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:25:41.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:41.006+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:25:41.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:41.006+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2685.625887 secs 2024-07-26T09:25:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2685.625887 secs 2024-07-26T09:25:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2685.625887 secs 2024-07-26T09:25:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:41 smithi044 ceph-mon[32404]: pgmap v1966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:25:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:41 smithi116 ceph-mon[37097]: pgmap v1966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:25:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:41 smithi171 ceph-mon[37789]: pgmap v1966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:25:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:43 smithi116 ceph-mon[37097]: pgmap v1967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:43 smithi171 ceph-mon[37789]: pgmap v1967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:43 smithi044 ceph-mon[32404]: pgmap v1967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:45 smithi116 ceph-mon[37097]: pgmap v1968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:45 smithi171 ceph-mon[37789]: pgmap v1968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:45 smithi044 ceph-mon[32404]: pgmap v1968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2690.625995 secs 2024-07-26T09:25:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2690.625995 secs 2024-07-26T09:25:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2690.625995 secs 2024-07-26T09:25:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:47 smithi171 ceph-mon[37789]: pgmap v1969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:47 smithi044 ceph-mon[32404]: pgmap v1969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:47 smithi116 ceph-mon[37097]: pgmap v1969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:49 smithi116 ceph-mon[37097]: pgmap v1970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:49 smithi171 ceph-mon[37789]: pgmap v1970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:49 smithi044 ceph-mon[32404]: pgmap v1970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2695.626175 secs 2024-07-26T09:25:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2695.626175 secs 2024-07-26T09:25:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2695.626175 secs 2024-07-26T09:25:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:51 smithi044 ceph-mon[32404]: pgmap v1971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:51 smithi116 ceph-mon[37097]: pgmap v1971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:51 smithi171 ceph-mon[37789]: pgmap v1971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:53 smithi116 ceph-mon[37097]: pgmap v1972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:53.601 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:53 smithi171 ceph-mon[37789]: pgmap v1972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:53 smithi044 ceph-mon[32404]: pgmap v1972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:25:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:55 smithi171 ceph-mon[37789]: pgmap v1973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:55 smithi044 ceph-mon[32404]: pgmap v1973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:55 smithi116 ceph-mon[37097]: pgmap v1973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:25:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:56.015+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:25:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:56.015+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:25:56.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:25:56.015+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2700.626311 secs 2024-07-26T09:25:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2700.626311 secs 2024-07-26T09:25:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2700.626311 secs 2024-07-26T09:25:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:57 smithi171 ceph-mon[37789]: pgmap v1974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:57 smithi044 ceph-mon[32404]: pgmap v1974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:57 smithi116 ceph-mon[37097]: pgmap v1974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:25:59 smithi171 ceph-mon[37789]: pgmap v1975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:25:59 smithi044 ceph-mon[32404]: pgmap v1975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:25:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:25:59 smithi116 ceph-mon[37097]: pgmap v1975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2705.626457 secs 2024-07-26T09:26:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2705.626457 secs 2024-07-26T09:26:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2705.626457 secs 2024-07-26T09:26:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:01 smithi044 ceph-mon[32404]: pgmap v1976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:01 smithi116 ceph-mon[37097]: pgmap v1976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:01 smithi171 ceph-mon[37789]: pgmap v1976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:03 smithi171 ceph-mon[37789]: pgmap v1977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:03 smithi044 ceph-mon[32404]: pgmap v1977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:03 smithi116 ceph-mon[37097]: pgmap v1977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:05 smithi171 ceph-mon[37789]: pgmap v1978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:05 smithi044 ceph-mon[32404]: pgmap v1978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:05 smithi116 ceph-mon[37097]: pgmap v1978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2710.626592 secs 2024-07-26T09:26:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2710.626592 secs 2024-07-26T09:26:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2710.626592 secs 2024-07-26T09:26:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:07 smithi171 ceph-mon[37789]: pgmap v1979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:07 smithi044 ceph-mon[32404]: pgmap v1979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:07 smithi116 ceph-mon[37097]: pgmap v1979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:09 smithi171 ceph-mon[37789]: pgmap v1980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:09 smithi044 ceph-mon[32404]: pgmap v1980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:09 smithi116 ceph-mon[37097]: pgmap v1980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:11.017+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:26:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:11.017+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:26:11.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:11.017+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2715.626734 secs 2024-07-26T09:26:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2715.626734 secs 2024-07-26T09:26:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2715.626734 secs 2024-07-26T09:26:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:11 smithi044 ceph-mon[32404]: pgmap v1981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:11 smithi116 ceph-mon[37097]: pgmap v1981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:11 smithi171 ceph-mon[37789]: pgmap v1981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:13 smithi171 ceph-mon[37789]: pgmap v1982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:13 smithi044 ceph-mon[32404]: pgmap v1982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:13 smithi116 ceph-mon[37097]: pgmap v1982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T09:26:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:26:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540c1eb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802a430 tx=0x7fcf5804d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:26:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540c1eb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcf5802a430 tx=0x7fcf5804d000 comp rx=0 tx=0).stop 2024-07-26T09:26:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540c1eb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T09:26:14.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540c1eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:26:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.685+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540c1eb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:26:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.686+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:26:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.686+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:26:14.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:14.688+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf5804e000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:26:15.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:15.141+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:26:15.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:15.141+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:26:15.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:15.141+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:26:15.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:15.141+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:26:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:15 smithi044 ceph-mon[32404]: pgmap v1983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:15 smithi171 ceph-mon[37789]: pgmap v1983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:15 smithi116 ceph-mon[37097]: pgmap v1983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:26:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2720.626828 secs 2024-07-26T09:26:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2720.626828 secs 2024-07-26T09:26:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2720.626828 secs 2024-07-26T09:26:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:17 smithi171 ceph-mon[37789]: pgmap v1984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:17 smithi044 ceph-mon[32404]: pgmap v1984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:17 smithi116 ceph-mon[37097]: pgmap v1984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:19 smithi171 ceph-mon[37789]: pgmap v1985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:19 smithi044 ceph-mon[32404]: pgmap v1985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:19 smithi116 ceph-mon[37097]: pgmap v1985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2725.626988 secs 2024-07-26T09:26:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2725.626988 secs 2024-07-26T09:26:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2725.626988 secs 2024-07-26T09:26:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:21 smithi044 ceph-mon[32404]: pgmap v1986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:21 smithi116 ceph-mon[37097]: pgmap v1986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:21 smithi171 ceph-mon[37789]: pgmap v1986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:23 smithi171 ceph-mon[37789]: pgmap v1987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:23 smithi044 ceph-mon[32404]: pgmap v1987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:23 smithi116 ceph-mon[37097]: pgmap v1987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:25 smithi171 ceph-mon[37789]: pgmap v1988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:25 smithi044 ceph-mon[32404]: pgmap v1988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:25 smithi116 ceph-mon[37097]: pgmap v1988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:26.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:26.025+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:26:26.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:26.025+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:26:26.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:26.025+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2730.627126 secs 2024-07-26T09:26:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2730.627126 secs 2024-07-26T09:26:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2730.627126 secs 2024-07-26T09:26:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:27 smithi171 ceph-mon[37789]: pgmap v1989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:27 smithi044 ceph-mon[32404]: pgmap v1989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:27 smithi116 ceph-mon[37097]: pgmap v1989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:29 smithi171 ceph-mon[37789]: pgmap v1990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:29 smithi044 ceph-mon[32404]: pgmap v1990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:29 smithi116 ceph-mon[37097]: pgmap v1990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2735.627285 secs 2024-07-26T09:26:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2735.627285 secs 2024-07-26T09:26:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2735.627285 secs 2024-07-26T09:26:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:31 smithi044 ceph-mon[32404]: pgmap v1991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:31 smithi116 ceph-mon[37097]: pgmap v1991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:31 smithi171 ceph-mon[37789]: pgmap v1991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:33 smithi171 ceph-mon[37789]: pgmap v1992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:33 smithi044 ceph-mon[32404]: pgmap v1992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:33 smithi116 ceph-mon[37097]: pgmap v1992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:26:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:36 smithi171 ceph-mon[37789]: pgmap v1993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2740.627393 secs 2024-07-26T09:26:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:36 smithi044 ceph-mon[32404]: pgmap v1993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2740.627393 secs 2024-07-26T09:26:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:36 smithi116 ceph-mon[37097]: pgmap v1993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:26:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2740.627393 secs 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: pgmap v1994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: pgmap v1995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: pgmap v1994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: pgmap v1995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:38.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:38.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:38.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: pgmap v1994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:26:38.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: pgmap v1995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:38.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:26:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:39 smithi044 ceph-mon[32404]: pgmap v1996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:39 smithi116 ceph-mon[37097]: pgmap v1996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:39 smithi171 ceph-mon[37789]: pgmap v1996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:26:41.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:41.028+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:26:41.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:41.028+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:41.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:41.028+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:26:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2745.627536 secs 2024-07-26T09:26:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:41 smithi116 ceph-mon[37097]: pgmap v1997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2745.627536 secs 2024-07-26T09:26:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:41 smithi171 ceph-mon[37789]: pgmap v1997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2745.627536 secs 2024-07-26T09:26:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:41 smithi044 ceph-mon[32404]: pgmap v1997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:43 smithi116 ceph-mon[37097]: pgmap v1998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:26:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:43 smithi171 ceph-mon[37789]: pgmap v1998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:26:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:43 smithi044 ceph-mon[32404]: pgmap v1998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:26:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:45 smithi171 ceph-mon[37789]: pgmap v1999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:45 smithi044 ceph-mon[32404]: pgmap v1999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:45 smithi116 ceph-mon[37097]: pgmap v1999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:26:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2750.627673 secs 2024-07-26T09:26:47.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2750.627673 secs 2024-07-26T09:26:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2750.627673 secs 2024-07-26T09:26:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:47 smithi171 ceph-mon[37789]: pgmap v2000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:26:48.203 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:47 smithi044 ceph-mon[32404]: pgmap v2000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:26:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:47 smithi116 ceph-mon[37097]: pgmap v2000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:26:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:49 smithi171 ceph-mon[37789]: pgmap v2001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:26:50.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:49 smithi044 ceph-mon[32404]: pgmap v2001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:26:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:49 smithi116 ceph-mon[37097]: pgmap v2001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:26:52.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2755.627809 secs 2024-07-26T09:26:52.070 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:51 smithi044 ceph-mon[32404]: pgmap v2002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:26:52.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2755.627809 secs 2024-07-26T09:26:52.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:51 smithi116 ceph-mon[37097]: pgmap v2002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2755.627809 secs 2024-07-26T09:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:51 smithi171 ceph-mon[37789]: pgmap v2002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:26:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:53 smithi171 ceph-mon[37789]: pgmap v2003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:26:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:53 smithi044 ceph-mon[32404]: pgmap v2003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:26:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:53 smithi116 ceph-mon[37097]: pgmap v2003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:26:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:56.035+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:26:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:56.035+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:56.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:26:56.035+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:26:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:55 smithi171 ceph-mon[37789]: pgmap v2004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:26:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:55 smithi044 ceph-mon[32404]: pgmap v2004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:26:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:55 smithi116 ceph-mon[37097]: pgmap v2004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:26:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2760.627926 secs 2024-07-26T09:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2760.627926 secs 2024-07-26T09:26:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2760.627926 secs 2024-07-26T09:26:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:57 smithi171 ceph-mon[37789]: pgmap v2005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:57 smithi044 ceph-mon[32404]: pgmap v2005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:26:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:57 smithi116 ceph-mon[37097]: pgmap v2005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:26:59 smithi171 ceph-mon[37789]: pgmap v2006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:27:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:26:59 smithi044 ceph-mon[32404]: pgmap v2006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:27:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:26:59 smithi116 ceph-mon[37097]: pgmap v2006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:27:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2765.628095 secs 2024-07-26T09:27:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2765.628095 secs 2024-07-26T09:27:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2765.628095 secs 2024-07-26T09:27:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:01 smithi171 ceph-mon[37789]: pgmap v2007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:01 smithi044 ceph-mon[32404]: pgmap v2007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:01 smithi116 ceph-mon[37097]: pgmap v2007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:03 smithi044 ceph-mon[32404]: pgmap v2008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:03 smithi116 ceph-mon[37097]: pgmap v2008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:04.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:03 smithi171 ceph-mon[37789]: pgmap v2008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:05 smithi044 ceph-mon[32404]: pgmap v2009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2770.628251 secs 2024-07-26T09:27:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:05 smithi116 ceph-mon[37097]: pgmap v2009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2770.628251 secs 2024-07-26T09:27:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:05 smithi171 ceph-mon[37789]: pgmap v2009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2770.628251 secs 2024-07-26T09:27:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:07 smithi044 ceph-mon[32404]: pgmap v2010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:07 smithi116 ceph-mon[37097]: pgmap v2010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:07 smithi171 ceph-mon[37789]: pgmap v2010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:09 smithi044 ceph-mon[32404]: pgmap v2011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:09 smithi116 ceph-mon[37097]: pgmap v2011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:09 smithi171 ceph-mon[37789]: pgmap v2011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:11.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:11.039+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:27:11.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:11.039+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:11.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:11.039+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:27:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2775.628385 secs 2024-07-26T09:27:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2775.628385 secs 2024-07-26T09:27:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2775.628385 secs 2024-07-26T09:27:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:11 smithi044 ceph-mon[32404]: pgmap v2012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:11 smithi116 ceph-mon[37097]: pgmap v2012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:11 smithi171 ceph-mon[37789]: pgmap v2012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:13 smithi044 ceph-mon[32404]: pgmap v2013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:13 smithi116 ceph-mon[37097]: pgmap v2013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:13 smithi171 ceph-mon[37789]: pgmap v2013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:15 smithi044 ceph-mon[32404]: pgmap v2014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2780.628523 secs 2024-07-26T09:27:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:15 smithi116 ceph-mon[37097]: pgmap v2014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2780.628523 secs 2024-07-26T09:27:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:15 smithi171 ceph-mon[37789]: pgmap v2014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2780.628523 secs 2024-07-26T09:27:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:17 smithi044 ceph-mon[32404]: pgmap v2015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:17 smithi116 ceph-mon[37097]: pgmap v2015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:17 smithi171 ceph-mon[37789]: pgmap v2015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:19 smithi044 ceph-mon[32404]: pgmap v2016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:19 smithi116 ceph-mon[37097]: pgmap v2016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:19 smithi171 ceph-mon[37789]: pgmap v2016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2785.628686 secs 2024-07-26T09:27:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2785.628686 secs 2024-07-26T09:27:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2785.628686 secs 2024-07-26T09:27:22.170 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:21 smithi044 ceph-mon[32404]: pgmap v2017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:22.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:21 smithi116 ceph-mon[37097]: pgmap v2017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:21 smithi171 ceph-mon[37789]: pgmap v2017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:23 smithi044 ceph-mon[32404]: pgmap v2018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:23 smithi116 ceph-mon[37097]: pgmap v2018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:23 smithi171 ceph-mon[37789]: pgmap v2018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:26.047+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:27:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:26.047+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:27:26.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:26.047+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:25 smithi044 ceph-mon[32404]: pgmap v2019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2790.628852 secs 2024-07-26T09:27:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:25 smithi116 ceph-mon[37097]: pgmap v2019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2790.628852 secs 2024-07-26T09:27:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:25 smithi171 ceph-mon[37789]: pgmap v2019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2790.628852 secs 2024-07-26T09:27:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:27 smithi044 ceph-mon[32404]: pgmap v2020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:27 smithi116 ceph-mon[37097]: pgmap v2020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:27 smithi171 ceph-mon[37789]: pgmap v2020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:29 smithi044 ceph-mon[32404]: pgmap v2021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:29 smithi116 ceph-mon[37097]: pgmap v2021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:29 smithi171 ceph-mon[37789]: pgmap v2021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:27:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2795.628997 secs 2024-07-26T09:27:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2795.628997 secs 2024-07-26T09:27:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2795.628997 secs 2024-07-26T09:27:32.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:31 smithi171 ceph-mon[37789]: pgmap v2022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:31 smithi044 ceph-mon[32404]: pgmap v2022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:31 smithi116 ceph-mon[37097]: pgmap v2022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:33 smithi044 ceph-mon[32404]: pgmap v2023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:27:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:33 smithi116 ceph-mon[37097]: pgmap v2023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:27:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:33 smithi171 ceph-mon[37789]: pgmap v2023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:27:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:35 smithi044 ceph-mon[32404]: pgmap v2024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2800.629300 secs 2024-07-26T09:27:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:35 smithi116 ceph-mon[37097]: pgmap v2024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:36.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2800.629300 secs 2024-07-26T09:27:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:35 smithi171 ceph-mon[37789]: pgmap v2024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:27:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2800.629300 secs 2024-07-26T09:27:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:37 smithi044 ceph-mon[32404]: pgmap v2025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:37 smithi116 ceph-mon[37097]: pgmap v2025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:37 smithi171 ceph-mon[37789]: pgmap v2025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:39 smithi044 ceph-mon[32404]: pgmap v2026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:27:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:39 smithi044 ceph-mon[32404]: pgmap v2027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:27:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:27:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:39 smithi116 ceph-mon[37097]: pgmap v2026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:27:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:39 smithi116 ceph-mon[37097]: pgmap v2027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:27:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:27:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:39 smithi171 ceph-mon[37789]: pgmap v2026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:27:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:39 smithi171 ceph-mon[37789]: pgmap v2027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:27:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:27:41.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:41.051+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:27:41.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:41.051+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:27:41.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:41.051+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:41 smithi044 ceph-mon[32404]: pgmap v2028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2805.629445 secs 2024-07-26T09:27:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:41 smithi116 ceph-mon[37097]: pgmap v2028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2805.629445 secs 2024-07-26T09:27:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:41 smithi171 ceph-mon[37789]: pgmap v2028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2805.629445 secs 2024-07-26T09:27:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:43 smithi116 ceph-mon[37097]: pgmap v2029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:43 smithi171 ceph-mon[37789]: pgmap v2029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:27:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:43 smithi044 ceph-mon[32404]: pgmap v2029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:27:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:45 smithi116 ceph-mon[37097]: pgmap v2030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:27:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:45 smithi171 ceph-mon[37789]: pgmap v2030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:27:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:45 smithi044 ceph-mon[32404]: pgmap v2030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:27:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2810.629606 secs 2024-07-26T09:27:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2810.629606 secs 2024-07-26T09:27:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2810.629606 secs 2024-07-26T09:27:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:47 smithi171 ceph-mon[37789]: pgmap v2031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:27:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:47 smithi044 ceph-mon[32404]: pgmap v2031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:27:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:47 smithi116 ceph-mon[37097]: pgmap v2031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:27:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:49 smithi171 ceph-mon[37789]: pgmap v2032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:27:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:49 smithi044 ceph-mon[32404]: pgmap v2032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:27:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:49 smithi116 ceph-mon[37097]: pgmap v2032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:27:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:51 smithi171 ceph-mon[37789]: pgmap v2033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2815.629741 secs 2024-07-26T09:27:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:51 smithi044 ceph-mon[32404]: pgmap v2033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2815.629741 secs 2024-07-26T09:27:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:51 smithi116 ceph-mon[37097]: pgmap v2033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:27:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2815.629741 secs 2024-07-26T09:27:53.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:53 smithi044 ceph-mon[32404]: pgmap v2034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:53.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:53 smithi116 ceph-mon[37097]: pgmap v2034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:53 smithi171 ceph-mon[37789]: pgmap v2034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:55 smithi171 ceph-mon[37789]: pgmap v2035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:55 smithi044 ceph-mon[32404]: pgmap v2035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:55 smithi116 ceph-mon[37097]: pgmap v2035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:56.056+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:27:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:56.056+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:27:56.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:27:56.056+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2820.629905 secs 2024-07-26T09:27:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2820.629905 secs 2024-07-26T09:27:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2820.629905 secs 2024-07-26T09:27:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:57 smithi171 ceph-mon[37789]: pgmap v2036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:57 smithi044 ceph-mon[32404]: pgmap v2036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:57 smithi116 ceph-mon[37097]: pgmap v2036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:27:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:27:59 smithi171 ceph-mon[37789]: pgmap v2037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:27:59 smithi044 ceph-mon[32404]: pgmap v2037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:27:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:27:59 smithi116 ceph-mon[37097]: pgmap v2037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:01 smithi044 ceph-mon[32404]: pgmap v2038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2825.630024 secs 2024-07-26T09:28:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:01 smithi116 ceph-mon[37097]: pgmap v2038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2825.630024 secs 2024-07-26T09:28:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:01 smithi171 ceph-mon[37789]: pgmap v2038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2825.630024 secs 2024-07-26T09:28:03.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:03 smithi171 ceph-mon[37789]: pgmap v2039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:03 smithi044 ceph-mon[32404]: pgmap v2039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:03 smithi116 ceph-mon[37097]: pgmap v2039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:05 smithi044 ceph-mon[32404]: pgmap v2040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:05 smithi116 ceph-mon[37097]: pgmap v2040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:05 smithi171 ceph-mon[37789]: pgmap v2040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2830.630152 secs 2024-07-26T09:28:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2830.630152 secs 2024-07-26T09:28:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2830.630152 secs 2024-07-26T09:28:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:07 smithi044 ceph-mon[32404]: pgmap v2041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:07 smithi116 ceph-mon[37097]: pgmap v2041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:07 smithi171 ceph-mon[37789]: pgmap v2041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:09 smithi044 ceph-mon[32404]: pgmap v2042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:09 smithi116 ceph-mon[37097]: pgmap v2042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:09 smithi171 ceph-mon[37789]: pgmap v2042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:11.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:11.059+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:28:11.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:11.060+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:28:11.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:11.060+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:11 smithi044 ceph-mon[32404]: pgmap v2043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2835.630254 secs 2024-07-26T09:28:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:11 smithi116 ceph-mon[37097]: pgmap v2043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2835.630254 secs 2024-07-26T09:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:11 smithi171 ceph-mon[37789]: pgmap v2043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2835.630254 secs 2024-07-26T09:28:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:13 smithi044 ceph-mon[32404]: pgmap v2044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:13 smithi116 ceph-mon[37097]: pgmap v2044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:13 smithi171 ceph-mon[37789]: pgmap v2044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:15 smithi044 ceph-mon[32404]: pgmap v2045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:15 smithi116 ceph-mon[37097]: pgmap v2045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:15 smithi171 ceph-mon[37789]: pgmap v2045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2840.630372 secs 2024-07-26T09:28:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2840.630372 secs 2024-07-26T09:28:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2840.630372 secs 2024-07-26T09:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:17 smithi044 ceph-mon[32404]: pgmap v2046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:17 smithi116 ceph-mon[37097]: pgmap v2046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:17 smithi171 ceph-mon[37789]: pgmap v2046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:19 smithi044 ceph-mon[32404]: pgmap v2047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:19 smithi116 ceph-mon[37097]: pgmap v2047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:19 smithi171 ceph-mon[37789]: pgmap v2047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:21 smithi044 ceph-mon[32404]: pgmap v2048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2845.630534 secs 2024-07-26T09:28:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:21 smithi116 ceph-mon[37097]: pgmap v2048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2845.630534 secs 2024-07-26T09:28:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:21 smithi171 ceph-mon[37789]: pgmap v2048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2845.630534 secs 2024-07-26T09:28:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:23 smithi044 ceph-mon[32404]: pgmap v2049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:23 smithi116 ceph-mon[37097]: pgmap v2049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:23 smithi171 ceph-mon[37789]: pgmap v2049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:25 smithi044 ceph-mon[32404]: pgmap v2050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:25 smithi116 ceph-mon[37097]: pgmap v2050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:25 smithi171 ceph-mon[37789]: pgmap v2050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:26.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:26.065+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:28:26.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:26.066+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:28:26.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:26.066+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2850.630706 secs 2024-07-26T09:28:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2850.630706 secs 2024-07-26T09:28:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2850.630706 secs 2024-07-26T09:28:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:27 smithi044 ceph-mon[32404]: pgmap v2051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:27 smithi116 ceph-mon[37097]: pgmap v2051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:27.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:27 smithi171 ceph-mon[37789]: pgmap v2051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:29 smithi044 ceph-mon[32404]: pgmap v2052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:29 smithi116 ceph-mon[37097]: pgmap v2052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:29 smithi171 ceph-mon[37789]: pgmap v2052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:31 smithi044 ceph-mon[32404]: pgmap v2053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2855.630851 secs 2024-07-26T09:28:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:31 smithi116 ceph-mon[37097]: pgmap v2053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2855.630851 secs 2024-07-26T09:28:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:31 smithi171 ceph-mon[37789]: pgmap v2053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2855.630851 secs 2024-07-26T09:28:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:33 smithi044 ceph-mon[32404]: pgmap v2054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:33 smithi116 ceph-mon[37097]: pgmap v2054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:33 smithi171 ceph-mon[37789]: pgmap v2054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:35 smithi044 ceph-mon[32404]: pgmap v2055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:35 smithi116 ceph-mon[37097]: pgmap v2055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:35 smithi171 ceph-mon[37789]: pgmap v2055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:28:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2860.630998 secs 2024-07-26T09:28:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2860.630998 secs 2024-07-26T09:28:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2860.630998 secs 2024-07-26T09:28:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:37 smithi044 ceph-mon[32404]: pgmap v2056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:37 smithi116 ceph-mon[37097]: pgmap v2056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:37 smithi171 ceph-mon[37789]: pgmap v2056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:39 smithi044 ceph-mon[32404]: pgmap v2057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:28:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:39 smithi116 ceph-mon[37097]: pgmap v2057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:28:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:39 smithi171 ceph-mon[37789]: pgmap v2057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:28:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:40.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:40.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:28:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:28:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:28:41.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:41.070+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:28:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:41.070+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:28:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:41.070+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:41 smithi044 ceph-mon[32404]: pgmap v2058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:28:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:41 smithi044 ceph-mon[32404]: pgmap v2059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2865.631166 secs 2024-07-26T09:28:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:41 smithi116 ceph-mon[37097]: pgmap v2058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:28:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:41 smithi116 ceph-mon[37097]: pgmap v2059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2865.631166 secs 2024-07-26T09:28:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:41 smithi171 ceph-mon[37789]: pgmap v2058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:28:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:41 smithi171 ceph-mon[37789]: pgmap v2059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2865.631166 secs 2024-07-26T09:28:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:42 smithi171 ceph-mon[37789]: pgmap v2060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:28:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:42 smithi044 ceph-mon[32404]: pgmap v2060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:28:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:42 smithi116 ceph-mon[37097]: pgmap v2060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:28:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:44 smithi044 ceph-mon[32404]: pgmap v2061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:28:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:44 smithi116 ceph-mon[37097]: pgmap v2061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:28:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:44 smithi171 ceph-mon[37789]: pgmap v2061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:28:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2870.631308 secs 2024-07-26T09:28:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:46 smithi044 ceph-mon[32404]: pgmap v2062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2870.631308 secs 2024-07-26T09:28:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:46 smithi116 ceph-mon[37097]: pgmap v2062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2870.631308 secs 2024-07-26T09:28:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:46 smithi171 ceph-mon[37789]: pgmap v2062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:28:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:48 smithi044 ceph-mon[32404]: pgmap v2063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:48 smithi116 ceph-mon[37097]: pgmap v2063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:48 smithi171 ceph-mon[37789]: pgmap v2063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:28:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:50 smithi044 ceph-mon[32404]: pgmap v2064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:28:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:50 smithi116 ceph-mon[37097]: pgmap v2064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:28:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:50 smithi171 ceph-mon[37789]: pgmap v2064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:28:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2875.631458 secs 2024-07-26T09:28:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2875.631458 secs 2024-07-26T09:28:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2875.631458 secs 2024-07-26T09:28:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:52 smithi044 ceph-mon[32404]: pgmap v2065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:52 smithi116 ceph-mon[37097]: pgmap v2065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:52 smithi171 ceph-mon[37789]: pgmap v2065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:54 smithi044 ceph-mon[32404]: pgmap v2066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:28:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:54 smithi116 ceph-mon[37097]: pgmap v2066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:28:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:54 smithi171 ceph-mon[37789]: pgmap v2066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:28:56.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:56.078+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:28:56.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:56.078+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:28:56.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:28:56.078+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:56 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 2880.631652 secs 2024-07-26T09:28:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:56 smithi044 ceph-mon[32404]: slow request 2880.631652 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T09:28:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:56 smithi044 ceph-mon[32404]: pgmap v2067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:56 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 2880.631652 secs 2024-07-26T09:28:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:56 smithi116 ceph-mon[37097]: slow request 2880.631652 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T09:28:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:56 smithi116 ceph-mon[37097]: pgmap v2067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:56 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 2880.631652 secs 2024-07-26T09:28:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:56 smithi171 ceph-mon[37789]: slow request 2880.631652 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T09:28:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:56 smithi171 ceph-mon[37789]: pgmap v2067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:28:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:28:58 smithi044 ceph-mon[32404]: pgmap v2068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:28:58 smithi116 ceph-mon[37097]: pgmap v2068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:28:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:28:58 smithi171 ceph-mon[37789]: pgmap v2068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:00 smithi044 ceph-mon[32404]: pgmap v2069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:00 smithi116 ceph-mon[37097]: pgmap v2069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:00 smithi171 ceph-mon[37789]: pgmap v2069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2885.631811 secs 2024-07-26T09:29:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2885.631811 secs 2024-07-26T09:29:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2885.631811 secs 2024-07-26T09:29:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:02 smithi044 ceph-mon[32404]: pgmap v2070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:02 smithi116 ceph-mon[37097]: pgmap v2070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:02 smithi171 ceph-mon[37789]: pgmap v2070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:04 smithi044 ceph-mon[32404]: pgmap v2071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:04 smithi116 ceph-mon[37097]: pgmap v2071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:04 smithi171 ceph-mon[37789]: pgmap v2071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2890.631958 secs 2024-07-26T09:29:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:06 smithi044 ceph-mon[32404]: pgmap v2072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2890.631958 secs 2024-07-26T09:29:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:06 smithi116 ceph-mon[37097]: pgmap v2072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2890.631958 secs 2024-07-26T09:29:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:06 smithi171 ceph-mon[37789]: pgmap v2072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:08 smithi044 ceph-mon[32404]: pgmap v2073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:08 smithi116 ceph-mon[37097]: pgmap v2073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:08 smithi171 ceph-mon[37789]: pgmap v2073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:10 smithi116 ceph-mon[37097]: pgmap v2074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:10 smithi171 ceph-mon[37789]: pgmap v2074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:10 smithi044 ceph-mon[32404]: pgmap v2074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:11.082+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:29:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:11.082+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:29:11.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:11.082+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2895.632101 secs 2024-07-26T09:29:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2895.632101 secs 2024-07-26T09:29:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2895.632101 secs 2024-07-26T09:29:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:12 smithi171 ceph-mon[37789]: pgmap v2075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:12 smithi044 ceph-mon[32404]: pgmap v2075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:12 smithi116 ceph-mon[37097]: pgmap v2075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:14 smithi171 ceph-mon[37789]: pgmap v2076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:14 smithi044 ceph-mon[32404]: pgmap v2076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:14 smithi116 ceph-mon[37097]: pgmap v2076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:15.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:15.173+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:29:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:15.173+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:29:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:15.174+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:29:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:15.175+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:29:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2900.632264 secs 2024-07-26T09:29:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:16 smithi171 ceph-mon[37789]: pgmap v2077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2900.632264 secs 2024-07-26T09:29:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:16 smithi044 ceph-mon[32404]: pgmap v2077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2900.632264 secs 2024-07-26T09:29:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:16 smithi116 ceph-mon[37097]: pgmap v2077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:18 smithi171 ceph-mon[37789]: pgmap v2078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:18 smithi044 ceph-mon[32404]: pgmap v2078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:18 smithi116 ceph-mon[37097]: pgmap v2078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:20 smithi171 ceph-mon[37789]: pgmap v2079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:20 smithi044 ceph-mon[32404]: pgmap v2079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:20 smithi116 ceph-mon[37097]: pgmap v2079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2905.632425 secs 2024-07-26T09:29:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2905.632425 secs 2024-07-26T09:29:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2905.632425 secs 2024-07-26T09:29:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:22 smithi171 ceph-mon[37789]: pgmap v2080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:22.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:22 smithi044 ceph-mon[32404]: pgmap v2080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:22 smithi116 ceph-mon[37097]: pgmap v2080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:24 smithi171 ceph-mon[37789]: pgmap v2081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:24 smithi044 ceph-mon[32404]: pgmap v2081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:24 smithi116 ceph-mon[37097]: pgmap v2081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:26.086+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:29:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:26.086+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:26.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:26.086+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:29:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2910.632570 secs 2024-07-26T09:29:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:26 smithi171 ceph-mon[37789]: pgmap v2082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2910.632570 secs 2024-07-26T09:29:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:26 smithi044 ceph-mon[32404]: pgmap v2082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2910.632570 secs 2024-07-26T09:29:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:26 smithi116 ceph-mon[37097]: pgmap v2082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:28 smithi044 ceph-mon[32404]: pgmap v2083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:29.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:28 smithi116 ceph-mon[37097]: pgmap v2083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:28 smithi171 ceph-mon[37789]: pgmap v2083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:30 smithi044 ceph-mon[32404]: pgmap v2084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:30 smithi116 ceph-mon[37097]: pgmap v2084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:30 smithi171 ceph-mon[37789]: pgmap v2084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2915.632720 secs 2024-07-26T09:29:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2915.632720 secs 2024-07-26T09:29:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2915.632720 secs 2024-07-26T09:29:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:32 smithi044 ceph-mon[32404]: pgmap v2085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:32 smithi116 ceph-mon[37097]: pgmap v2085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:32 smithi171 ceph-mon[37789]: pgmap v2085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:34.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:34 smithi171 ceph-mon[37789]: pgmap v2086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:34 smithi044 ceph-mon[32404]: pgmap v2086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:34 smithi116 ceph-mon[37097]: pgmap v2086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:29:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2920.632867 secs 2024-07-26T09:29:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:36 smithi044 ceph-mon[32404]: pgmap v2087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2920.632867 secs 2024-07-26T09:29:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:36 smithi116 ceph-mon[37097]: pgmap v2087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2920.632867 secs 2024-07-26T09:29:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:36 smithi171 ceph-mon[37789]: pgmap v2087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:38 smithi044 ceph-mon[32404]: pgmap v2088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:38 smithi116 ceph-mon[37097]: pgmap v2088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:38 smithi171 ceph-mon[37789]: pgmap v2088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:40 smithi044 ceph-mon[32404]: pgmap v2089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:29:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:40 smithi116 ceph-mon[37097]: pgmap v2089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:29:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:40 smithi171 ceph-mon[37789]: pgmap v2089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:29:41.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:41.090+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:29:41.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:41.090+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:29:41.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:41.090+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2925.632997 secs 2024-07-26T09:29:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:41 smithi044 ceph-mon[32404]: pgmap v2090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:29:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:41 smithi044 ceph-mon[32404]: pgmap v2091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2925.632997 secs 2024-07-26T09:29:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:41 smithi116 ceph-mon[37097]: pgmap v2090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:29:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:41 smithi116 ceph-mon[37097]: pgmap v2091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2925.632997 secs 2024-07-26T09:29:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:41 smithi171 ceph-mon[37789]: pgmap v2090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:29:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:41 smithi171 ceph-mon[37789]: pgmap v2091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:43 smithi044 ceph-mon[32404]: pgmap v2092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:43 smithi116 ceph-mon[37097]: pgmap v2092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:43 smithi171 ceph-mon[37789]: pgmap v2092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:29:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:45 smithi044 ceph-mon[32404]: pgmap v2093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:45 smithi116 ceph-mon[37097]: pgmap v2093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:45 smithi171 ceph-mon[37789]: pgmap v2093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2930.633110 secs 2024-07-26T09:29:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2930.633110 secs 2024-07-26T09:29:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2930.633110 secs 2024-07-26T09:29:48.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:47 smithi044 ceph-mon[32404]: pgmap v2094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:29:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:47 smithi116 ceph-mon[37097]: pgmap v2094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:29:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:47 smithi171 ceph-mon[37789]: pgmap v2094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:29:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:49 smithi044 ceph-mon[32404]: pgmap v2095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:49 smithi116 ceph-mon[37097]: pgmap v2095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:49 smithi171 ceph-mon[37789]: pgmap v2095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:29:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2935.633257 secs 2024-07-26T09:29:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:51 smithi044 ceph-mon[32404]: pgmap v2096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:29:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2935.633257 secs 2024-07-26T09:29:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:51 smithi116 ceph-mon[37097]: pgmap v2096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:29:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2935.633257 secs 2024-07-26T09:29:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:51 smithi171 ceph-mon[37789]: pgmap v2096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:29:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:53 smithi044 ceph-mon[32404]: pgmap v2097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:53 smithi116 ceph-mon[37097]: pgmap v2097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:53 smithi171 ceph-mon[37789]: pgmap v2097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:29:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:55 smithi044 ceph-mon[32404]: pgmap v2098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:55 smithi116 ceph-mon[37097]: pgmap v2098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:55 smithi171 ceph-mon[37789]: pgmap v2098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:29:56.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:56.099+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:29:56.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:56.099+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:29:56.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:29:56.099+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2940.633415 secs 2024-07-26T09:29:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2940.633415 secs 2024-07-26T09:29:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2940.633415 secs 2024-07-26T09:29:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:57 smithi044 ceph-mon[32404]: pgmap v2099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:57 smithi116 ceph-mon[37097]: pgmap v2099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:29:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:57 smithi171 ceph-mon[37789]: pgmap v2099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:29:59 smithi044 ceph-mon[32404]: pgmap v2100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:29:59 smithi116 ceph-mon[37097]: pgmap v2100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:29:59 smithi171 ceph-mon[37789]: pgmap v2100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:30:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:30:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:30:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2945.633562 secs 2024-07-26T09:30:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:01 smithi044 ceph-mon[32404]: pgmap v2101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2945.633562 secs 2024-07-26T09:30:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:01 smithi116 ceph-mon[37097]: pgmap v2101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2945.633562 secs 2024-07-26T09:30:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:01 smithi171 ceph-mon[37789]: pgmap v2101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:03 smithi044 ceph-mon[32404]: pgmap v2102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:03 smithi116 ceph-mon[37097]: pgmap v2102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:03 smithi171 ceph-mon[37789]: pgmap v2102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:05 smithi044 ceph-mon[32404]: pgmap v2103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:05 smithi116 ceph-mon[37097]: pgmap v2103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:05 smithi171 ceph-mon[37789]: pgmap v2103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2950.633724 secs 2024-07-26T09:30:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2950.633724 secs 2024-07-26T09:30:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2950.633724 secs 2024-07-26T09:30:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:07 smithi044 ceph-mon[32404]: pgmap v2104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:07 smithi116 ceph-mon[37097]: pgmap v2104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:07 smithi171 ceph-mon[37789]: pgmap v2104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:09 smithi044 ceph-mon[32404]: pgmap v2105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:09 smithi116 ceph-mon[37097]: pgmap v2105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:09 smithi171 ceph-mon[37789]: pgmap v2105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:11.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:11.102+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:30:11.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:11.102+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:30:11.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:11.102+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2955.633882 secs 2024-07-26T09:30:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:11 smithi044 ceph-mon[32404]: pgmap v2106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2955.633882 secs 2024-07-26T09:30:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:11 smithi116 ceph-mon[37097]: pgmap v2106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2955.633882 secs 2024-07-26T09:30:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:11 smithi171 ceph-mon[37789]: pgmap v2106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:13 smithi116 ceph-mon[37097]: pgmap v2107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:13 smithi171 ceph-mon[37789]: pgmap v2107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:13 smithi044 ceph-mon[32404]: pgmap v2107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:15 smithi171 ceph-mon[37789]: pgmap v2108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:15 smithi044 ceph-mon[32404]: pgmap v2108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:15 smithi116 ceph-mon[37097]: pgmap v2108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2960.633979 secs 2024-07-26T09:30:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2960.633979 secs 2024-07-26T09:30:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2960.633979 secs 2024-07-26T09:30:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:17 smithi171 ceph-mon[37789]: pgmap v2109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:17 smithi044 ceph-mon[32404]: pgmap v2109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:17 smithi116 ceph-mon[37097]: pgmap v2109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:19 smithi171 ceph-mon[37789]: pgmap v2110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:19 smithi044 ceph-mon[32404]: pgmap v2110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:19 smithi116 ceph-mon[37097]: pgmap v2110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2965.634113 secs 2024-07-26T09:30:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:21 smithi171 ceph-mon[37789]: pgmap v2111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2965.634113 secs 2024-07-26T09:30:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:21 smithi044 ceph-mon[32404]: pgmap v2111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2965.634113 secs 2024-07-26T09:30:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:21 smithi116 ceph-mon[37097]: pgmap v2111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:23 smithi171 ceph-mon[37789]: pgmap v2112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:23 smithi044 ceph-mon[32404]: pgmap v2112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:23 smithi116 ceph-mon[37097]: pgmap v2112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:26.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:25 smithi116 ceph-mon[37097]: pgmap v2113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:30:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:25 smithi171 ceph-mon[37789]: pgmap v2113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:30:26.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:26.103+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:30:26.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:26.104+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:30:26.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:26.104+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:26.107 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:25 smithi044 ceph-mon[32404]: pgmap v2113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:30:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2970.634278 secs 2024-07-26T09:30:27.091 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2970.634278 secs 2024-07-26T09:30:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2970.634278 secs 2024-07-26T09:30:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:27 smithi171 ceph-mon[37789]: pgmap v2114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:27 smithi044 ceph-mon[32404]: pgmap v2114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:27 smithi116 ceph-mon[37097]: pgmap v2114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:29 smithi171 ceph-mon[37789]: pgmap v2115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:30:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:29 smithi044 ceph-mon[32404]: pgmap v2115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:30:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:29 smithi116 ceph-mon[37097]: pgmap v2115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:30:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2975.634445 secs 2024-07-26T09:30:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:31 smithi044 ceph-mon[32404]: pgmap v2116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2975.634445 secs 2024-07-26T09:30:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:31 smithi116 ceph-mon[37097]: pgmap v2116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2975.634445 secs 2024-07-26T09:30:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:31 smithi171 ceph-mon[37789]: pgmap v2116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:33 smithi044 ceph-mon[32404]: pgmap v2117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:33 smithi116 ceph-mon[37097]: pgmap v2117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:33 smithi171 ceph-mon[37789]: pgmap v2117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:30:36.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:35 smithi171 ceph-mon[37789]: pgmap v2118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:35 smithi044 ceph-mon[32404]: pgmap v2118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:35 smithi116 ceph-mon[37097]: pgmap v2118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:30:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2980.634608 secs 2024-07-26T09:30:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2980.634608 secs 2024-07-26T09:30:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2980.634608 secs 2024-07-26T09:30:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:37 smithi044 ceph-mon[32404]: pgmap v2119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:37 smithi116 ceph-mon[37097]: pgmap v2119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:37 smithi171 ceph-mon[37789]: pgmap v2119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:30:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:39 smithi044 ceph-mon[32404]: pgmap v2120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:39 smithi116 ceph-mon[37097]: pgmap v2120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:39 smithi171 ceph-mon[37789]: pgmap v2120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:30:41.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:41.107+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:30:41.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:41.108+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:30:41.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:41.108+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2985.634756 secs 2024-07-26T09:30:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2985.634756 secs 2024-07-26T09:30:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2985.634756 secs 2024-07-26T09:30:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:41 smithi044 ceph-mon[32404]: pgmap v2121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:30:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:30:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:41 smithi116 ceph-mon[37097]: pgmap v2121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:30:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:30:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:41 smithi171 ceph-mon[37789]: pgmap v2121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:30:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:30:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:42 smithi044 ceph-mon[32404]: pgmap v2122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:30:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:42 smithi044 ceph-mon[32404]: pgmap v2123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:42 smithi116 ceph-mon[37097]: pgmap v2122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:30:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:42 smithi116 ceph-mon[37097]: pgmap v2123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:42 smithi171 ceph-mon[37789]: pgmap v2122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:30:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:42 smithi171 ceph-mon[37789]: pgmap v2123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:45.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:45.190+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:30:45.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:45.190+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:30:45.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:44 smithi044 ceph-mon[32404]: pgmap v2124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:45.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:45.191+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:30:45.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:45.191+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:30:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:44 smithi116 ceph-mon[37097]: pgmap v2124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:44 smithi171 ceph-mon[37789]: pgmap v2124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2990.634901 secs 2024-07-26T09:30:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2990.634901 secs 2024-07-26T09:30:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2990.634901 secs 2024-07-26T09:30:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:46 smithi044 ceph-mon[32404]: pgmap v2125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:30:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:46 smithi116 ceph-mon[37097]: pgmap v2125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:30:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:46 smithi171 ceph-mon[37789]: pgmap v2125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:30:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:48 smithi044 ceph-mon[32404]: pgmap v2126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:48 smithi116 ceph-mon[37097]: pgmap v2126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:48 smithi171 ceph-mon[37789]: pgmap v2126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:30:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:50 smithi044 ceph-mon[32404]: pgmap v2127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 2995.635044 secs 2024-07-26T09:30:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:50 smithi116 ceph-mon[37097]: pgmap v2127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 2995.635044 secs 2024-07-26T09:30:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:50 smithi171 ceph-mon[37789]: pgmap v2127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:30:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 2995.635044 secs 2024-07-26T09:30:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:52 smithi044 ceph-mon[32404]: pgmap v2128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:30:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:52 smithi116 ceph-mon[37097]: pgmap v2128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:30:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:52 smithi171 ceph-mon[37789]: pgmap v2128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:30:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:54 smithi044 ceph-mon[32404]: pgmap v2129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:54 smithi116 ceph-mon[37097]: pgmap v2129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:54 smithi171 ceph-mon[37789]: pgmap v2129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:56.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:56.117+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:30:56.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:56.117+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:56.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:30:56.117+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:30:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3000.635180 secs 2024-07-26T09:30:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3000.635180 secs 2024-07-26T09:30:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3000.635180 secs 2024-07-26T09:30:57.172 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:56 smithi044 ceph-mon[32404]: pgmap v2130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:56 smithi116 ceph-mon[37097]: pgmap v2130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:56 smithi171 ceph-mon[37789]: pgmap v2130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:30:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:30:58 smithi044 ceph-mon[32404]: pgmap v2131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:30:58 smithi116 ceph-mon[37097]: pgmap v2131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:30:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:30:58 smithi171 ceph-mon[37789]: pgmap v2131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:00 smithi044 ceph-mon[32404]: pgmap v2132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3005.635335 secs 2024-07-26T09:31:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:00 smithi116 ceph-mon[37097]: pgmap v2132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3005.635335 secs 2024-07-26T09:31:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:00 smithi171 ceph-mon[37789]: pgmap v2132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:01.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3005.635335 secs 2024-07-26T09:31:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:02 smithi044 ceph-mon[32404]: pgmap v2133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:02 smithi116 ceph-mon[37097]: pgmap v2133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:02 smithi171 ceph-mon[37789]: pgmap v2133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:04 smithi044 ceph-mon[32404]: pgmap v2134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:04 smithi116 ceph-mon[37097]: pgmap v2134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:04 smithi171 ceph-mon[37789]: pgmap v2134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:06.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3010.635499 secs 2024-07-26T09:31:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3010.635499 secs 2024-07-26T09:31:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3010.635499 secs 2024-07-26T09:31:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:06 smithi044 ceph-mon[32404]: pgmap v2135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:06 smithi116 ceph-mon[37097]: pgmap v2135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:06 smithi171 ceph-mon[37789]: pgmap v2135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:08 smithi044 ceph-mon[32404]: pgmap v2136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:08 smithi116 ceph-mon[37097]: pgmap v2136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:08 smithi171 ceph-mon[37789]: pgmap v2136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:11.119+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:31:11.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:11.119+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:11.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:11.119+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:31:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:10 smithi044 ceph-mon[32404]: pgmap v2137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3015.635660 secs 2024-07-26T09:31:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:10 smithi116 ceph-mon[37097]: pgmap v2137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3015.635660 secs 2024-07-26T09:31:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:10 smithi171 ceph-mon[37789]: pgmap v2137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3015.635660 secs 2024-07-26T09:31:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:12 smithi044 ceph-mon[32404]: pgmap v2138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:12 smithi116 ceph-mon[37097]: pgmap v2138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:12 smithi171 ceph-mon[37789]: pgmap v2138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:15 smithi044 ceph-mon[32404]: pgmap v2139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:15.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:15 smithi116 ceph-mon[37097]: pgmap v2139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:15 smithi171 ceph-mon[37789]: pgmap v2139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3020.635824 secs 2024-07-26T09:31:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3020.635824 secs 2024-07-26T09:31:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3020.635824 secs 2024-07-26T09:31:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:17 smithi116 ceph-mon[37097]: pgmap v2140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:17 smithi171 ceph-mon[37789]: pgmap v2140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:17 smithi044 ceph-mon[32404]: pgmap v2140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:19 smithi171 ceph-mon[37789]: pgmap v2141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:19 smithi044 ceph-mon[32404]: pgmap v2141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:19 smithi116 ceph-mon[37097]: pgmap v2141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:21 smithi171 ceph-mon[37789]: pgmap v2142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:21.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3025.635913 secs 2024-07-26T09:31:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:21 smithi044 ceph-mon[32404]: pgmap v2142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3025.635913 secs 2024-07-26T09:31:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:21 smithi116 ceph-mon[37097]: pgmap v2142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3025.635913 secs 2024-07-26T09:31:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:23 smithi171 ceph-mon[37789]: pgmap v2143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:23 smithi044 ceph-mon[32404]: pgmap v2143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:23 smithi116 ceph-mon[37097]: pgmap v2143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:25 smithi171 ceph-mon[37789]: pgmap v2144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:25 smithi044 ceph-mon[32404]: pgmap v2144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:25 smithi116 ceph-mon[37097]: pgmap v2144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:26.127+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:31:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:26.127+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:26.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:26.127+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:31:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3030.636073 secs 2024-07-26T09:31:26.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3030.636073 secs 2024-07-26T09:31:26.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3030.636073 secs 2024-07-26T09:31:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:27 smithi044 ceph-mon[32404]: pgmap v2145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:27 smithi171 ceph-mon[37789]: pgmap v2145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:27 smithi116 ceph-mon[37097]: pgmap v2145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:29 smithi171 ceph-mon[37789]: pgmap v2146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:29 smithi044 ceph-mon[32404]: pgmap v2146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:29 smithi116 ceph-mon[37097]: pgmap v2146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:31 smithi171 ceph-mon[37789]: pgmap v2147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3035.636236 secs 2024-07-26T09:31:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:31 smithi044 ceph-mon[32404]: pgmap v2147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3035.636236 secs 2024-07-26T09:31:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:31 smithi116 ceph-mon[37097]: pgmap v2147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3035.636236 secs 2024-07-26T09:31:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:33 smithi171 ceph-mon[37789]: pgmap v2148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:33 smithi044 ceph-mon[32404]: pgmap v2148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:33 smithi116 ceph-mon[37097]: pgmap v2148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:35 smithi171 ceph-mon[37789]: pgmap v2149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:35 smithi044 ceph-mon[32404]: pgmap v2149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:35 smithi116 ceph-mon[37097]: pgmap v2149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3040.636398 secs 2024-07-26T09:31:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3040.636398 secs 2024-07-26T09:31:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3040.636398 secs 2024-07-26T09:31:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:37 smithi044 ceph-mon[32404]: pgmap v2150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:37 smithi116 ceph-mon[37097]: pgmap v2150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:37 smithi171 ceph-mon[37789]: pgmap v2150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:39 smithi044 ceph-mon[32404]: pgmap v2151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:31:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:39 smithi116 ceph-mon[37097]: pgmap v2151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:31:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:39 smithi171 ceph-mon[37789]: pgmap v2151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:31:41.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:41.130+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:31:41.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:41.130+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:41.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:41.130+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:31:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:41 smithi044 ceph-mon[32404]: pgmap v2152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3045.636563 secs 2024-07-26T09:31:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:41 smithi116 ceph-mon[37097]: pgmap v2152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3045.636563 secs 2024-07-26T09:31:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:41 smithi171 ceph-mon[37789]: pgmap v2152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:31:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3045.636563 secs 2024-07-26T09:31:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:43 smithi044 ceph-mon[32404]: pgmap v2153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:31:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:43 smithi116 ceph-mon[37097]: pgmap v2153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:31:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:43 smithi171 ceph-mon[37789]: pgmap v2153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:43.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:31:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:44 smithi044 ceph-mon[32404]: pgmap v2154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:31:44.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:44 smithi044 ceph-mon[32404]: pgmap v2155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:44 smithi116 ceph-mon[37097]: pgmap v2154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:31:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:44 smithi116 ceph-mon[37097]: pgmap v2155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:44 smithi171 ceph-mon[37789]: pgmap v2154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:31:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:44 smithi171 ceph-mon[37789]: pgmap v2155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:46 smithi044 ceph-mon[32404]: pgmap v2156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3050.636707 secs 2024-07-26T09:31:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:46 smithi116 ceph-mon[37097]: pgmap v2156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3050.636707 secs 2024-07-26T09:31:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:46 smithi171 ceph-mon[37789]: pgmap v2156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3050.636707 secs 2024-07-26T09:31:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:48 smithi044 ceph-mon[32404]: pgmap v2157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:31:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:48 smithi116 ceph-mon[37097]: pgmap v2157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:31:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:48 smithi171 ceph-mon[37789]: pgmap v2157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:31:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:50 smithi044 ceph-mon[32404]: pgmap v2158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:50 smithi116 ceph-mon[37097]: pgmap v2158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:50 smithi171 ceph-mon[37789]: pgmap v2158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:31:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3055.636808 secs 2024-07-26T09:31:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3055.636808 secs 2024-07-26T09:31:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3055.636808 secs 2024-07-26T09:31:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:52 smithi044 ceph-mon[32404]: pgmap v2159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:31:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:52 smithi116 ceph-mon[37097]: pgmap v2159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:31:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:52 smithi171 ceph-mon[37789]: pgmap v2159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:31:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:54 smithi044 ceph-mon[32404]: pgmap v2160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:31:54.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:54 smithi116 ceph-mon[37097]: pgmap v2160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:31:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:54 smithi171 ceph-mon[37789]: pgmap v2160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:31:56.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:56.138+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:31:56.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:56.138+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:56.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:31:56.138+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:31:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:56 smithi044 ceph-mon[32404]: pgmap v2161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3060.636974 secs 2024-07-26T09:31:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:56 smithi116 ceph-mon[37097]: pgmap v2161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3060.636974 secs 2024-07-26T09:31:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:56 smithi171 ceph-mon[37789]: pgmap v2161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:31:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3060.636974 secs 2024-07-26T09:31:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:31:58 smithi044 ceph-mon[32404]: pgmap v2162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:31:58 smithi116 ceph-mon[37097]: pgmap v2162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:31:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:31:58 smithi171 ceph-mon[37789]: pgmap v2162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:00 smithi044 ceph-mon[32404]: pgmap v2163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:00 smithi116 ceph-mon[37097]: pgmap v2163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:00 smithi171 ceph-mon[37789]: pgmap v2163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3065.637144 secs 2024-07-26T09:32:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3065.637144 secs 2024-07-26T09:32:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3065.637144 secs 2024-07-26T09:32:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:02 smithi044 ceph-mon[32404]: pgmap v2164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:02 smithi116 ceph-mon[37097]: pgmap v2164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:02 smithi171 ceph-mon[37789]: pgmap v2164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:04 smithi044 ceph-mon[32404]: pgmap v2165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:04 smithi116 ceph-mon[37097]: pgmap v2165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:04 smithi171 ceph-mon[37789]: pgmap v2165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:06 smithi044 ceph-mon[32404]: pgmap v2166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3070.637311 secs 2024-07-26T09:32:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:06 smithi116 ceph-mon[37097]: pgmap v2166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3070.637311 secs 2024-07-26T09:32:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:06 smithi171 ceph-mon[37789]: pgmap v2166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3070.637311 secs 2024-07-26T09:32:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:08 smithi044 ceph-mon[32404]: pgmap v2167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:08 smithi116 ceph-mon[37097]: pgmap v2167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:08 smithi171 ceph-mon[37789]: pgmap v2167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:10 smithi044 ceph-mon[32404]: pgmap v2168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:10 smithi116 ceph-mon[37097]: pgmap v2168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:10 smithi171 ceph-mon[37789]: pgmap v2168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:11.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:11.141+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:32:11.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:11.141+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:32:11.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:11.141+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3075.637472 secs 2024-07-26T09:32:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3075.637472 secs 2024-07-26T09:32:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3075.637472 secs 2024-07-26T09:32:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:12 smithi044 ceph-mon[32404]: pgmap v2169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:12 smithi116 ceph-mon[37097]: pgmap v2169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:12 smithi171 ceph-mon[37789]: pgmap v2169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:14 smithi044 ceph-mon[32404]: pgmap v2170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T09:32:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:14 smithi116 ceph-mon[37097]: pgmap v2170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T09:32:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:14 smithi171 ceph-mon[37789]: pgmap v2170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T09:32:15.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:15.206+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:32:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:15.206+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:32:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:15.206+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:32:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:15.206+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:32:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:16 smithi044 ceph-mon[32404]: pgmap v2171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:32:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3080.637566 secs 2024-07-26T09:32:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:16 smithi116 ceph-mon[37097]: pgmap v2171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:32:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3080.637566 secs 2024-07-26T09:32:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:16 smithi171 ceph-mon[37789]: pgmap v2171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:32:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3080.637566 secs 2024-07-26T09:32:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:18 smithi044 ceph-mon[32404]: pgmap v2172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:32:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:18 smithi116 ceph-mon[37097]: pgmap v2172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:32:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:18 smithi171 ceph-mon[37789]: pgmap v2172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:32:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:20 smithi044 ceph-mon[32404]: pgmap v2173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:32:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:20 smithi116 ceph-mon[37097]: pgmap v2173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:32:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:20 smithi171 ceph-mon[37789]: pgmap v2173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:32:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3085.637736 secs 2024-07-26T09:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3085.637736 secs 2024-07-26T09:32:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3085.637736 secs 2024-07-26T09:32:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:22 smithi116 ceph-mon[37097]: pgmap v2174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T09:32:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:22 smithi171 ceph-mon[37789]: pgmap v2174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T09:32:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:22 smithi044 ceph-mon[32404]: pgmap v2174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T09:32:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:24 smithi171 ceph-mon[37789]: pgmap v2175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T09:32:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:24 smithi044 ceph-mon[32404]: pgmap v2175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T09:32:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:24 smithi116 ceph-mon[37097]: pgmap v2175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T09:32:26.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:26.150+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:32:26.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:26.150+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:26.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:26.150+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:32:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:26 smithi044 ceph-mon[32404]: pgmap v2176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:32:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3090.637884 secs 2024-07-26T09:32:26.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:26 smithi116 ceph-mon[37097]: pgmap v2176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:32:26.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3090.637884 secs 2024-07-26T09:32:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:26 smithi171 ceph-mon[37789]: pgmap v2176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:32:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3090.637884 secs 2024-07-26T09:32:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:28 smithi171 ceph-mon[37789]: pgmap v2177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:32:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:28 smithi044 ceph-mon[32404]: pgmap v2177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:32:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:28 smithi116 ceph-mon[37097]: pgmap v2177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:32:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:30 smithi171 ceph-mon[37789]: pgmap v2178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:30 smithi044 ceph-mon[32404]: pgmap v2178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:30 smithi116 ceph-mon[37097]: pgmap v2178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3095.638021 secs 2024-07-26T09:32:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3095.638021 secs 2024-07-26T09:32:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3095.638021 secs 2024-07-26T09:32:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:32 smithi171 ceph-mon[37789]: pgmap v2179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:32 smithi044 ceph-mon[32404]: pgmap v2179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:32 smithi116 ceph-mon[37097]: pgmap v2179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:33 smithi171 ceph-mon[37789]: pgmap v2180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:33 smithi044 ceph-mon[32404]: pgmap v2180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:33 smithi116 ceph-mon[37097]: pgmap v2180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:36 smithi044 ceph-mon[32404]: pgmap v2181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3100.638188 secs 2024-07-26T09:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:36 smithi116 ceph-mon[37097]: pgmap v2181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3100.638188 secs 2024-07-26T09:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:36 smithi171 ceph-mon[37789]: pgmap v2181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3100.638188 secs 2024-07-26T09:32:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:38 smithi044 ceph-mon[32404]: pgmap v2182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:38.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:38 smithi116 ceph-mon[37097]: pgmap v2182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:38 smithi171 ceph-mon[37789]: pgmap v2182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:40 smithi044 ceph-mon[32404]: pgmap v2183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:40 smithi116 ceph-mon[37097]: pgmap v2183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:40 smithi171 ceph-mon[37789]: pgmap v2183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:41.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:41.153+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:32:41.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:41.153+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:41.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:41.153+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:32:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3105.638320 secs 2024-07-26T09:32:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3105.638320 secs 2024-07-26T09:32:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3105.638320 secs 2024-07-26T09:32:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:42 smithi044 ceph-mon[32404]: pgmap v2184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:42 smithi116 ceph-mon[37097]: pgmap v2184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:42 smithi171 ceph-mon[37789]: pgmap v2184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:44.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:44 smithi116 ceph-mon[37097]: pgmap v2185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:44.470 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:44 smithi044 ceph-mon[32404]: pgmap v2185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:44 smithi171 ceph-mon[37789]: pgmap v2185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:32:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:46 smithi044 ceph-mon[32404]: pgmap v2186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3110.638454 secs 2024-07-26T09:32:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:46 smithi116 ceph-mon[37097]: pgmap v2186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3110.638454 secs 2024-07-26T09:32:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:46 smithi171 ceph-mon[37789]: pgmap v2186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:32:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3110.638454 secs 2024-07-26T09:32:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:48.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:48.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:48 smithi171 ceph-mon[37789]: pgmap v2187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:48 smithi044 ceph-mon[32404]: pgmap v2187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:48 smithi116 ceph-mon[37097]: pgmap v2187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: pgmap v2188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:50.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: pgmap v2188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:50.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:50.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: pgmap v2188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:32:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:50.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:32:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:51 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:32:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:51 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:32:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:51 smithi044 ceph-mon[32404]: pgmap v2189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:51 smithi044 ceph-mon[32404]: pgmap v2190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:32:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3115.638608 secs 2024-07-26T09:32:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:51 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:32:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:51 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:32:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:51 smithi116 ceph-mon[37097]: pgmap v2189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:51 smithi116 ceph-mon[37097]: pgmap v2190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:32:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3115.638608 secs 2024-07-26T09:32:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:51 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:32:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:51 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:32:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:51 smithi171 ceph-mon[37789]: pgmap v2189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:32:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:51 smithi171 ceph-mon[37789]: pgmap v2190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:32:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3115.638608 secs 2024-07-26T09:32:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:53 smithi044 ceph-mon[32404]: pgmap v2191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:32:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:53 smithi116 ceph-mon[37097]: pgmap v2191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:32:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:53 smithi171 ceph-mon[37789]: pgmap v2191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:32:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:55 smithi044 ceph-mon[32404]: pgmap v2192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T09:32:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:55 smithi116 ceph-mon[37097]: pgmap v2192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T09:32:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:55 smithi171 ceph-mon[37789]: pgmap v2192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T09:32:56.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:56.162+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:32:56.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:56.162+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:56.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:32:56.162+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:32:56.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3120.638762 secs 2024-07-26T09:32:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3120.638762 secs 2024-07-26T09:32:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3120.638762 secs 2024-07-26T09:32:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:57 smithi116 ceph-mon[37097]: pgmap v2193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:32:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:57 smithi044 ceph-mon[32404]: pgmap v2193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:32:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:57 smithi171 ceph-mon[37789]: pgmap v2193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:32:59.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:32:59 smithi044 ceph-mon[32404]: pgmap v2194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:32:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:32:59 smithi116 ceph-mon[37097]: pgmap v2194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:32:59.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:32:59 smithi171 ceph-mon[37789]: pgmap v2194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:33:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:01 smithi044 ceph-mon[32404]: pgmap v2195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:33:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3125.638914 secs 2024-07-26T09:33:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:01 smithi116 ceph-mon[37097]: pgmap v2195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:33:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3125.638914 secs 2024-07-26T09:33:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:01 smithi171 ceph-mon[37789]: pgmap v2195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:33:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3125.638914 secs 2024-07-26T09:33:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:03 smithi044 ceph-mon[32404]: pgmap v2196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:03 smithi116 ceph-mon[37097]: pgmap v2196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:03 smithi171 ceph-mon[37789]: pgmap v2196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:05 smithi044 ceph-mon[32404]: pgmap v2197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:05 smithi116 ceph-mon[37097]: pgmap v2197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:05 smithi171 ceph-mon[37789]: pgmap v2197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:06.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3130.639088 secs 2024-07-26T09:33:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3130.639088 secs 2024-07-26T09:33:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3130.639088 secs 2024-07-26T09:33:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:07 smithi044 ceph-mon[32404]: pgmap v2198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:07 smithi116 ceph-mon[37097]: pgmap v2198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:07 smithi171 ceph-mon[37789]: pgmap v2198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:09 smithi044 ceph-mon[32404]: pgmap v2199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:09 smithi116 ceph-mon[37097]: pgmap v2199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:09 smithi171 ceph-mon[37789]: pgmap v2199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:11.164+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:33:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:11.164+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:11.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:11.164+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:33:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:11 smithi044 ceph-mon[32404]: pgmap v2200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3135.639211 secs 2024-07-26T09:33:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:11 smithi116 ceph-mon[37097]: pgmap v2200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3135.639211 secs 2024-07-26T09:33:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:11 smithi171 ceph-mon[37789]: pgmap v2200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3135.639211 secs 2024-07-26T09:33:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:13 smithi044 ceph-mon[32404]: pgmap v2201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:13 smithi116 ceph-mon[37097]: pgmap v2201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:13 smithi171 ceph-mon[37789]: pgmap v2201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:15 smithi044 ceph-mon[32404]: pgmap v2202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:15 smithi116 ceph-mon[37097]: pgmap v2202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:15 smithi171 ceph-mon[37789]: pgmap v2202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3140.639387 secs 2024-07-26T09:33:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3140.639387 secs 2024-07-26T09:33:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3140.639387 secs 2024-07-26T09:33:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:17 smithi116 ceph-mon[37097]: pgmap v2203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:17 smithi171 ceph-mon[37789]: pgmap v2203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:17 smithi044 ceph-mon[32404]: pgmap v2203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:19 smithi171 ceph-mon[37789]: pgmap v2204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:19 smithi044 ceph-mon[32404]: pgmap v2204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:19 smithi116 ceph-mon[37097]: pgmap v2204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:21 smithi171 ceph-mon[37789]: pgmap v2205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3145.639520 secs 2024-07-26T09:33:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:21 smithi044 ceph-mon[32404]: pgmap v2205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3145.639520 secs 2024-07-26T09:33:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:21 smithi116 ceph-mon[37097]: pgmap v2205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3145.639520 secs 2024-07-26T09:33:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:23 smithi171 ceph-mon[37789]: pgmap v2206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:23 smithi044 ceph-mon[32404]: pgmap v2206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:23 smithi116 ceph-mon[37097]: pgmap v2206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:25 smithi171 ceph-mon[37789]: pgmap v2207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:25 smithi044 ceph-mon[32404]: pgmap v2207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:25.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:25 smithi116 ceph-mon[37097]: pgmap v2207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:26.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:26.166+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:33:26.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:26.166+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:26.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:26.166+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:33:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3150.639669 secs 2024-07-26T09:33:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3150.639669 secs 2024-07-26T09:33:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3150.639669 secs 2024-07-26T09:33:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:27 smithi171 ceph-mon[37789]: pgmap v2208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:27 smithi044 ceph-mon[32404]: pgmap v2208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:27 smithi116 ceph-mon[37097]: pgmap v2208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:29 smithi171 ceph-mon[37789]: pgmap v2209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:29 smithi044 ceph-mon[32404]: pgmap v2209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:29 smithi116 ceph-mon[37097]: pgmap v2209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:31 smithi171 ceph-mon[37789]: pgmap v2210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3155.639775 secs 2024-07-26T09:33:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:31 smithi044 ceph-mon[32404]: pgmap v2210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3155.639775 secs 2024-07-26T09:33:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:31 smithi116 ceph-mon[37097]: pgmap v2210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3155.639775 secs 2024-07-26T09:33:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:32 smithi171 ceph-mon[37789]: pgmap v2211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:32 smithi044 ceph-mon[32404]: pgmap v2211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:32 smithi116 ceph-mon[37097]: pgmap v2211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:35 smithi171 ceph-mon[37789]: pgmap v2212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:35 smithi044 ceph-mon[32404]: pgmap v2212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:35 smithi116 ceph-mon[37097]: pgmap v2212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3160.639917 secs 2024-07-26T09:33:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3160.639917 secs 2024-07-26T09:33:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3160.639917 secs 2024-07-26T09:33:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:37 smithi171 ceph-mon[37789]: pgmap v2213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:37 smithi044 ceph-mon[32404]: pgmap v2213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:37 smithi116 ceph-mon[37097]: pgmap v2213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:39 smithi171 ceph-mon[37789]: pgmap v2214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:39 smithi044 ceph-mon[32404]: pgmap v2214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:39 smithi116 ceph-mon[37097]: pgmap v2214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:41.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:41.169+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:33:41.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:41.169+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:41.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:41.169+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:33:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:41 smithi171 ceph-mon[37789]: pgmap v2215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3165.640080 secs 2024-07-26T09:33:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:41 smithi044 ceph-mon[32404]: pgmap v2215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3165.640080 secs 2024-07-26T09:33:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:41 smithi116 ceph-mon[37097]: pgmap v2215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:33:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3165.640080 secs 2024-07-26T09:33:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:43 smithi171 ceph-mon[37789]: pgmap v2216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:43 smithi044 ceph-mon[32404]: pgmap v2216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:43 smithi116 ceph-mon[37097]: pgmap v2216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:45 smithi044 ceph-mon[32404]: pgmap v2217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:45 smithi116 ceph-mon[37097]: pgmap v2217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:45 smithi171 ceph-mon[37789]: pgmap v2217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3170.640222 secs 2024-07-26T09:33:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3170.640222 secs 2024-07-26T09:33:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3170.640222 secs 2024-07-26T09:33:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:47 smithi044 ceph-mon[32404]: pgmap v2218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:33:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:47 smithi116 ceph-mon[37097]: pgmap v2218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:33:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:47 smithi171 ceph-mon[37789]: pgmap v2218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:33:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:49 smithi044 ceph-mon[32404]: pgmap v2219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:49 smithi116 ceph-mon[37097]: pgmap v2219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:49 smithi171 ceph-mon[37789]: pgmap v2219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:51.358 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: pgmap v2220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:33:51.358 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3175.640333 secs 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: pgmap v2220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3175.640333 secs 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: pgmap v2220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3175.640333 secs 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:51.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:52.378 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:52 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:33:52.379 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:52 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:33:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:52 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:33:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:52 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:33:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:52 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:33:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:52 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:33:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:53 smithi044 ceph-mon[32404]: pgmap v2221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:53 smithi116 ceph-mon[37097]: pgmap v2221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:53.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:53 smithi171 ceph-mon[37789]: pgmap v2221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: pgmap v2222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: pgmap v2223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:33:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: pgmap v2224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:55.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: pgmap v2222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: pgmap v2223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: pgmap v2224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: pgmap v2222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:33:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: pgmap v2223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: pgmap v2224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:33:56.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:56.176+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:33:56.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:56.176+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:56.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:33:56.176+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:33:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3180.640463 secs 2024-07-26T09:33:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3180.640463 secs 2024-07-26T09:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3180.640463 secs 2024-07-26T09:33:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:57 smithi044 ceph-mon[32404]: pgmap v2225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T09:33:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:57 smithi116 ceph-mon[37097]: pgmap v2225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T09:33:57.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:57 smithi171 ceph-mon[37789]: pgmap v2225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T09:33:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:33:59 smithi171 ceph-mon[37789]: pgmap v2226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:33:59 smithi044 ceph-mon[32404]: pgmap v2226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:33:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:33:59 smithi116 ceph-mon[37097]: pgmap v2226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:01 smithi044 ceph-mon[32404]: pgmap v2227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3185.640608 secs 2024-07-26T09:34:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:01 smithi116 ceph-mon[37097]: pgmap v2227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3185.640608 secs 2024-07-26T09:34:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:01 smithi171 ceph-mon[37789]: pgmap v2227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3185.640608 secs 2024-07-26T09:34:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:03 smithi044 ceph-mon[32404]: pgmap v2228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:03 smithi116 ceph-mon[37097]: pgmap v2228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:03.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:03 smithi171 ceph-mon[37789]: pgmap v2228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:34:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:05 smithi044 ceph-mon[32404]: pgmap v2229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:34:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:05 smithi116 ceph-mon[37097]: pgmap v2229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:34:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:05 smithi171 ceph-mon[37789]: pgmap v2229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:34:06.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3190.640728 secs 2024-07-26T09:34:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3190.640728 secs 2024-07-26T09:34:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3190.640728 secs 2024-07-26T09:34:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:07 smithi044 ceph-mon[32404]: pgmap v2230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:07 smithi116 ceph-mon[37097]: pgmap v2230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:07 smithi171 ceph-mon[37789]: pgmap v2230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:09 smithi044 ceph-mon[32404]: pgmap v2231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:09 smithi116 ceph-mon[37097]: pgmap v2231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:09 smithi171 ceph-mon[37789]: pgmap v2231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:11.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:11.178+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:34:11.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:11.178+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:11.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:11.178+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:34:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:11 smithi044 ceph-mon[32404]: pgmap v2232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3195.640891 secs 2024-07-26T09:34:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:11 smithi116 ceph-mon[37097]: pgmap v2232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3195.640891 secs 2024-07-26T09:34:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:11 smithi171 ceph-mon[37789]: pgmap v2232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3195.640891 secs 2024-07-26T09:34:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:13 smithi044 ceph-mon[32404]: pgmap v2233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:13 smithi116 ceph-mon[37097]: pgmap v2233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:13 smithi171 ceph-mon[37789]: pgmap v2233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:15 smithi044 ceph-mon[32404]: pgmap v2234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:15 smithi116 ceph-mon[37097]: pgmap v2234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:15 smithi171 ceph-mon[37789]: pgmap v2234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3200.641032 secs 2024-07-26T09:34:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3200.641032 secs 2024-07-26T09:34:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3200.641032 secs 2024-07-26T09:34:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:17 smithi044 ceph-mon[32404]: pgmap v2235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:17 smithi116 ceph-mon[37097]: pgmap v2235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:17 smithi171 ceph-mon[37789]: pgmap v2235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:19 smithi044 ceph-mon[32404]: pgmap v2236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:19 smithi116 ceph-mon[37097]: pgmap v2236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:19 smithi171 ceph-mon[37789]: pgmap v2236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:21 smithi044 ceph-mon[32404]: pgmap v2237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3205.641194 secs 2024-07-26T09:34:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:21 smithi116 ceph-mon[37097]: pgmap v2237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3205.641194 secs 2024-07-26T09:34:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:21 smithi171 ceph-mon[37789]: pgmap v2237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3205.641194 secs 2024-07-26T09:34:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:23 smithi044 ceph-mon[32404]: pgmap v2238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:23 smithi116 ceph-mon[37097]: pgmap v2238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:23 smithi171 ceph-mon[37789]: pgmap v2238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:25 smithi044 ceph-mon[32404]: pgmap v2239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:25 smithi116 ceph-mon[37097]: pgmap v2239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:25 smithi171 ceph-mon[37789]: pgmap v2239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:26.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:26.186+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:34:26.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:26.186+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:34:26.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:26.186+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3210.641358 secs 2024-07-26T09:34:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3210.641358 secs 2024-07-26T09:34:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3210.641358 secs 2024-07-26T09:34:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:27 smithi044 ceph-mon[32404]: pgmap v2240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:27 smithi116 ceph-mon[37097]: pgmap v2240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:27 smithi171 ceph-mon[37789]: pgmap v2240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:29 smithi044 ceph-mon[32404]: pgmap v2241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:29 smithi116 ceph-mon[37097]: pgmap v2241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:29 smithi171 ceph-mon[37789]: pgmap v2241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:31 smithi044 ceph-mon[32404]: pgmap v2242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3215.641507 secs 2024-07-26T09:34:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:31 smithi116 ceph-mon[37097]: pgmap v2242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3215.641507 secs 2024-07-26T09:34:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:31 smithi171 ceph-mon[37789]: pgmap v2242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3215.641507 secs 2024-07-26T09:34:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:33 smithi044 ceph-mon[32404]: pgmap v2243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:33 smithi116 ceph-mon[37097]: pgmap v2243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:33 smithi171 ceph-mon[37789]: pgmap v2243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:35 smithi044 ceph-mon[32404]: pgmap v2244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:35 smithi116 ceph-mon[37097]: pgmap v2244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:35 smithi171 ceph-mon[37789]: pgmap v2244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:36.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3220.641688 secs 2024-07-26T09:34:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3220.641688 secs 2024-07-26T09:34:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3220.641688 secs 2024-07-26T09:34:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:37 smithi044 ceph-mon[32404]: pgmap v2245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:37 smithi116 ceph-mon[37097]: pgmap v2245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:37 smithi171 ceph-mon[37789]: pgmap v2245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:39 smithi044 ceph-mon[32404]: pgmap v2246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:39 smithi116 ceph-mon[37097]: pgmap v2246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:39 smithi171 ceph-mon[37789]: pgmap v2246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:41.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:41.190+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:34:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:41.190+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:41.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:41.190+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:34:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:41 smithi044 ceph-mon[32404]: pgmap v2247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3225.641827 secs 2024-07-26T09:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:41 smithi116 ceph-mon[37097]: pgmap v2247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3225.641827 secs 2024-07-26T09:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:41 smithi171 ceph-mon[37789]: pgmap v2247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3225.641827 secs 2024-07-26T09:34:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:43 smithi044 ceph-mon[32404]: pgmap v2248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:43 smithi116 ceph-mon[37097]: pgmap v2248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:43 smithi171 ceph-mon[37789]: pgmap v2248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:45 smithi044 ceph-mon[32404]: pgmap v2249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:45 smithi116 ceph-mon[37097]: pgmap v2249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:45 smithi171 ceph-mon[37789]: pgmap v2249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3230.641999 secs 2024-07-26T09:34:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3230.641999 secs 2024-07-26T09:34:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3230.641999 secs 2024-07-26T09:34:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:47 smithi044 ceph-mon[32404]: pgmap v2250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:47 smithi116 ceph-mon[37097]: pgmap v2250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:47 smithi171 ceph-mon[37789]: pgmap v2250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:49 smithi116 ceph-mon[37097]: pgmap v2251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:49 smithi171 ceph-mon[37789]: pgmap v2251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:49.762 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:49 smithi044 ceph-mon[32404]: pgmap v2251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:34:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:51 smithi171 ceph-mon[37789]: pgmap v2252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3235.642164 secs 2024-07-26T09:34:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:51 smithi044 ceph-mon[32404]: pgmap v2252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3235.642164 secs 2024-07-26T09:34:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:51 smithi116 ceph-mon[37097]: pgmap v2252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3235.642164 secs 2024-07-26T09:34:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:53 smithi171 ceph-mon[37789]: pgmap v2253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:53 smithi044 ceph-mon[32404]: pgmap v2253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:53 smithi116 ceph-mon[37097]: pgmap v2253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:55 smithi171 ceph-mon[37789]: pgmap v2254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:55.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:55.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:34:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:55 smithi044 ceph-mon[32404]: pgmap v2254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:34:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:55 smithi116 ceph-mon[37097]: pgmap v2254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:34:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:34:56.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:56.199+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:34:56.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:56.199+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:56.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:34:56.199+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:34:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:56 smithi044 ceph-mon[32404]: pgmap v2255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:34:56.528 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:56 smithi044 ceph-mon[32404]: pgmap v2256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:34:56.528 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3240.642245 secs 2024-07-26T09:34:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:56 smithi171 ceph-mon[37789]: pgmap v2255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:34:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:56 smithi171 ceph-mon[37789]: pgmap v2256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:34:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3240.642245 secs 2024-07-26T09:34:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:56 smithi116 ceph-mon[37097]: pgmap v2255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:34:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:56 smithi116 ceph-mon[37097]: pgmap v2256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:34:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3240.642245 secs 2024-07-26T09:34:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:34:58 smithi171 ceph-mon[37789]: pgmap v2257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:34:58 smithi044 ceph-mon[32404]: pgmap v2257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:34:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:34:58 smithi116 ceph-mon[37097]: pgmap v2257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:00 smithi171 ceph-mon[37789]: pgmap v2258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:00 smithi044 ceph-mon[32404]: pgmap v2258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:00 smithi116 ceph-mon[37097]: pgmap v2258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3245.642406 secs 2024-07-26T09:35:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3245.642406 secs 2024-07-26T09:35:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3245.642406 secs 2024-07-26T09:35:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:02 smithi171 ceph-mon[37789]: pgmap v2259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:02 smithi044 ceph-mon[32404]: pgmap v2259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:02 smithi116 ceph-mon[37097]: pgmap v2259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:03 smithi044 ceph-mon[32404]: pgmap v2260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:03 smithi116 ceph-mon[37097]: pgmap v2260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:03 smithi171 ceph-mon[37789]: pgmap v2260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:35:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:06 smithi171 ceph-mon[37789]: pgmap v2261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:35:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3250.642553 secs 2024-07-26T09:35:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:06 smithi044 ceph-mon[32404]: pgmap v2261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:35:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3250.642553 secs 2024-07-26T09:35:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:06 smithi116 ceph-mon[37097]: pgmap v2261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:35:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3250.642553 secs 2024-07-26T09:35:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:08 smithi044 ceph-mon[32404]: pgmap v2262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:08 smithi116 ceph-mon[37097]: pgmap v2262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:08 smithi171 ceph-mon[37789]: pgmap v2262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:09 smithi044 ceph-mon[32404]: pgmap v2263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:09 smithi116 ceph-mon[37097]: pgmap v2263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:09 smithi171 ceph-mon[37789]: pgmap v2263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:11.203+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:35:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:11.203+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:35:11.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:11.203+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3255.642695 secs 2024-07-26T09:35:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3255.642695 secs 2024-07-26T09:35:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3255.642695 secs 2024-07-26T09:35:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:11 smithi044 ceph-mon[32404]: pgmap v2264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:11 smithi116 ceph-mon[37097]: pgmap v2264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:11 smithi171 ceph-mon[37789]: pgmap v2264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:14 smithi044 ceph-mon[32404]: pgmap v2265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:14 smithi116 ceph-mon[37097]: pgmap v2265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:14 smithi171 ceph-mon[37789]: pgmap v2265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:15.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:15.239+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:35:15.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:15.239+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:35:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:15.240+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:35:15.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:15.240+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:35:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:15 smithi044 ceph-mon[32404]: pgmap v2266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:15 smithi116 ceph-mon[37097]: pgmap v2266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:15 smithi171 ceph-mon[37789]: pgmap v2266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3260.642862 secs 2024-07-26T09:35:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3260.642862 secs 2024-07-26T09:35:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3260.642862 secs 2024-07-26T09:35:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:17 smithi171 ceph-mon[37789]: pgmap v2267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:17 smithi044 ceph-mon[32404]: pgmap v2267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:17 smithi116 ceph-mon[37097]: pgmap v2267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:20 smithi044 ceph-mon[32404]: pgmap v2268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:20 smithi116 ceph-mon[37097]: pgmap v2268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:20 smithi171 ceph-mon[37789]: pgmap v2268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3265.643023 secs 2024-07-26T09:35:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3265.643023 secs 2024-07-26T09:35:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3265.643023 secs 2024-07-26T09:35:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:22 smithi044 ceph-mon[32404]: pgmap v2269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:22 smithi116 ceph-mon[37097]: pgmap v2269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:22 smithi171 ceph-mon[37789]: pgmap v2269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:23 smithi044 ceph-mon[32404]: pgmap v2270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:23 smithi116 ceph-mon[37097]: pgmap v2270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:23 smithi171 ceph-mon[37789]: pgmap v2270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:26.211+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:35:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:26.212+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:35:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:26.212+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:26.470 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:26 smithi044 ceph-mon[32404]: pgmap v2271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:26.471 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3270.643184 secs 2024-07-26T09:35:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:26 smithi116 ceph-mon[37097]: pgmap v2271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3270.643184 secs 2024-07-26T09:35:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:26 smithi171 ceph-mon[37789]: pgmap v2271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3270.643184 secs 2024-07-26T09:35:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:27 smithi044 ceph-mon[32404]: pgmap v2272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:27 smithi116 ceph-mon[37097]: pgmap v2272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:27 smithi171 ceph-mon[37789]: pgmap v2272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:29 smithi044 ceph-mon[32404]: pgmap v2273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:29 smithi116 ceph-mon[37097]: pgmap v2273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:29 smithi171 ceph-mon[37789]: pgmap v2273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3275.643350 secs 2024-07-26T09:35:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3275.643350 secs 2024-07-26T09:35:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3275.643350 secs 2024-07-26T09:35:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:31 smithi044 ceph-mon[32404]: pgmap v2274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:35:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:31 smithi116 ceph-mon[37097]: pgmap v2274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:35:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:31 smithi171 ceph-mon[37789]: pgmap v2274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:35:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:33 smithi044 ceph-mon[32404]: pgmap v2275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:33 smithi116 ceph-mon[37097]: pgmap v2275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:33 smithi171 ceph-mon[37789]: pgmap v2275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:35 smithi044 ceph-mon[32404]: pgmap v2276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:35 smithi116 ceph-mon[37097]: pgmap v2276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:35 smithi171 ceph-mon[37789]: pgmap v2276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:36.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3280.643512 secs 2024-07-26T09:35:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3280.643512 secs 2024-07-26T09:35:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3280.643512 secs 2024-07-26T09:35:37.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:37 smithi044 ceph-mon[32404]: pgmap v2277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:37 smithi116 ceph-mon[37097]: pgmap v2277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:37 smithi171 ceph-mon[37789]: pgmap v2277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:39 smithi044 ceph-mon[32404]: pgmap v2278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:39 smithi116 ceph-mon[37097]: pgmap v2278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:39 smithi171 ceph-mon[37789]: pgmap v2278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:41.216+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:35:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:41.216+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:41.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:41.216+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:35:41.219 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3285.643674 secs 2024-07-26T09:35:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3285.643674 secs 2024-07-26T09:35:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3285.643674 secs 2024-07-26T09:35:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:41 smithi044 ceph-mon[32404]: pgmap v2279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:35:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:41 smithi116 ceph-mon[37097]: pgmap v2279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:35:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:41 smithi171 ceph-mon[37789]: pgmap v2279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:35:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:43 smithi044 ceph-mon[32404]: pgmap v2280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:43 smithi116 ceph-mon[37097]: pgmap v2280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:43 smithi171 ceph-mon[37789]: pgmap v2280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:45 smithi044 ceph-mon[32404]: pgmap v2281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:45 smithi116 ceph-mon[37097]: pgmap v2281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:45 smithi171 ceph-mon[37789]: pgmap v2281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3290.643819 secs 2024-07-26T09:35:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3290.643819 secs 2024-07-26T09:35:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3290.643819 secs 2024-07-26T09:35:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:47 smithi044 ceph-mon[32404]: pgmap v2282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:47 smithi116 ceph-mon[37097]: pgmap v2282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:47 smithi171 ceph-mon[37789]: pgmap v2282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:49 smithi044 ceph-mon[32404]: pgmap v2283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:49 smithi116 ceph-mon[37097]: pgmap v2283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:49 smithi171 ceph-mon[37789]: pgmap v2283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:35:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3295.643963 secs 2024-07-26T09:35:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3295.643963 secs 2024-07-26T09:35:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3295.643963 secs 2024-07-26T09:35:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:51 smithi044 ceph-mon[32404]: pgmap v2284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:51 smithi116 ceph-mon[37097]: pgmap v2284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:51 smithi171 ceph-mon[37789]: pgmap v2284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:53 smithi044 ceph-mon[32404]: pgmap v2285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:53 smithi116 ceph-mon[37097]: pgmap v2285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:53 smithi171 ceph-mon[37789]: pgmap v2285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:35:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:55 smithi044 ceph-mon[32404]: pgmap v2286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:55 smithi116 ceph-mon[37097]: pgmap v2286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:55 smithi171 ceph-mon[37789]: pgmap v2286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:35:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:56.224+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:35:56.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:56.224+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:35:56.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:35:56.224+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3300.644119 secs 2024-07-26T09:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3300.644119 secs 2024-07-26T09:35:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3300.644119 secs 2024-07-26T09:35:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:57 smithi044 ceph-mon[32404]: pgmap v2287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:57 smithi044 ceph-mon[32404]: pgmap v2288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:35:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:35:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:57 smithi116 ceph-mon[37097]: pgmap v2287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:57 smithi116 ceph-mon[37097]: pgmap v2288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:35:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:35:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:35:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:57.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:57.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:57 smithi171 ceph-mon[37789]: pgmap v2287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:35:57.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:57 smithi171 ceph-mon[37789]: pgmap v2288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:35:57.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:35:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:00 smithi171 ceph-mon[37789]: pgmap v2289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:00 smithi044 ceph-mon[32404]: pgmap v2289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:00 smithi116 ceph-mon[37097]: pgmap v2289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3305.644222 secs 2024-07-26T09:36:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3305.644222 secs 2024-07-26T09:36:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3305.644222 secs 2024-07-26T09:36:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:02 smithi171 ceph-mon[37789]: pgmap v2290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:02 smithi044 ceph-mon[32404]: pgmap v2290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:02 smithi116 ceph-mon[37097]: pgmap v2290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:04 smithi171 ceph-mon[37789]: pgmap v2291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:04 smithi044 ceph-mon[32404]: pgmap v2291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:04 smithi116 ceph-mon[37097]: pgmap v2291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:06 smithi171 ceph-mon[37789]: pgmap v2292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:06.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3310.644369 secs 2024-07-26T09:36:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:06 smithi044 ceph-mon[32404]: pgmap v2292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3310.644369 secs 2024-07-26T09:36:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:06 smithi116 ceph-mon[37097]: pgmap v2292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:36:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3310.644369 secs 2024-07-26T09:36:08.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:08 smithi171 ceph-mon[37789]: pgmap v2293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:36:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:08 smithi044 ceph-mon[32404]: pgmap v2293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:36:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:08 smithi116 ceph-mon[37097]: pgmap v2293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:36:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:10 smithi171 ceph-mon[37789]: pgmap v2294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:10 smithi044 ceph-mon[32404]: pgmap v2294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:10 smithi116 ceph-mon[37097]: pgmap v2294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:11.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:11.227+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:36:11.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:11.227+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:36:11.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:11.227+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3315.644511 secs 2024-07-26T09:36:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3315.644511 secs 2024-07-26T09:36:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3315.644511 secs 2024-07-26T09:36:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:12 smithi044 ceph-mon[32404]: pgmap v2295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:12 smithi116 ceph-mon[37097]: pgmap v2295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:12 smithi171 ceph-mon[37789]: pgmap v2295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:14 smithi044 ceph-mon[32404]: pgmap v2296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:36:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:14 smithi116 ceph-mon[37097]: pgmap v2296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:36:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:14 smithi171 ceph-mon[37789]: pgmap v2296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:36:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:16 smithi044 ceph-mon[32404]: pgmap v2297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3320.644658 secs 2024-07-26T09:36:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:16 smithi116 ceph-mon[37097]: pgmap v2297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3320.644658 secs 2024-07-26T09:36:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:16 smithi171 ceph-mon[37789]: pgmap v2297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:36:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3320.644658 secs 2024-07-26T09:36:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:17 smithi044 ceph-mon[32404]: pgmap v2298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:17 smithi116 ceph-mon[37097]: pgmap v2298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:17 smithi171 ceph-mon[37789]: pgmap v2298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:36:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:20 smithi171 ceph-mon[37789]: pgmap v2299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:20 smithi044 ceph-mon[32404]: pgmap v2299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:20 smithi116 ceph-mon[37097]: pgmap v2299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:36:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3325.644796 secs 2024-07-26T09:36:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3325.644796 secs 2024-07-26T09:36:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3325.644796 secs 2024-07-26T09:36:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:22 smithi171 ceph-mon[37789]: pgmap v2300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:36:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:22 smithi044 ceph-mon[32404]: pgmap v2300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:36:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:22 smithi116 ceph-mon[37097]: pgmap v2300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:36:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:24 smithi171 ceph-mon[37789]: pgmap v2301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:36:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:24 smithi044 ceph-mon[32404]: pgmap v2301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:36:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:24 smithi116 ceph-mon[37097]: pgmap v2301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:36:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:26.235+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:36:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:26.235+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:26.235+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:36:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:26 smithi044 ceph-mon[32404]: pgmap v2302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3330.644930 secs 2024-07-26T09:36:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:26 smithi171 ceph-mon[37789]: pgmap v2302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3330.644930 secs 2024-07-26T09:36:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:26 smithi116 ceph-mon[37097]: pgmap v2302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3330.644930 secs 2024-07-26T09:36:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:28 smithi171 ceph-mon[37789]: pgmap v2303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:28 smithi044 ceph-mon[32404]: pgmap v2303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:28 smithi116 ceph-mon[37097]: pgmap v2303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:30 smithi171 ceph-mon[37789]: pgmap v2304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:30 smithi044 ceph-mon[32404]: pgmap v2304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:30 smithi116 ceph-mon[37097]: pgmap v2304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3335.645076 secs 2024-07-26T09:36:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3335.645076 secs 2024-07-26T09:36:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3335.645076 secs 2024-07-26T09:36:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:32 smithi044 ceph-mon[32404]: pgmap v2305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:32 smithi116 ceph-mon[37097]: pgmap v2305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:32 smithi171 ceph-mon[37789]: pgmap v2305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:34 smithi044 ceph-mon[32404]: pgmap v2306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:34 smithi116 ceph-mon[37097]: pgmap v2306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:34 smithi171 ceph-mon[37789]: pgmap v2306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:35 smithi044 ceph-mon[32404]: pgmap v2307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:35 smithi116 ceph-mon[37097]: pgmap v2307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:35 smithi171 ceph-mon[37789]: pgmap v2307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:36.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3340.645243 secs 2024-07-26T09:36:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3340.645243 secs 2024-07-26T09:36:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3340.645243 secs 2024-07-26T09:36:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:37 smithi044 ceph-mon[32404]: pgmap v2308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:37 smithi116 ceph-mon[37097]: pgmap v2308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:37 smithi171 ceph-mon[37789]: pgmap v2308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:40 smithi171 ceph-mon[37789]: pgmap v2309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:40 smithi044 ceph-mon[32404]: pgmap v2309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:40 smithi116 ceph-mon[37097]: pgmap v2309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:41.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:41.239+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:36:41.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:41.239+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:41.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:41.239+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:36:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3345.645406 secs 2024-07-26T09:36:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3345.645406 secs 2024-07-26T09:36:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3345.645406 secs 2024-07-26T09:36:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:42 smithi171 ceph-mon[37789]: pgmap v2310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:42 smithi044 ceph-mon[32404]: pgmap v2310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:42 smithi116 ceph-mon[37097]: pgmap v2310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:44 smithi171 ceph-mon[37789]: pgmap v2311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:44 smithi044 ceph-mon[32404]: pgmap v2311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:44 smithi116 ceph-mon[37097]: pgmap v2311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:46 smithi044 ceph-mon[32404]: pgmap v2312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3350.645499 secs 2024-07-26T09:36:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:46 smithi116 ceph-mon[37097]: pgmap v2312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3350.645499 secs 2024-07-26T09:36:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:46 smithi171 ceph-mon[37789]: pgmap v2312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3350.645499 secs 2024-07-26T09:36:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:48 smithi044 ceph-mon[32404]: pgmap v2313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:48 smithi116 ceph-mon[37097]: pgmap v2313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:48 smithi171 ceph-mon[37789]: pgmap v2313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:50 smithi044 ceph-mon[32404]: pgmap v2314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:50 smithi116 ceph-mon[37097]: pgmap v2314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:50 smithi171 ceph-mon[37789]: pgmap v2314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:36:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3355.645594 secs 2024-07-26T09:36:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3355.645594 secs 2024-07-26T09:36:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3355.645594 secs 2024-07-26T09:36:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:52 smithi044 ceph-mon[32404]: pgmap v2315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:52 smithi116 ceph-mon[37097]: pgmap v2315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:52 smithi171 ceph-mon[37789]: pgmap v2315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:53.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:53 smithi044 ceph-mon[32404]: pgmap v2316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:53 smithi116 ceph-mon[37097]: pgmap v2316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:53 smithi171 ceph-mon[37789]: pgmap v2316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:36:56.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:56.241+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:36:56.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:56.241+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:56.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:36:56.241+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:36:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:56 smithi044 ceph-mon[32404]: pgmap v2317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3360.645699 secs 2024-07-26T09:36:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:56 smithi171 ceph-mon[37789]: pgmap v2317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3360.645699 secs 2024-07-26T09:36:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:56 smithi116 ceph-mon[37097]: pgmap v2317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:36:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3360.645699 secs 2024-07-26T09:36:58.582 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:58 smithi044 ceph-mon[32404]: pgmap v2318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:58.582 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:58.582 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.582 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.582 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.582 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:36:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:58 smithi171 ceph-mon[37789]: pgmap v2318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:36:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.615 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:58 smithi116 ceph-mon[37097]: pgmap v2318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:36:58.615 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:58.616 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.616 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.616 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:36:58.616 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:36:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:00 smithi171 ceph-mon[37789]: pgmap v2319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:00 smithi044 ceph-mon[32404]: pgmap v2319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:00 smithi116 ceph-mon[37097]: pgmap v2319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3365.645793 secs 2024-07-26T09:37:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3365.645793 secs 2024-07-26T09:37:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3365.645793 secs 2024-07-26T09:37:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:02 smithi044 ceph-mon[32404]: pgmap v2320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:02.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:02 smithi116 ceph-mon[37097]: pgmap v2320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:02 smithi171 ceph-mon[37789]: pgmap v2320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:02.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:37:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:03 smithi044 ceph-mon[32404]: pgmap v2321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:03 smithi044 ceph-mon[32404]: pgmap v2322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:37:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:03 smithi116 ceph-mon[37097]: pgmap v2321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:03 smithi116 ceph-mon[37097]: pgmap v2322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:37:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:03 smithi171 ceph-mon[37789]: pgmap v2321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:37:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:03 smithi171 ceph-mon[37789]: pgmap v2322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:37:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:05 smithi044 ceph-mon[32404]: pgmap v2323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:05 smithi116 ceph-mon[37097]: pgmap v2323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:05 smithi171 ceph-mon[37789]: pgmap v2323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:06.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3370.645890 secs 2024-07-26T09:37:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3370.645890 secs 2024-07-26T09:37:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3370.645890 secs 2024-07-26T09:37:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:07 smithi044 ceph-mon[32404]: pgmap v2324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:37:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:07 smithi116 ceph-mon[37097]: pgmap v2324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:37:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:07 smithi171 ceph-mon[37789]: pgmap v2324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:37:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:08 smithi044 ceph-mon[32404]: pgmap v2325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:08 smithi116 ceph-mon[37097]: pgmap v2325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:08 smithi171 ceph-mon[37789]: pgmap v2325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:11.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:11.244+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:37:11.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:11.244+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:37:11.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:11.244+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:11 smithi044 ceph-mon[32404]: pgmap v2326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3375.646010 secs 2024-07-26T09:37:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:11 smithi116 ceph-mon[37097]: pgmap v2326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3375.646010 secs 2024-07-26T09:37:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:11 smithi171 ceph-mon[37789]: pgmap v2326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:37:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3375.646010 secs 2024-07-26T09:37:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:13 smithi044 ceph-mon[32404]: pgmap v2327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:13 smithi116 ceph-mon[37097]: pgmap v2327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:13 smithi171 ceph-mon[37789]: pgmap v2327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:37:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:15 smithi044 ceph-mon[32404]: pgmap v2328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:15 smithi116 ceph-mon[37097]: pgmap v2328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:15 smithi171 ceph-mon[37789]: pgmap v2328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3380.646109 secs 2024-07-26T09:37:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3380.646109 secs 2024-07-26T09:37:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3380.646109 secs 2024-07-26T09:37:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:17 smithi044 ceph-mon[32404]: pgmap v2329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:17 smithi116 ceph-mon[37097]: pgmap v2329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:17 smithi171 ceph-mon[37789]: pgmap v2329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:19 smithi116 ceph-mon[37097]: pgmap v2330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:19 smithi171 ceph-mon[37789]: pgmap v2330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:19 smithi044 ceph-mon[32404]: pgmap v2330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:21 smithi171 ceph-mon[37789]: pgmap v2331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3385.646267 secs 2024-07-26T09:37:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:21 smithi044 ceph-mon[32404]: pgmap v2331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3385.646267 secs 2024-07-26T09:37:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:21 smithi116 ceph-mon[37097]: pgmap v2331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3385.646267 secs 2024-07-26T09:37:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:23 smithi171 ceph-mon[37789]: pgmap v2332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:23 smithi044 ceph-mon[32404]: pgmap v2332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:23 smithi116 ceph-mon[37097]: pgmap v2332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:25 smithi171 ceph-mon[37789]: pgmap v2333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:25 smithi044 ceph-mon[32404]: pgmap v2333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:25 smithi116 ceph-mon[37097]: pgmap v2333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:26.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:26.252+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:37:26.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:26.252+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:37:26.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:26.252+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3390.646416 secs 2024-07-26T09:37:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3390.646416 secs 2024-07-26T09:37:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3390.646416 secs 2024-07-26T09:37:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:27 smithi171 ceph-mon[37789]: pgmap v2334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:27 smithi044 ceph-mon[32404]: pgmap v2334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:27 smithi116 ceph-mon[37097]: pgmap v2334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:29 smithi171 ceph-mon[37789]: pgmap v2335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:29 smithi044 ceph-mon[32404]: pgmap v2335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:29 smithi116 ceph-mon[37097]: pgmap v2335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:31 smithi171 ceph-mon[37789]: pgmap v2336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3395.646545 secs 2024-07-26T09:37:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:31 smithi044 ceph-mon[32404]: pgmap v2336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3395.646545 secs 2024-07-26T09:37:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:31 smithi116 ceph-mon[37097]: pgmap v2336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3395.646545 secs 2024-07-26T09:37:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:33 smithi171 ceph-mon[37789]: pgmap v2337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:33 smithi044 ceph-mon[32404]: pgmap v2337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:33 smithi116 ceph-mon[37097]: pgmap v2337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:35 smithi171 ceph-mon[37789]: pgmap v2338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:35 smithi044 ceph-mon[32404]: pgmap v2338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:35 smithi116 ceph-mon[37097]: pgmap v2338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3400.646713 secs 2024-07-26T09:37:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3400.646713 secs 2024-07-26T09:37:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3400.646713 secs 2024-07-26T09:37:37.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:37 smithi044 ceph-mon[32404]: pgmap v2339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:37 smithi116 ceph-mon[37097]: pgmap v2339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:37 smithi171 ceph-mon[37789]: pgmap v2339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:39 smithi044 ceph-mon[32404]: pgmap v2340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:39 smithi116 ceph-mon[37097]: pgmap v2340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:39 smithi171 ceph-mon[37789]: pgmap v2340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:41.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:41.256+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:37:41.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:41.256+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:41.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:41.256+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:37:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:41 smithi044 ceph-mon[32404]: pgmap v2341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3405.646856 secs 2024-07-26T09:37:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:41 smithi116 ceph-mon[37097]: pgmap v2341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3405.646856 secs 2024-07-26T09:37:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:41 smithi171 ceph-mon[37789]: pgmap v2341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3405.646856 secs 2024-07-26T09:37:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:43 smithi044 ceph-mon[32404]: pgmap v2342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:43 smithi116 ceph-mon[37097]: pgmap v2342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:43 smithi171 ceph-mon[37789]: pgmap v2342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:45 smithi044 ceph-mon[32404]: pgmap v2343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:45 smithi116 ceph-mon[37097]: pgmap v2343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:45 smithi171 ceph-mon[37789]: pgmap v2343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3410.647014 secs 2024-07-26T09:37:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3410.647014 secs 2024-07-26T09:37:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3410.647014 secs 2024-07-26T09:37:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:47 smithi044 ceph-mon[32404]: pgmap v2344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:47 smithi116 ceph-mon[37097]: pgmap v2344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:47 smithi171 ceph-mon[37789]: pgmap v2344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:48 smithi044 ceph-mon[32404]: pgmap v2345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:48 smithi116 ceph-mon[37097]: pgmap v2345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:48 smithi171 ceph-mon[37789]: pgmap v2345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:51 smithi116 ceph-mon[37097]: pgmap v2346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3415.647178 secs 2024-07-26T09:37:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:51 smithi171 ceph-mon[37789]: pgmap v2346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3415.647178 secs 2024-07-26T09:37:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:51 smithi044 ceph-mon[32404]: pgmap v2346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3415.647178 secs 2024-07-26T09:37:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:53 smithi116 ceph-mon[37097]: pgmap v2347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:53 smithi171 ceph-mon[37789]: pgmap v2347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:53 smithi044 ceph-mon[32404]: pgmap v2347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:37:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:55 smithi171 ceph-mon[37789]: pgmap v2348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:55.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:55 smithi044 ceph-mon[32404]: pgmap v2348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:55 smithi116 ceph-mon[37097]: pgmap v2348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:37:56.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:56.258+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:37:56.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:56.258+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:56.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:37:56.258+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:37:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3420.647292 secs 2024-07-26T09:37:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3420.647292 secs 2024-07-26T09:37:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3420.647292 secs 2024-07-26T09:37:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:57 smithi171 ceph-mon[37789]: pgmap v2349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:57 smithi044 ceph-mon[32404]: pgmap v2349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:57 smithi116 ceph-mon[37097]: pgmap v2349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:37:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:37:59 smithi171 ceph-mon[37789]: pgmap v2350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:59.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:37:59 smithi044 ceph-mon[32404]: pgmap v2350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:37:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:37:59 smithi116 ceph-mon[37097]: pgmap v2350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:38:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:01 smithi171 ceph-mon[37789]: pgmap v2351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:01.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3425.647426 secs 2024-07-26T09:38:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:01 smithi044 ceph-mon[32404]: pgmap v2351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3425.647426 secs 2024-07-26T09:38:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:01 smithi116 ceph-mon[37097]: pgmap v2351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3425.647426 secs 2024-07-26T09:38:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:03 smithi171 ceph-mon[37789]: pgmap v2352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:03 smithi044 ceph-mon[32404]: pgmap v2352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:03 smithi116 ceph-mon[37097]: pgmap v2352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:38:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:38:04.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:04.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:04.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:38:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:05 smithi171 ceph-mon[37789]: pgmap v2353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:38:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:05 smithi171 ceph-mon[37789]: pgmap v2354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:05 smithi044 ceph-mon[32404]: pgmap v2353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:38:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:05 smithi044 ceph-mon[32404]: pgmap v2354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:05 smithi116 ceph-mon[37097]: pgmap v2353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:38:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:05 smithi116 ceph-mon[37097]: pgmap v2354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3430.647586 secs 2024-07-26T09:38:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3430.647586 secs 2024-07-26T09:38:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3430.647586 secs 2024-07-26T09:38:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:07 smithi044 ceph-mon[32404]: pgmap v2355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:07 smithi116 ceph-mon[37097]: pgmap v2355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:07 smithi171 ceph-mon[37789]: pgmap v2355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:09 smithi044 ceph-mon[32404]: pgmap v2356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:38:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:09 smithi116 ceph-mon[37097]: pgmap v2356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:38:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:09 smithi171 ceph-mon[37789]: pgmap v2356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:38:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:11.261+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:38:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:11.261+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:38:11.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:11.261+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:11 smithi044 ceph-mon[32404]: pgmap v2357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3435.647700 secs 2024-07-26T09:38:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:11 smithi116 ceph-mon[37097]: pgmap v2357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3435.647700 secs 2024-07-26T09:38:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:11 smithi171 ceph-mon[37789]: pgmap v2357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:38:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3435.647700 secs 2024-07-26T09:38:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:13 smithi044 ceph-mon[32404]: pgmap v2358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:38:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:13 smithi116 ceph-mon[37097]: pgmap v2358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:38:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:13 smithi171 ceph-mon[37789]: pgmap v2358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:38:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:15.271+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:38:15.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:15.271+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:38:15.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:15.272+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:38:15.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:15.273+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:38:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:15 smithi044 ceph-mon[32404]: pgmap v2359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:38:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:15 smithi116 ceph-mon[37097]: pgmap v2359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:38:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:15 smithi171 ceph-mon[37789]: pgmap v2359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:38:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3440.647799 secs 2024-07-26T09:38:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3440.647799 secs 2024-07-26T09:38:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3440.647799 secs 2024-07-26T09:38:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:17 smithi044 ceph-mon[32404]: pgmap v2360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:17 smithi116 ceph-mon[37097]: pgmap v2360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:17 smithi171 ceph-mon[37789]: pgmap v2360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:18 smithi044 ceph-mon[32404]: pgmap v2361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:18 smithi116 ceph-mon[37097]: pgmap v2361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:18 smithi171 ceph-mon[37789]: pgmap v2361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:20 smithi171 ceph-mon[37789]: pgmap v2362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:20 smithi044 ceph-mon[32404]: pgmap v2362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:20 smithi116 ceph-mon[37097]: pgmap v2362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3445.647955 secs 2024-07-26T09:38:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3445.647955 secs 2024-07-26T09:38:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3445.647955 secs 2024-07-26T09:38:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:22 smithi171 ceph-mon[37789]: pgmap v2363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:22 smithi044 ceph-mon[32404]: pgmap v2363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:22 smithi116 ceph-mon[37097]: pgmap v2363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:24 smithi044 ceph-mon[32404]: pgmap v2364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:24 smithi116 ceph-mon[37097]: pgmap v2364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:24 smithi171 ceph-mon[37789]: pgmap v2364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:26.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:26.268+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:38:26.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:26.268+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:38:26.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:26.268+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:26.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3450.648102 secs 2024-07-26T09:38:26.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:26 smithi044 ceph-mon[32404]: pgmap v2365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3450.648102 secs 2024-07-26T09:38:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:26 smithi116 ceph-mon[37097]: pgmap v2365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3450.648102 secs 2024-07-26T09:38:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:26 smithi171 ceph-mon[37789]: pgmap v2365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:28 smithi044 ceph-mon[32404]: pgmap v2366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:28 smithi116 ceph-mon[37097]: pgmap v2366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:28 smithi171 ceph-mon[37789]: pgmap v2366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:30 smithi044 ceph-mon[32404]: pgmap v2367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:30 smithi116 ceph-mon[37097]: pgmap v2367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:30 smithi171 ceph-mon[37789]: pgmap v2367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3455.648245 secs 2024-07-26T09:38:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3455.648245 secs 2024-07-26T09:38:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3455.648245 secs 2024-07-26T09:38:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:32 smithi044 ceph-mon[32404]: pgmap v2368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:32 smithi116 ceph-mon[37097]: pgmap v2368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:32 smithi171 ceph-mon[37789]: pgmap v2368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:34 smithi044 ceph-mon[32404]: pgmap v2369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:34 smithi116 ceph-mon[37097]: pgmap v2369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:34 smithi171 ceph-mon[37789]: pgmap v2369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:36.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3460.648344 secs 2024-07-26T09:38:36.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:36 smithi171 ceph-mon[37789]: pgmap v2370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3460.648344 secs 2024-07-26T09:38:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:36 smithi044 ceph-mon[32404]: pgmap v2370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3460.648344 secs 2024-07-26T09:38:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:36 smithi116 ceph-mon[37097]: pgmap v2370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:38 smithi044 ceph-mon[32404]: pgmap v2371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:38 smithi116 ceph-mon[37097]: pgmap v2371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:38 smithi171 ceph-mon[37789]: pgmap v2371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:40 smithi044 ceph-mon[32404]: pgmap v2372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:40 smithi116 ceph-mon[37097]: pgmap v2372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:40 smithi171 ceph-mon[37789]: pgmap v2372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:41.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:41.272+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:38:41.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:41.272+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:41.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:41.272+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:38:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3465.648484 secs 2024-07-26T09:38:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3465.648484 secs 2024-07-26T09:38:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3465.648484 secs 2024-07-26T09:38:42.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:42 smithi044 ceph-mon[32404]: pgmap v2373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:42 smithi116 ceph-mon[37097]: pgmap v2373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:42 smithi171 ceph-mon[37789]: pgmap v2373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:44 smithi044 ceph-mon[32404]: pgmap v2374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:44 smithi116 ceph-mon[37097]: pgmap v2374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:44 smithi171 ceph-mon[37789]: pgmap v2374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3470.648650 secs 2024-07-26T09:38:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:46 smithi044 ceph-mon[32404]: pgmap v2375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3470.648650 secs 2024-07-26T09:38:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:46 smithi116 ceph-mon[37097]: pgmap v2375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3470.648650 secs 2024-07-26T09:38:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:46 smithi171 ceph-mon[37789]: pgmap v2375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:48 smithi044 ceph-mon[32404]: pgmap v2376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:48 smithi116 ceph-mon[37097]: pgmap v2376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:48 smithi171 ceph-mon[37789]: pgmap v2376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:50 smithi044 ceph-mon[32404]: pgmap v2377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:50 smithi116 ceph-mon[37097]: pgmap v2377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:50 smithi171 ceph-mon[37789]: pgmap v2377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:38:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3475.648748 secs 2024-07-26T09:38:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3475.648748 secs 2024-07-26T09:38:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3475.648748 secs 2024-07-26T09:38:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:52 smithi044 ceph-mon[32404]: pgmap v2378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:52 smithi116 ceph-mon[37097]: pgmap v2378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:52 smithi171 ceph-mon[37789]: pgmap v2378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:38:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:54 smithi044 ceph-mon[32404]: pgmap v2379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:55.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:54 smithi116 ceph-mon[37097]: pgmap v2379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:54 smithi171 ceph-mon[37789]: pgmap v2379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:38:56.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:56.281+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:38:56.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:56.281+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:56.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:38:56.281+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:38:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3480.648895 secs 2024-07-26T09:38:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:56 smithi044 ceph-mon[32404]: pgmap v2380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3480.648895 secs 2024-07-26T09:38:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:56 smithi116 ceph-mon[37097]: pgmap v2380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3480.648895 secs 2024-07-26T09:38:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:56 smithi171 ceph-mon[37789]: pgmap v2380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:38:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:38:58 smithi044 ceph-mon[32404]: pgmap v2381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:38:58 smithi116 ceph-mon[37097]: pgmap v2381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:38:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:38:58 smithi171 ceph-mon[37789]: pgmap v2381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:00.281+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:39:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:00.281+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:39:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:00.281+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:39:00.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:00.281+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:39:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:00 smithi044 ceph-mon[32404]: pgmap v2382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:00 smithi116 ceph-mon[37097]: pgmap v2382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:00 smithi171 ceph-mon[37789]: pgmap v2382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3485.649041 secs 2024-07-26T09:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3485.649041 secs 2024-07-26T09:39:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3485.649041 secs 2024-07-26T09:39:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:02 smithi044 ceph-mon[32404]: pgmap v2383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:02 smithi116 ceph-mon[37097]: pgmap v2383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:02 smithi171 ceph-mon[37789]: pgmap v2383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:39:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:04 smithi044 ceph-mon[32404]: pgmap v2384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:04 smithi116 ceph-mon[37097]: pgmap v2384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:04 smithi171 ceph-mon[37789]: pgmap v2384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:39:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:05 smithi171 ceph-mon[37789]: pgmap v2385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:39:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:05 smithi171 ceph-mon[37789]: pgmap v2386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:39:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:05 smithi044 ceph-mon[32404]: pgmap v2385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:39:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:05 smithi044 ceph-mon[32404]: pgmap v2386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:39:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:05 smithi116 ceph-mon[37097]: pgmap v2385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:39:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:05 smithi116 ceph-mon[37097]: pgmap v2386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:39:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3490.649130 secs 2024-07-26T09:39:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3490.649130 secs 2024-07-26T09:39:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3490.649130 secs 2024-07-26T09:39:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:07 smithi171 ceph-mon[37789]: pgmap v2387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:07 smithi044 ceph-mon[32404]: pgmap v2387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:07 smithi116 ceph-mon[37097]: pgmap v2387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:09 smithi044 ceph-mon[32404]: pgmap v2388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:09 smithi116 ceph-mon[37097]: pgmap v2388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:09 smithi171 ceph-mon[37789]: pgmap v2388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:39:11.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:11.285+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:39:11.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:11.285+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:39:11.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:11.285+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3495.649284 secs 2024-07-26T09:39:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:11 smithi044 ceph-mon[32404]: pgmap v2389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:39:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3495.649284 secs 2024-07-26T09:39:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:11 smithi116 ceph-mon[37097]: pgmap v2389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:39:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3495.649284 secs 2024-07-26T09:39:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:11 smithi171 ceph-mon[37789]: pgmap v2389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:39:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:13 smithi044 ceph-mon[32404]: pgmap v2390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:39:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:13 smithi116 ceph-mon[37097]: pgmap v2390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:39:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:13 smithi171 ceph-mon[37789]: pgmap v2390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:39:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:15 smithi044 ceph-mon[32404]: pgmap v2391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:39:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:15 smithi116 ceph-mon[37097]: pgmap v2391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:39:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:15 smithi171 ceph-mon[37789]: pgmap v2391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:39:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3500.649435 secs 2024-07-26T09:39:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3500.649435 secs 2024-07-26T09:39:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3500.649435 secs 2024-07-26T09:39:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:17 smithi044 ceph-mon[32404]: pgmap v2392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:17 smithi116 ceph-mon[37097]: pgmap v2392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:17 smithi171 ceph-mon[37789]: pgmap v2392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:19 smithi044 ceph-mon[32404]: pgmap v2393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:19 smithi116 ceph-mon[37097]: pgmap v2393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:19 smithi171 ceph-mon[37789]: pgmap v2393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3505.649593 secs 2024-07-26T09:39:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:21 smithi044 ceph-mon[32404]: pgmap v2394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3505.649593 secs 2024-07-26T09:39:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:21 smithi116 ceph-mon[37097]: pgmap v2394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3505.649593 secs 2024-07-26T09:39:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:21 smithi171 ceph-mon[37789]: pgmap v2394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:23 smithi044 ceph-mon[32404]: pgmap v2395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:23 smithi116 ceph-mon[37097]: pgmap v2395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:23 smithi171 ceph-mon[37789]: pgmap v2395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:25 smithi044 ceph-mon[32404]: pgmap v2396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:25 smithi116 ceph-mon[37097]: pgmap v2396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:26.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:26.293+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:39:26.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:26.293+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:39:26.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:26.293+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:25 smithi171 ceph-mon[37789]: pgmap v2396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3510.649753 secs 2024-07-26T09:39:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3510.649753 secs 2024-07-26T09:39:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3510.649753 secs 2024-07-26T09:39:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:27 smithi044 ceph-mon[32404]: pgmap v2397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:27 smithi116 ceph-mon[37097]: pgmap v2397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:27 smithi171 ceph-mon[37789]: pgmap v2397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:29 smithi044 ceph-mon[32404]: pgmap v2398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:29 smithi116 ceph-mon[37097]: pgmap v2398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:29 smithi171 ceph-mon[37789]: pgmap v2398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3515.649899 secs 2024-07-26T09:39:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:31 smithi044 ceph-mon[32404]: pgmap v2399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3515.649899 secs 2024-07-26T09:39:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:31 smithi116 ceph-mon[37097]: pgmap v2399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3515.649899 secs 2024-07-26T09:39:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:31 smithi171 ceph-mon[37789]: pgmap v2399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:33 smithi044 ceph-mon[32404]: pgmap v2400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:33 smithi116 ceph-mon[37097]: pgmap v2400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:33 smithi171 ceph-mon[37789]: pgmap v2400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:35 smithi044 ceph-mon[32404]: pgmap v2401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3520.650045 secs 2024-07-26T09:39:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:35 smithi116 ceph-mon[37097]: pgmap v2401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3520.650045 secs 2024-07-26T09:39:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:35 smithi171 ceph-mon[37789]: pgmap v2401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3520.650045 secs 2024-07-26T09:39:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:37 smithi044 ceph-mon[32404]: pgmap v2402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:37 smithi116 ceph-mon[37097]: pgmap v2402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:37 smithi171 ceph-mon[37789]: pgmap v2402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:39 smithi044 ceph-mon[32404]: pgmap v2403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:39 smithi116 ceph-mon[37097]: pgmap v2403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:39 smithi171 ceph-mon[37789]: pgmap v2403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3525.650209 secs 2024-07-26T09:39:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3525.650209 secs 2024-07-26T09:39:41.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:41.297+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:39:41.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:41.297+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:39:41.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:41.298+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3525.650209 secs 2024-07-26T09:39:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:42 smithi044 ceph-mon[32404]: pgmap v2404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:42 smithi116 ceph-mon[37097]: pgmap v2404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:42 smithi171 ceph-mon[37789]: pgmap v2404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:39:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:44 smithi044 ceph-mon[32404]: pgmap v2405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:44 smithi116 ceph-mon[37097]: pgmap v2405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:44 smithi171 ceph-mon[37789]: pgmap v2405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:46 smithi044 ceph-mon[32404]: pgmap v2406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3530.650360 secs 2024-07-26T09:39:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:46 smithi116 ceph-mon[37097]: pgmap v2406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3530.650360 secs 2024-07-26T09:39:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:46 smithi171 ceph-mon[37789]: pgmap v2406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:39:46.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3530.650360 secs 2024-07-26T09:39:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:47 smithi044 ceph-mon[32404]: pgmap v2407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:47 smithi116 ceph-mon[37097]: pgmap v2407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:47 smithi171 ceph-mon[37789]: pgmap v2407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:49 smithi044 ceph-mon[32404]: pgmap v2408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:49 smithi116 ceph-mon[37097]: pgmap v2408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:49 smithi171 ceph-mon[37789]: pgmap v2408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:39:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3535.650526 secs 2024-07-26T09:39:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:51 smithi044 ceph-mon[32404]: pgmap v2409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:39:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3535.650526 secs 2024-07-26T09:39:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:51 smithi116 ceph-mon[37097]: pgmap v2409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:39:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3535.650526 secs 2024-07-26T09:39:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:51 smithi171 ceph-mon[37789]: pgmap v2409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:39:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:53 smithi044 ceph-mon[32404]: pgmap v2410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:53 smithi116 ceph-mon[37097]: pgmap v2410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:53 smithi171 ceph-mon[37789]: pgmap v2410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:55 smithi044 ceph-mon[32404]: pgmap v2411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:39:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:55 smithi116 ceph-mon[37097]: pgmap v2411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:39:56.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:56.307+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:39:56.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:56.307+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:39:56.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:39:56.307+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:55 smithi171 ceph-mon[37789]: pgmap v2411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:39:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3540.650694 secs 2024-07-26T09:39:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3540.650694 secs 2024-07-26T09:39:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3540.650694 secs 2024-07-26T09:39:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:57 smithi044 ceph-mon[32404]: pgmap v2412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:57 smithi116 ceph-mon[37097]: pgmap v2412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:39:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:57 smithi171 ceph-mon[37789]: pgmap v2412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:39:59 smithi044 ceph-mon[32404]: pgmap v2413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:39:59 smithi116 ceph-mon[37097]: pgmap v2413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:39:59 smithi171 ceph-mon[37789]: pgmap v2413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:40:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:40:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:40:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3545.650857 secs 2024-07-26T09:40:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:01 smithi044 ceph-mon[32404]: pgmap v2414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:40:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3545.650857 secs 2024-07-26T09:40:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:01 smithi116 ceph-mon[37097]: pgmap v2414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:40:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3545.650857 secs 2024-07-26T09:40:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:01 smithi171 ceph-mon[37789]: pgmap v2414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:40:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:03 smithi044 ceph-mon[32404]: pgmap v2415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:03 smithi116 ceph-mon[37097]: pgmap v2415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:03 smithi171 ceph-mon[37789]: pgmap v2415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:40:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:05 smithi044 ceph-mon[32404]: pgmap v2416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:05 smithi116 ceph-mon[37097]: pgmap v2416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:05 smithi171 ceph-mon[37789]: pgmap v2416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:40:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3550.650993 secs 2024-07-26T09:40:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:06 smithi044 ceph-mon[32404]: pgmap v2417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:40:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:06 smithi044 ceph-mon[32404]: pgmap v2418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:40:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3550.650993 secs 2024-07-26T09:40:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:06 smithi116 ceph-mon[37097]: pgmap v2417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:40:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:06 smithi116 ceph-mon[37097]: pgmap v2418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:40:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3550.650993 secs 2024-07-26T09:40:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:07.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:06 smithi171 ceph-mon[37789]: pgmap v2417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:40:07.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:06 smithi171 ceph-mon[37789]: pgmap v2418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:07.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:40:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:09 smithi044 ceph-mon[32404]: pgmap v2419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:09 smithi116 ceph-mon[37097]: pgmap v2419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:09 smithi171 ceph-mon[37789]: pgmap v2419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:11.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:11.310+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:40:11.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:11.310+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:11.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:11.310+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:40:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:11 smithi044 ceph-mon[32404]: pgmap v2420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3555.651120 secs 2024-07-26T09:40:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:11 smithi116 ceph-mon[37097]: pgmap v2420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3555.651120 secs 2024-07-26T09:40:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:11 smithi171 ceph-mon[37789]: pgmap v2420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3555.651120 secs 2024-07-26T09:40:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:13 smithi044 ceph-mon[32404]: pgmap v2421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:13 smithi116 ceph-mon[37097]: pgmap v2421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:13 smithi171 ceph-mon[37789]: pgmap v2421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:15 smithi044 ceph-mon[32404]: pgmap v2422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:15 smithi116 ceph-mon[37097]: pgmap v2422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:15 smithi171 ceph-mon[37789]: pgmap v2422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:40:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3560.651228 secs 2024-07-26T09:40:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3560.651228 secs 2024-07-26T09:40:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3560.651228 secs 2024-07-26T09:40:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:17 smithi044 ceph-mon[32404]: pgmap v2423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:40:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:17 smithi116 ceph-mon[37097]: pgmap v2423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:40:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:17 smithi171 ceph-mon[37789]: pgmap v2423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:40:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:19 smithi044 ceph-mon[32404]: pgmap v2424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:19 smithi116 ceph-mon[37097]: pgmap v2424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:19 smithi171 ceph-mon[37789]: pgmap v2424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:21 smithi044 ceph-mon[32404]: pgmap v2425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3565.651390 secs 2024-07-26T09:40:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:21 smithi116 ceph-mon[37097]: pgmap v2425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3565.651390 secs 2024-07-26T09:40:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:21 smithi171 ceph-mon[37789]: pgmap v2425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3565.651390 secs 2024-07-26T09:40:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:23 smithi044 ceph-mon[32404]: pgmap v2426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:23 smithi116 ceph-mon[37097]: pgmap v2426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:23 smithi171 ceph-mon[37789]: pgmap v2426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:25 smithi044 ceph-mon[32404]: pgmap v2427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:25 smithi116 ceph-mon[37097]: pgmap v2427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:25 smithi171 ceph-mon[37789]: pgmap v2427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:26.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:26.318+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:40:26.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:26.318+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:26.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:26.318+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:40:26.438 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3570.651554 secs 2024-07-26T09:40:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3570.651554 secs 2024-07-26T09:40:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3570.651554 secs 2024-07-26T09:40:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:27 smithi044 ceph-mon[32404]: pgmap v2428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:27 smithi116 ceph-mon[37097]: pgmap v2428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:27 smithi171 ceph-mon[37789]: pgmap v2428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:29 smithi044 ceph-mon[32404]: pgmap v2429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:29 smithi116 ceph-mon[37097]: pgmap v2429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:29 smithi171 ceph-mon[37789]: pgmap v2429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:31 smithi044 ceph-mon[32404]: pgmap v2430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3575.651693 secs 2024-07-26T09:40:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:31 smithi116 ceph-mon[37097]: pgmap v2430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3575.651693 secs 2024-07-26T09:40:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:31 smithi171 ceph-mon[37789]: pgmap v2430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3575.651693 secs 2024-07-26T09:40:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:33 smithi044 ceph-mon[32404]: pgmap v2431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:33 smithi116 ceph-mon[37097]: pgmap v2431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:33 smithi171 ceph-mon[37789]: pgmap v2431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:35 smithi044 ceph-mon[32404]: pgmap v2432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:35 smithi116 ceph-mon[37097]: pgmap v2432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:35 smithi171 ceph-mon[37789]: pgmap v2432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:36.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3580.651858 secs 2024-07-26T09:40:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3580.651858 secs 2024-07-26T09:40:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3580.651858 secs 2024-07-26T09:40:37.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:37 smithi171 ceph-mon[37789]: pgmap v2433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:37 smithi044 ceph-mon[32404]: pgmap v2433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:37 smithi116 ceph-mon[37097]: pgmap v2433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:39 smithi044 ceph-mon[32404]: pgmap v2434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:39 smithi116 ceph-mon[37097]: pgmap v2434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:39.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:39 smithi171 ceph-mon[37789]: pgmap v2434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:41.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:41.320+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:40:41.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:41.320+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:40:41.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:41.320+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:41 smithi044 ceph-mon[32404]: pgmap v2435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3585.651961 secs 2024-07-26T09:40:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:41 smithi116 ceph-mon[37097]: pgmap v2435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3585.651961 secs 2024-07-26T09:40:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:41 smithi171 ceph-mon[37789]: pgmap v2435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3585.651961 secs 2024-07-26T09:40:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:43 smithi044 ceph-mon[32404]: pgmap v2436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:43 smithi116 ceph-mon[37097]: pgmap v2436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:43 smithi171 ceph-mon[37789]: pgmap v2436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:45 smithi044 ceph-mon[32404]: pgmap v2437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:45 smithi116 ceph-mon[37097]: pgmap v2437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:45 smithi171 ceph-mon[37789]: pgmap v2437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3590.652104 secs 2024-07-26T09:40:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3590.652104 secs 2024-07-26T09:40:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3590.652104 secs 2024-07-26T09:40:47.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:47 smithi044 ceph-mon[32404]: pgmap v2438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:47 smithi116 ceph-mon[37097]: pgmap v2438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:47 smithi171 ceph-mon[37789]: pgmap v2438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:49 smithi171 ceph-mon[37789]: pgmap v2439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:49 smithi044 ceph-mon[32404]: pgmap v2439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:49 smithi116 ceph-mon[37097]: pgmap v2439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:51 smithi171 ceph-mon[37789]: pgmap v2440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3595.652270 secs 2024-07-26T09:40:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:51 smithi044 ceph-mon[32404]: pgmap v2440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3595.652270 secs 2024-07-26T09:40:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:51 smithi116 ceph-mon[37097]: pgmap v2440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3595.652270 secs 2024-07-26T09:40:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:53 smithi171 ceph-mon[37789]: pgmap v2441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:53 smithi044 ceph-mon[32404]: pgmap v2441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:53 smithi116 ceph-mon[37097]: pgmap v2441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:55 smithi171 ceph-mon[37789]: pgmap v2442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:55 smithi044 ceph-mon[32404]: pgmap v2442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:55 smithi116 ceph-mon[37097]: pgmap v2442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:40:56.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:56.328+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:40:56.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:56.328+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:40:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:40:56.328+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3600.652413 secs 2024-07-26T09:40:56.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3600.652413 secs 2024-07-26T09:40:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3600.652413 secs 2024-07-26T09:40:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:57 smithi171 ceph-mon[37789]: pgmap v2443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:57 smithi044 ceph-mon[32404]: pgmap v2443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:57 smithi116 ceph-mon[37097]: pgmap v2443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:40:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:40:59 smithi171 ceph-mon[37789]: pgmap v2444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:40:59 smithi044 ceph-mon[32404]: pgmap v2444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:40:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:40:59 smithi116 ceph-mon[37097]: pgmap v2444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:01 smithi171 ceph-mon[37789]: pgmap v2445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3605.652571 secs 2024-07-26T09:41:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:01 smithi044 ceph-mon[32404]: pgmap v2445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3605.652571 secs 2024-07-26T09:41:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:01 smithi116 ceph-mon[37097]: pgmap v2445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3605.652571 secs 2024-07-26T09:41:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:03 smithi171 ceph-mon[37789]: pgmap v2446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:03 smithi044 ceph-mon[32404]: pgmap v2446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:03 smithi116 ceph-mon[37097]: pgmap v2446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:05 smithi171 ceph-mon[37789]: pgmap v2447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:05 smithi044 ceph-mon[32404]: pgmap v2447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:05 smithi116 ceph-mon[37097]: pgmap v2447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3610.652715 secs 2024-07-26T09:41:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3610.652715 secs 2024-07-26T09:41:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3610.652715 secs 2024-07-26T09:41:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:07 smithi044 ceph-mon[32404]: pgmap v2448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:41:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:41:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:41:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:07 smithi116 ceph-mon[37097]: pgmap v2448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:41:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:41:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:41:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:07 smithi171 ceph-mon[37789]: pgmap v2448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:41:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:41:07.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:41:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:08 smithi044 ceph-mon[32404]: pgmap v2449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:41:08.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:08 smithi044 ceph-mon[32404]: pgmap v2450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:08 smithi116 ceph-mon[37097]: pgmap v2449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:41:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:08 smithi116 ceph-mon[37097]: pgmap v2450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:08 smithi171 ceph-mon[37789]: pgmap v2449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:41:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:08 smithi171 ceph-mon[37789]: pgmap v2450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:10 smithi044 ceph-mon[32404]: pgmap v2451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:10 smithi116 ceph-mon[37097]: pgmap v2451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:10 smithi171 ceph-mon[37789]: pgmap v2451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:11.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:11.332+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:41:11.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:11.332+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:41:11.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:11.332+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3615.652861 secs 2024-07-26T09:41:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3615.652861 secs 2024-07-26T09:41:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3615.652861 secs 2024-07-26T09:41:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:12 smithi044 ceph-mon[32404]: pgmap v2452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:41:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:12 smithi116 ceph-mon[37097]: pgmap v2452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:41:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:12 smithi171 ceph-mon[37789]: pgmap v2452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:41:14.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T09:41:14.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:41:14.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf5804e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:41:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf5804e000 comp rx=0 tx=0).stop 2024-07-26T09:41:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T09:41:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:41:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf4b0 0x7fcf540c18d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:41:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.689+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:41:14.693 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:14 smithi044 ceph-mon[32404]: pgmap v2453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:14.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.690+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:41:14.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:14.693+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b4730 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcf5805c000 tx=0x7fcf58052040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:41:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:14 smithi116 ceph-mon[37097]: pgmap v2453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:14 smithi171 ceph-mon[37789]: pgmap v2453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:15.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:15.305+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:41:15.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:15.305+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:41:15.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:15.306+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:41:15.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:15.307+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:41:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3620.652996 secs 2024-07-26T09:41:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:16 smithi044 ceph-mon[32404]: pgmap v2454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3620.652996 secs 2024-07-26T09:41:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:16 smithi116 ceph-mon[37097]: pgmap v2454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3620.652996 secs 2024-07-26T09:41:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:16 smithi171 ceph-mon[37789]: pgmap v2454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:41:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:18 smithi044 ceph-mon[32404]: pgmap v2455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:41:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:18 smithi116 ceph-mon[37097]: pgmap v2455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:41:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:18 smithi171 ceph-mon[37789]: pgmap v2455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:41:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:20 smithi044 ceph-mon[32404]: pgmap v2456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:20 smithi116 ceph-mon[37097]: pgmap v2456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:20 smithi171 ceph-mon[37789]: pgmap v2456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3625.653157 secs 2024-07-26T09:41:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3625.653157 secs 2024-07-26T09:41:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3625.653157 secs 2024-07-26T09:41:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:22 smithi044 ceph-mon[32404]: pgmap v2457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:22 smithi116 ceph-mon[37097]: pgmap v2457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:22 smithi171 ceph-mon[37789]: pgmap v2457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:24 smithi044 ceph-mon[32404]: pgmap v2458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:24 smithi116 ceph-mon[37097]: pgmap v2458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:24 smithi171 ceph-mon[37789]: pgmap v2458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:26.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:26.341+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:41:26.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:26.341+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:41:26.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:26.341+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3630.653318 secs 2024-07-26T09:41:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:26 smithi044 ceph-mon[32404]: pgmap v2459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3630.653318 secs 2024-07-26T09:41:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:26 smithi116 ceph-mon[37097]: pgmap v2459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3630.653318 secs 2024-07-26T09:41:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:26 smithi171 ceph-mon[37789]: pgmap v2459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:27 smithi044 ceph-mon[32404]: pgmap v2460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:27 smithi116 ceph-mon[37097]: pgmap v2460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:27 smithi171 ceph-mon[37789]: pgmap v2460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:30 smithi044 ceph-mon[32404]: pgmap v2461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:30 smithi116 ceph-mon[37097]: pgmap v2461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:30 smithi171 ceph-mon[37789]: pgmap v2461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3635.653475 secs 2024-07-26T09:41:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3635.653475 secs 2024-07-26T09:41:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3635.653475 secs 2024-07-26T09:41:32.706 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:32 smithi044 ceph-mon[32404]: pgmap v2462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:32 smithi116 ceph-mon[37097]: pgmap v2462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:32 smithi171 ceph-mon[37789]: pgmap v2462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:34 smithi044 ceph-mon[32404]: pgmap v2463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:34 smithi116 ceph-mon[37097]: pgmap v2463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:34 smithi171 ceph-mon[37789]: pgmap v2463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3640.653679 secs 2024-07-26T09:41:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:36 smithi044 ceph-mon[32404]: pgmap v2464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3640.653679 secs 2024-07-26T09:41:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:36 smithi116 ceph-mon[37097]: pgmap v2464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3640.653679 secs 2024-07-26T09:41:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:36 smithi171 ceph-mon[37789]: pgmap v2464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:38.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:38 smithi044 ceph-mon[32404]: pgmap v2465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:38.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:38 smithi116 ceph-mon[37097]: pgmap v2465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:38 smithi171 ceph-mon[37789]: pgmap v2465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:40 smithi044 ceph-mon[32404]: pgmap v2466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:40 smithi116 ceph-mon[37097]: pgmap v2466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:40 smithi171 ceph-mon[37789]: pgmap v2466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:41.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:41.345+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:41:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:41.345+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:41:41.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:41.345+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3645.653783 secs 2024-07-26T09:41:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3645.653783 secs 2024-07-26T09:41:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3645.653783 secs 2024-07-26T09:41:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:42 smithi044 ceph-mon[32404]: pgmap v2467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:42 smithi116 ceph-mon[37097]: pgmap v2467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:42 smithi171 ceph-mon[37789]: pgmap v2467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:43 smithi044 ceph-mon[32404]: pgmap v2468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:43 smithi116 ceph-mon[37097]: pgmap v2468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:43 smithi171 ceph-mon[37789]: pgmap v2468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3650.653923 secs 2024-07-26T09:41:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:46 smithi044 ceph-mon[32404]: pgmap v2469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3650.653923 secs 2024-07-26T09:41:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:46 smithi116 ceph-mon[37097]: pgmap v2469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3650.653923 secs 2024-07-26T09:41:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:46 smithi171 ceph-mon[37789]: pgmap v2469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:48 smithi044 ceph-mon[32404]: pgmap v2470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:48 smithi116 ceph-mon[37097]: pgmap v2470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:48 smithi171 ceph-mon[37789]: pgmap v2470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:41:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:50 smithi044 ceph-mon[32404]: pgmap v2471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:50 smithi116 ceph-mon[37097]: pgmap v2471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:50 smithi171 ceph-mon[37789]: pgmap v2471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:41:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3655.654048 secs 2024-07-26T09:41:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3655.654048 secs 2024-07-26T09:41:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3655.654048 secs 2024-07-26T09:41:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:52 smithi044 ceph-mon[32404]: pgmap v2472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:41:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:52 smithi116 ceph-mon[37097]: pgmap v2472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:41:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:52 smithi171 ceph-mon[37789]: pgmap v2472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:41:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:54 smithi044 ceph-mon[32404]: pgmap v2473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:54 smithi116 ceph-mon[37097]: pgmap v2473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:54 smithi171 ceph-mon[37789]: pgmap v2473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:41:56.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:56.353+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:41:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:56.353+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:41:56.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:41:56.353+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3660.654207 secs 2024-07-26T09:41:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:56 smithi044 ceph-mon[32404]: pgmap v2474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3660.654207 secs 2024-07-26T09:41:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:56 smithi116 ceph-mon[37097]: pgmap v2474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3660.654207 secs 2024-07-26T09:41:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:56 smithi171 ceph-mon[37789]: pgmap v2474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:41:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:58 smithi044 ceph-mon[32404]: pgmap v2475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:41:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:58 smithi116 ceph-mon[37097]: pgmap v2475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:41:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:58 smithi171 ceph-mon[37789]: pgmap v2475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:41:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:41:59 smithi044 ceph-mon[32404]: pgmap v2476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:41:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:41:59 smithi116 ceph-mon[37097]: pgmap v2476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:41:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:41:59 smithi171 ceph-mon[37789]: pgmap v2476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:42:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3665.654325 secs 2024-07-26T09:42:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3665.654325 secs 2024-07-26T09:42:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3665.654325 secs 2024-07-26T09:42:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:02 smithi171 ceph-mon[37789]: pgmap v2477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:42:02.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:02 smithi044 ceph-mon[32404]: pgmap v2477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:42:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:02 smithi116 ceph-mon[37097]: pgmap v2477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:42:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:04 smithi044 ceph-mon[32404]: pgmap v2478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:42:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:04 smithi116 ceph-mon[37097]: pgmap v2478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:42:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:04 smithi171 ceph-mon[37789]: pgmap v2478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:42:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3670.654457 secs 2024-07-26T09:42:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:06 smithi044 ceph-mon[32404]: pgmap v2479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:42:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3670.654457 secs 2024-07-26T09:42:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:06 smithi116 ceph-mon[37097]: pgmap v2479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:42:06.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3670.654457 secs 2024-07-26T09:42:06.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:06 smithi171 ceph-mon[37789]: pgmap v2479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:42:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:08 smithi044 ceph-mon[32404]: pgmap v2480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:08.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:42:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:08 smithi116 ceph-mon[37097]: pgmap v2480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:42:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:08 smithi171 ceph-mon[37789]: pgmap v2480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:42:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:42:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:42:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:42:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:42:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:10 smithi044 ceph-mon[32404]: pgmap v2481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:10 smithi116 ceph-mon[37097]: pgmap v2481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:10 smithi171 ceph-mon[37789]: pgmap v2481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:11.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:11.357+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:42:11.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:11.357+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:42:11.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:11.357+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3675.654586 secs 2024-07-26T09:42:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3675.654586 secs 2024-07-26T09:42:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3675.654586 secs 2024-07-26T09:42:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:12 smithi044 ceph-mon[32404]: pgmap v2482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:42:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:12 smithi116 ceph-mon[37097]: pgmap v2482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:42:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:12 smithi171 ceph-mon[37789]: pgmap v2482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:42:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:13 smithi044 ceph-mon[32404]: pgmap v2483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:13 smithi116 ceph-mon[37097]: pgmap v2483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:13 smithi171 ceph-mon[37789]: pgmap v2483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:15 smithi044 ceph-mon[32404]: pgmap v2484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:42:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:15 smithi116 ceph-mon[37097]: pgmap v2484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:42:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:15 smithi171 ceph-mon[37789]: pgmap v2484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:42:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3680.654667 secs 2024-07-26T09:42:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3680.654667 secs 2024-07-26T09:42:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3680.654667 secs 2024-07-26T09:42:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:17 smithi044 ceph-mon[32404]: pgmap v2485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:42:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:17 smithi116 ceph-mon[37097]: pgmap v2485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:42:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:17 smithi171 ceph-mon[37789]: pgmap v2485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:42:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:19 smithi044 ceph-mon[32404]: pgmap v2486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:19 smithi116 ceph-mon[37097]: pgmap v2486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:19 smithi171 ceph-mon[37789]: pgmap v2486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T09:42:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3685.654764 secs 2024-07-26T09:42:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:21 smithi044 ceph-mon[32404]: pgmap v2487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3685.654764 secs 2024-07-26T09:42:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:21 smithi116 ceph-mon[37097]: pgmap v2487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3685.654764 secs 2024-07-26T09:42:22.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:21 smithi171 ceph-mon[37789]: pgmap v2487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:23 smithi044 ceph-mon[32404]: pgmap v2488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:23 smithi116 ceph-mon[37097]: pgmap v2488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:23 smithi171 ceph-mon[37789]: pgmap v2488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:25 smithi044 ceph-mon[32404]: pgmap v2489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:25 smithi116 ceph-mon[37097]: pgmap v2489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:25 smithi171 ceph-mon[37789]: pgmap v2489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:26.366+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:42:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:26.366+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:42:26.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:26.366+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3690.654857 secs 2024-07-26T09:42:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3690.654857 secs 2024-07-26T09:42:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3690.654857 secs 2024-07-26T09:42:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:27 smithi044 ceph-mon[32404]: pgmap v2490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:27 smithi116 ceph-mon[37097]: pgmap v2490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:27 smithi171 ceph-mon[37789]: pgmap v2490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:29 smithi044 ceph-mon[32404]: pgmap v2491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:29 smithi116 ceph-mon[37097]: pgmap v2491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:29 smithi171 ceph-mon[37789]: pgmap v2491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:32.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3695.654998 secs 2024-07-26T09:42:32.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:31 smithi044 ceph-mon[32404]: pgmap v2492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3695.654998 secs 2024-07-26T09:42:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:31 smithi116 ceph-mon[37097]: pgmap v2492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3695.654998 secs 2024-07-26T09:42:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:31 smithi171 ceph-mon[37789]: pgmap v2492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:33 smithi044 ceph-mon[32404]: pgmap v2493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:33 smithi116 ceph-mon[37097]: pgmap v2493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:33 smithi171 ceph-mon[37789]: pgmap v2493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:35 smithi044 ceph-mon[32404]: pgmap v2494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:35 smithi116 ceph-mon[37097]: pgmap v2494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:35 smithi171 ceph-mon[37789]: pgmap v2494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:36.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3700.655134 secs 2024-07-26T09:42:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3700.655134 secs 2024-07-26T09:42:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3700.655134 secs 2024-07-26T09:42:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:37 smithi044 ceph-mon[32404]: pgmap v2495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:37 smithi116 ceph-mon[37097]: pgmap v2495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:37 smithi171 ceph-mon[37789]: pgmap v2495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:39 smithi044 ceph-mon[32404]: pgmap v2496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:39 smithi116 ceph-mon[37097]: pgmap v2496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:39 smithi171 ceph-mon[37789]: pgmap v2496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:41.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:41.369+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:42:41.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:41.369+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:42:41.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:41.369+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3705.655284 secs 2024-07-26T09:42:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:41 smithi044 ceph-mon[32404]: pgmap v2497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3705.655284 secs 2024-07-26T09:42:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:41 smithi116 ceph-mon[37097]: pgmap v2497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3705.655284 secs 2024-07-26T09:42:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:41 smithi171 ceph-mon[37789]: pgmap v2497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:43 smithi044 ceph-mon[32404]: pgmap v2498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:43 smithi116 ceph-mon[37097]: pgmap v2498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:43 smithi171 ceph-mon[37789]: pgmap v2498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:45 smithi044 ceph-mon[32404]: pgmap v2499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:45 smithi116 ceph-mon[37097]: pgmap v2499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:45 smithi171 ceph-mon[37789]: pgmap v2499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3710.655436 secs 2024-07-26T09:42:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3710.655436 secs 2024-07-26T09:42:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3710.655436 secs 2024-07-26T09:42:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:47 smithi171 ceph-mon[37789]: pgmap v2500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:47 smithi044 ceph-mon[32404]: pgmap v2500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:47 smithi116 ceph-mon[37097]: pgmap v2500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:50.070 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:49 smithi044 ceph-mon[32404]: pgmap v2501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:49 smithi171 ceph-mon[37789]: pgmap v2501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:49 smithi116 ceph-mon[37097]: pgmap v2501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:42:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3715.655596 secs 2024-07-26T09:42:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:51 smithi171 ceph-mon[37789]: pgmap v2502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3715.655596 secs 2024-07-26T09:42:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:51 smithi044 ceph-mon[32404]: pgmap v2502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3715.655596 secs 2024-07-26T09:42:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:51 smithi116 ceph-mon[37097]: pgmap v2502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:42:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:53 smithi171 ceph-mon[37789]: pgmap v2503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:53 smithi044 ceph-mon[32404]: pgmap v2503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:53 smithi116 ceph-mon[37097]: pgmap v2503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:42:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:55 smithi171 ceph-mon[37789]: pgmap v2504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:55 smithi044 ceph-mon[32404]: pgmap v2504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:55 smithi116 ceph-mon[37097]: pgmap v2504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:42:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:56.375+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:42:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:56.375+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:42:56.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:42:56.375+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3720.655761 secs 2024-07-26T09:42:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3720.655761 secs 2024-07-26T09:42:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3720.655761 secs 2024-07-26T09:42:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:57 smithi044 ceph-mon[32404]: pgmap v2505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:57 smithi116 ceph-mon[37097]: pgmap v2505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:42:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:57 smithi171 ceph-mon[37789]: pgmap v2505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:00.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:42:59 smithi044 ceph-mon[32404]: pgmap v2506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:42:59 smithi116 ceph-mon[37097]: pgmap v2506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:42:59 smithi171 ceph-mon[37789]: pgmap v2506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3725.655922 secs 2024-07-26T09:43:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:01 smithi044 ceph-mon[32404]: pgmap v2507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3725.655922 secs 2024-07-26T09:43:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:01 smithi116 ceph-mon[37097]: pgmap v2507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3725.655922 secs 2024-07-26T09:43:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:01 smithi171 ceph-mon[37789]: pgmap v2507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:03 smithi044 ceph-mon[32404]: pgmap v2508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:03 smithi116 ceph-mon[37097]: pgmap v2508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:03 smithi171 ceph-mon[37789]: pgmap v2508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:06.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:05 smithi171 ceph-mon[37789]: pgmap v2509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:05 smithi044 ceph-mon[32404]: pgmap v2509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:05 smithi116 ceph-mon[37097]: pgmap v2509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3730.656080 secs 2024-07-26T09:43:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3730.656080 secs 2024-07-26T09:43:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3730.656080 secs 2024-07-26T09:43:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:07 smithi044 ceph-mon[32404]: pgmap v2510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:07 smithi116 ceph-mon[37097]: pgmap v2510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:07 smithi171 ceph-mon[37789]: pgmap v2510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:43:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:09.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: pgmap v2511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:43:10.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:43:10.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: pgmap v2511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:43:10.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:43:10.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:43:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: pgmap v2511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:43:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:43:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:43:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:10 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:43:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:10 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:43:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:10 smithi044 ceph-mon[32404]: pgmap v2512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:43:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:10 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:43:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:10 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:43:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:10 smithi116 ceph-mon[37097]: pgmap v2512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:43:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:10 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:43:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:10 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:43:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:10 smithi171 ceph-mon[37789]: pgmap v2512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:43:11.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:11.379+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:43:11.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:11.379+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:43:11.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:11.379+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3735.656224 secs 2024-07-26T09:43:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3735.656224 secs 2024-07-26T09:43:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3735.656224 secs 2024-07-26T09:43:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:12 smithi044 ceph-mon[32404]: pgmap v2513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:43:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:12 smithi116 ceph-mon[37097]: pgmap v2513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:43:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:12 smithi171 ceph-mon[37789]: pgmap v2513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:43:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:14 smithi044 ceph-mon[32404]: pgmap v2514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:43:15.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:14 smithi116 ceph-mon[37097]: pgmap v2514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:43:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:14 smithi171 ceph-mon[37789]: pgmap v2514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:43:17.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3740.656348 secs 2024-07-26T09:43:17.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:16 smithi044 ceph-mon[32404]: pgmap v2515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:43:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3740.656348 secs 2024-07-26T09:43:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:16 smithi116 ceph-mon[37097]: pgmap v2515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:43:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3740.656348 secs 2024-07-26T09:43:17.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:16 smithi171 ceph-mon[37789]: pgmap v2515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:43:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:18 smithi044 ceph-mon[32404]: pgmap v2516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:43:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:18 smithi116 ceph-mon[37097]: pgmap v2516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:43:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:18 smithi171 ceph-mon[37789]: pgmap v2516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:43:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:20 smithi044 ceph-mon[32404]: pgmap v2517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:43:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:20 smithi116 ceph-mon[37097]: pgmap v2517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:43:21.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:20 smithi171 ceph-mon[37789]: pgmap v2517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:43:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3745.656478 secs 2024-07-26T09:43:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3745.656478 secs 2024-07-26T09:43:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3745.656478 secs 2024-07-26T09:43:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:22 smithi044 ceph-mon[32404]: pgmap v2518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:22 smithi116 ceph-mon[37097]: pgmap v2518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:22 smithi171 ceph-mon[37789]: pgmap v2518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:24 smithi044 ceph-mon[32404]: pgmap v2519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:25.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:24 smithi116 ceph-mon[37097]: pgmap v2519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:24 smithi171 ceph-mon[37789]: pgmap v2519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:43:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:26.388+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:43:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:26.388+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:43:26.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:26.388+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3750.656661 secs 2024-07-26T09:43:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:26 smithi044 ceph-mon[32404]: pgmap v2520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3750.656661 secs 2024-07-26T09:43:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:26 smithi116 ceph-mon[37097]: pgmap v2520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3750.656661 secs 2024-07-26T09:43:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:26 smithi171 ceph-mon[37789]: pgmap v2520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:28 smithi044 ceph-mon[32404]: pgmap v2521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:28 smithi116 ceph-mon[37097]: pgmap v2521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:28 smithi171 ceph-mon[37789]: pgmap v2521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:30 smithi044 ceph-mon[32404]: pgmap v2522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:31 smithi116 ceph-mon[37097]: pgmap v2522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:31 smithi171 ceph-mon[37789]: pgmap v2522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3755.656758 secs 2024-07-26T09:43:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3755.656758 secs 2024-07-26T09:43:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3755.656758 secs 2024-07-26T09:43:33.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:33 smithi044 ceph-mon[32404]: pgmap v2523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:33.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:33 smithi116 ceph-mon[37097]: pgmap v2523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:33 smithi171 ceph-mon[37789]: pgmap v2523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:35 smithi116 ceph-mon[37097]: pgmap v2524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:35 smithi171 ceph-mon[37789]: pgmap v2524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:35 smithi044 ceph-mon[32404]: pgmap v2524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3760.656852 secs 2024-07-26T09:43:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3760.656852 secs 2024-07-26T09:43:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3760.656852 secs 2024-07-26T09:43:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:37 smithi171 ceph-mon[37789]: pgmap v2525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:37 smithi044 ceph-mon[32404]: pgmap v2525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:37 smithi116 ceph-mon[37097]: pgmap v2525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:39 smithi171 ceph-mon[37789]: pgmap v2526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:39 smithi044 ceph-mon[32404]: pgmap v2526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:39 smithi116 ceph-mon[37097]: pgmap v2526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:41 smithi171 ceph-mon[37789]: pgmap v2527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:41.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3765.656948 secs 2024-07-26T09:43:41.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:41.390+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:43:41.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:41.390+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:41.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:41.390+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:43:41.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:41 smithi044 ceph-mon[32404]: pgmap v2527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:41.395 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3765.656948 secs 2024-07-26T09:43:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:41 smithi116 ceph-mon[37097]: pgmap v2527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3765.656948 secs 2024-07-26T09:43:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:43 smithi171 ceph-mon[37789]: pgmap v2528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:43 smithi044 ceph-mon[32404]: pgmap v2528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:43 smithi116 ceph-mon[37097]: pgmap v2528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:45 smithi171 ceph-mon[37789]: pgmap v2529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:45 smithi044 ceph-mon[32404]: pgmap v2529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:45 smithi116 ceph-mon[37097]: pgmap v2529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3770.657077 secs 2024-07-26T09:43:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3770.657077 secs 2024-07-26T09:43:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3770.657077 secs 2024-07-26T09:43:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:47 smithi171 ceph-mon[37789]: pgmap v2530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:47 smithi044 ceph-mon[32404]: pgmap v2530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:47 smithi116 ceph-mon[37097]: pgmap v2530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:49 smithi171 ceph-mon[37789]: pgmap v2531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:49 smithi044 ceph-mon[32404]: pgmap v2531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:49 smithi116 ceph-mon[37097]: pgmap v2531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:51 smithi171 ceph-mon[37789]: pgmap v2532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3775.657197 secs 2024-07-26T09:43:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:51 smithi044 ceph-mon[32404]: pgmap v2532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3775.657197 secs 2024-07-26T09:43:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:51 smithi116 ceph-mon[37097]: pgmap v2532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:43:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3775.657197 secs 2024-07-26T09:43:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:53 smithi044 ceph-mon[32404]: pgmap v2533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:53 smithi116 ceph-mon[37097]: pgmap v2533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:53 smithi171 ceph-mon[37789]: pgmap v2533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:55 smithi044 ceph-mon[32404]: pgmap v2534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:55 smithi116 ceph-mon[37097]: pgmap v2534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:55 smithi171 ceph-mon[37789]: pgmap v2534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:56.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:56.400+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:43:56.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:56.400+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:56.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:43:56.400+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:43:56.403 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3780.657340 secs 2024-07-26T09:43:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3780.657340 secs 2024-07-26T09:43:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3780.657340 secs 2024-07-26T09:43:57.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:57 smithi116 ceph-mon[37097]: pgmap v2535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:57 smithi044 ceph-mon[32404]: pgmap v2535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:57 smithi171 ceph-mon[37789]: pgmap v2535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:43:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:43:59 smithi044 ceph-mon[32404]: pgmap v2536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:43:59 smithi116 ceph-mon[37097]: pgmap v2536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:43:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:43:59 smithi171 ceph-mon[37789]: pgmap v2536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:01 smithi044 ceph-mon[32404]: pgmap v2537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3785.657462 secs 2024-07-26T09:44:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:01 smithi116 ceph-mon[37097]: pgmap v2537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3785.657462 secs 2024-07-26T09:44:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:01 smithi171 ceph-mon[37789]: pgmap v2537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:01.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3785.657462 secs 2024-07-26T09:44:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:03 smithi044 ceph-mon[32404]: pgmap v2538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:03 smithi116 ceph-mon[37097]: pgmap v2538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:03 smithi171 ceph-mon[37789]: pgmap v2538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:05 smithi044 ceph-mon[32404]: pgmap v2539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:05 smithi116 ceph-mon[37097]: pgmap v2539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:05 smithi171 ceph-mon[37789]: pgmap v2539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:06.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3790.657579 secs 2024-07-26T09:44:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3790.657579 secs 2024-07-26T09:44:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3790.657579 secs 2024-07-26T09:44:07.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:07 smithi044 ceph-mon[32404]: pgmap v2540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:07 smithi116 ceph-mon[37097]: pgmap v2540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:07 smithi171 ceph-mon[37789]: pgmap v2540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:09 smithi044 ceph-mon[32404]: pgmap v2541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:09 smithi116 ceph-mon[37097]: pgmap v2541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:09 smithi171 ceph-mon[37789]: pgmap v2541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:44:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:44:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:44:11.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:11.401+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:44:11.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:11.401+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:11.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:11.401+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:44:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: pgmap v2542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3795.657672 secs 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: pgmap v2542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3795.657672 secs 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: pgmap v2542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3795.657672 secs 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:11.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:44:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:44:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:44:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:44:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:44:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: pgmap v2543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:44:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: pgmap v2544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:44:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: pgmap v2543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: pgmap v2544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:44:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:44:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: pgmap v2543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: pgmap v2544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:44:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:44:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:14 smithi044 ceph-mon[32404]: pgmap v2545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:14 smithi116 ceph-mon[37097]: pgmap v2545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:14 smithi171 ceph-mon[37789]: pgmap v2545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:15.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:15.339+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:44:15.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:15.339+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:44:15.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:15.340+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:44:15.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:15.340+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:44:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:16 smithi044 ceph-mon[32404]: pgmap v2546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3800.657757 secs 2024-07-26T09:44:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:16 smithi116 ceph-mon[37097]: pgmap v2546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3800.657757 secs 2024-07-26T09:44:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:16 smithi171 ceph-mon[37789]: pgmap v2546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3800.657757 secs 2024-07-26T09:44:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:18 smithi044 ceph-mon[32404]: pgmap v2547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:18 smithi116 ceph-mon[37097]: pgmap v2547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:18 smithi171 ceph-mon[37789]: pgmap v2547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:20 smithi044 ceph-mon[32404]: pgmap v2548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:20 smithi116 ceph-mon[37097]: pgmap v2548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:20 smithi171 ceph-mon[37789]: pgmap v2548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:44:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3805.657844 secs 2024-07-26T09:44:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3805.657844 secs 2024-07-26T09:44:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3805.657844 secs 2024-07-26T09:44:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:22 smithi116 ceph-mon[37097]: pgmap v2549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:44:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:22 smithi171 ceph-mon[37789]: pgmap v2549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:44:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:22 smithi044 ceph-mon[32404]: pgmap v2549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:44:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:24 smithi171 ceph-mon[37789]: pgmap v2550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:24 smithi044 ceph-mon[32404]: pgmap v2550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:24 smithi116 ceph-mon[37097]: pgmap v2550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:26.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:26.410+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:44:26.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:26.410+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:44:26.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:26.410+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:26.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:26 smithi044 ceph-mon[32404]: pgmap v2551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:26.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3810.657931 secs 2024-07-26T09:44:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:26 smithi171 ceph-mon[37789]: pgmap v2551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3810.657931 secs 2024-07-26T09:44:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:26 smithi116 ceph-mon[37097]: pgmap v2551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3810.657931 secs 2024-07-26T09:44:28.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:28 smithi116 ceph-mon[37097]: pgmap v2552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:28 smithi171 ceph-mon[37789]: pgmap v2552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:28 smithi044 ceph-mon[32404]: pgmap v2552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:30 smithi171 ceph-mon[37789]: pgmap v2553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:30 smithi044 ceph-mon[32404]: pgmap v2553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:30 smithi116 ceph-mon[37097]: pgmap v2553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3815.658020 secs 2024-07-26T09:44:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3815.658020 secs 2024-07-26T09:44:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3815.658020 secs 2024-07-26T09:44:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:32 smithi171 ceph-mon[37789]: pgmap v2554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:32 smithi044 ceph-mon[32404]: pgmap v2554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:32 smithi116 ceph-mon[37097]: pgmap v2554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:34 smithi171 ceph-mon[37789]: pgmap v2555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:34 smithi044 ceph-mon[32404]: pgmap v2555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:34 smithi116 ceph-mon[37097]: pgmap v2555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:36 smithi171 ceph-mon[37789]: pgmap v2556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3820.658111 secs 2024-07-26T09:44:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:36 smithi044 ceph-mon[32404]: pgmap v2556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3820.658111 secs 2024-07-26T09:44:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:36 smithi116 ceph-mon[37097]: pgmap v2556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3820.658111 secs 2024-07-26T09:44:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:38 smithi171 ceph-mon[37789]: pgmap v2557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:38 smithi044 ceph-mon[32404]: pgmap v2557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:38 smithi116 ceph-mon[37097]: pgmap v2557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:40 smithi171 ceph-mon[37789]: pgmap v2558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:40 smithi044 ceph-mon[32404]: pgmap v2558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:40 smithi116 ceph-mon[37097]: pgmap v2558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:41.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:41.413+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:44:41.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:41.413+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:41.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:41.413+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:44:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3825.658191 secs 2024-07-26T09:44:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3825.658191 secs 2024-07-26T09:44:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3825.658191 secs 2024-07-26T09:44:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:42 smithi044 ceph-mon[32404]: pgmap v2559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:42 smithi116 ceph-mon[37097]: pgmap v2559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:42 smithi171 ceph-mon[37789]: pgmap v2559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:44 smithi044 ceph-mon[32404]: pgmap v2560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:44 smithi116 ceph-mon[37097]: pgmap v2560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:44 smithi171 ceph-mon[37789]: pgmap v2560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:46 smithi044 ceph-mon[32404]: pgmap v2561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3830.658282 secs 2024-07-26T09:44:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:46 smithi116 ceph-mon[37097]: pgmap v2561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3830.658282 secs 2024-07-26T09:44:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:46 smithi171 ceph-mon[37789]: pgmap v2561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:46.863 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3830.658282 secs 2024-07-26T09:44:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:48 smithi044 ceph-mon[32404]: pgmap v2562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:48 smithi116 ceph-mon[37097]: pgmap v2562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:48 smithi171 ceph-mon[37789]: pgmap v2562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:50 smithi044 ceph-mon[32404]: pgmap v2563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:50 smithi116 ceph-mon[37097]: pgmap v2563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:50 smithi171 ceph-mon[37789]: pgmap v2563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3835.658431 secs 2024-07-26T09:44:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3835.658431 secs 2024-07-26T09:44:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3835.658431 secs 2024-07-26T09:44:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:52 smithi044 ceph-mon[32404]: pgmap v2564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:52 smithi116 ceph-mon[37097]: pgmap v2564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:52 smithi171 ceph-mon[37789]: pgmap v2564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:54 smithi044 ceph-mon[32404]: pgmap v2565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:54 smithi116 ceph-mon[37097]: pgmap v2565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:54 smithi171 ceph-mon[37789]: pgmap v2565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:44:56.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:56.422+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:44:56.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:56.422+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:44:56.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:44:56.422+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:56.672 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:56 smithi044 ceph-mon[32404]: pgmap v2566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:56.672 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3840.658593 secs 2024-07-26T09:44:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:56 smithi116 ceph-mon[37097]: pgmap v2566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3840.658593 secs 2024-07-26T09:44:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:56 smithi171 ceph-mon[37789]: pgmap v2566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:44:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3840.658593 secs 2024-07-26T09:44:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:44:58 smithi044 ceph-mon[32404]: pgmap v2567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:44:58 smithi116 ceph-mon[37097]: pgmap v2567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:44:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:44:58 smithi171 ceph-mon[37789]: pgmap v2567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:00 smithi044 ceph-mon[32404]: pgmap v2568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:00 smithi116 ceph-mon[37097]: pgmap v2568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:00 smithi171 ceph-mon[37789]: pgmap v2568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3845.658741 secs 2024-07-26T09:45:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3845.658741 secs 2024-07-26T09:45:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3845.658741 secs 2024-07-26T09:45:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:02 smithi044 ceph-mon[32404]: pgmap v2569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:02 smithi116 ceph-mon[37097]: pgmap v2569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:02 smithi171 ceph-mon[37789]: pgmap v2569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:04 smithi044 ceph-mon[32404]: pgmap v2570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:45:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:04 smithi116 ceph-mon[37097]: pgmap v2570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:04 smithi171 ceph-mon[37789]: pgmap v2570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:45:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:06 smithi044 ceph-mon[32404]: pgmap v2571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3850.658904 secs 2024-07-26T09:45:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:06 smithi116 ceph-mon[37097]: pgmap v2571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3850.658904 secs 2024-07-26T09:45:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:06 smithi171 ceph-mon[37789]: pgmap v2571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3850.658904 secs 2024-07-26T09:45:08.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:08 smithi044 ceph-mon[32404]: pgmap v2572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:08 smithi116 ceph-mon[37097]: pgmap v2572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:08 smithi171 ceph-mon[37789]: pgmap v2572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:10 smithi044 ceph-mon[32404]: pgmap v2573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:10 smithi116 ceph-mon[37097]: pgmap v2573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:10 smithi171 ceph-mon[37789]: pgmap v2573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:11.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:11.425+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:45:11.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:11.425+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:45:11.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:11.425+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3855.658992 secs 2024-07-26T09:45:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3855.658992 secs 2024-07-26T09:45:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3855.658992 secs 2024-07-26T09:45:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:12 smithi044 ceph-mon[32404]: pgmap v2574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:12 smithi116 ceph-mon[37097]: pgmap v2574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:12 smithi171 ceph-mon[37789]: pgmap v2574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:45:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:45:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:45:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:14 smithi044 ceph-mon[32404]: pgmap v2575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:45:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:14 smithi044 ceph-mon[32404]: pgmap v2576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:14 smithi116 ceph-mon[37097]: pgmap v2575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:45:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:14 smithi116 ceph-mon[37097]: pgmap v2576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:14 smithi171 ceph-mon[37789]: pgmap v2575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:45:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:14 smithi171 ceph-mon[37789]: pgmap v2576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:15 smithi044 ceph-mon[32404]: pgmap v2577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:15 smithi116 ceph-mon[37097]: pgmap v2577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:15 smithi171 ceph-mon[37789]: pgmap v2577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3860.659154 secs 2024-07-26T09:45:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3860.659154 secs 2024-07-26T09:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3860.659154 secs 2024-07-26T09:45:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:17 smithi044 ceph-mon[32404]: pgmap v2578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:17 smithi116 ceph-mon[37097]: pgmap v2578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:17 smithi171 ceph-mon[37789]: pgmap v2578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:45:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:19 smithi044 ceph-mon[32404]: pgmap v2579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:19 smithi116 ceph-mon[37097]: pgmap v2579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:19 smithi171 ceph-mon[37789]: pgmap v2579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3865.659302 secs 2024-07-26T09:45:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:21 smithi044 ceph-mon[32404]: pgmap v2580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3865.659302 secs 2024-07-26T09:45:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:21 smithi116 ceph-mon[37097]: pgmap v2580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3865.659302 secs 2024-07-26T09:45:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:21 smithi171 ceph-mon[37789]: pgmap v2580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:45:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:23 smithi116 ceph-mon[37097]: pgmap v2581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:45:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:23 smithi171 ceph-mon[37789]: pgmap v2581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:45:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:23 smithi044 ceph-mon[32404]: pgmap v2581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:45:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:25 smithi171 ceph-mon[37789]: pgmap v2582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:25 smithi044 ceph-mon[32404]: pgmap v2582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:25 smithi116 ceph-mon[37097]: pgmap v2582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:26.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:26.434+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:45:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:26.434+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:26.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:26.434+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:45:26.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3870.659430 secs 2024-07-26T09:45:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3870.659430 secs 2024-07-26T09:45:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3870.659430 secs 2024-07-26T09:45:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:27 smithi171 ceph-mon[37789]: pgmap v2583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:27 smithi044 ceph-mon[32404]: pgmap v2583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:27 smithi116 ceph-mon[37097]: pgmap v2583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:29.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:29 smithi116 ceph-mon[37097]: pgmap v2584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:29 smithi171 ceph-mon[37789]: pgmap v2584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:29 smithi044 ceph-mon[32404]: pgmap v2584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3875.659580 secs 2024-07-26T09:45:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:31 smithi171 ceph-mon[37789]: pgmap v2585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3875.659580 secs 2024-07-26T09:45:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:31 smithi044 ceph-mon[32404]: pgmap v2585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3875.659580 secs 2024-07-26T09:45:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:31 smithi116 ceph-mon[37097]: pgmap v2585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:33 smithi171 ceph-mon[37789]: pgmap v2586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:33 smithi044 ceph-mon[32404]: pgmap v2586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:33 smithi116 ceph-mon[37097]: pgmap v2586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:35 smithi171 ceph-mon[37789]: pgmap v2587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:35 smithi044 ceph-mon[32404]: pgmap v2587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:35 smithi116 ceph-mon[37097]: pgmap v2587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3880.659733 secs 2024-07-26T09:45:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3880.659733 secs 2024-07-26T09:45:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3880.659733 secs 2024-07-26T09:45:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:37 smithi171 ceph-mon[37789]: pgmap v2588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:37 smithi044 ceph-mon[32404]: pgmap v2588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:37 smithi116 ceph-mon[37097]: pgmap v2588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:39 smithi171 ceph-mon[37789]: pgmap v2589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:39 smithi044 ceph-mon[32404]: pgmap v2589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:39 smithi116 ceph-mon[37097]: pgmap v2589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:41.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:41.438+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:45:41.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:41.438+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:45:41.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:41.438+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3885.659887 secs 2024-07-26T09:45:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:41 smithi044 ceph-mon[32404]: pgmap v2590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3885.659887 secs 2024-07-26T09:45:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:41 smithi116 ceph-mon[37097]: pgmap v2590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3885.659887 secs 2024-07-26T09:45:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:41 smithi171 ceph-mon[37789]: pgmap v2590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:43 smithi044 ceph-mon[32404]: pgmap v2591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:43 smithi116 ceph-mon[37097]: pgmap v2591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:43 smithi171 ceph-mon[37789]: pgmap v2591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:45 smithi044 ceph-mon[32404]: pgmap v2592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:45 smithi116 ceph-mon[37097]: pgmap v2592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:45 smithi171 ceph-mon[37789]: pgmap v2592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3890.660023 secs 2024-07-26T09:45:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3890.660023 secs 2024-07-26T09:45:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3890.660023 secs 2024-07-26T09:45:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:47 smithi044 ceph-mon[32404]: pgmap v2593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:47 smithi116 ceph-mon[37097]: pgmap v2593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:47 smithi171 ceph-mon[37789]: pgmap v2593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:49 smithi044 ceph-mon[32404]: pgmap v2594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:49 smithi116 ceph-mon[37097]: pgmap v2594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:50.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:49 smithi171 ceph-mon[37789]: pgmap v2594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:45:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3895.660168 secs 2024-07-26T09:45:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:51 smithi044 ceph-mon[32404]: pgmap v2595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3895.660168 secs 2024-07-26T09:45:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:51 smithi116 ceph-mon[37097]: pgmap v2595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:52.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3895.660168 secs 2024-07-26T09:45:52.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:51 smithi171 ceph-mon[37789]: pgmap v2595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:45:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:53 smithi044 ceph-mon[32404]: pgmap v2596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:53 smithi116 ceph-mon[37097]: pgmap v2596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:53 smithi171 ceph-mon[37789]: pgmap v2596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:45:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:55 smithi044 ceph-mon[32404]: pgmap v2597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:55 smithi116 ceph-mon[37097]: pgmap v2597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:55 smithi171 ceph-mon[37789]: pgmap v2597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:45:56.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:56.446+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:45:56.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:56.446+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:56.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:45:56.446+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:45:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3900.660312 secs 2024-07-26T09:45:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3900.660312 secs 2024-07-26T09:45:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3900.660312 secs 2024-07-26T09:45:57.923 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:57 smithi044 ceph-mon[32404]: pgmap v2598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:45:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:57 smithi116 ceph-mon[37097]: pgmap v2598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:45:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:57 smithi171 ceph-mon[37789]: pgmap v2598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:45:59 smithi044 ceph-mon[32404]: pgmap v2599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:46:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:45:59 smithi116 ceph-mon[37097]: pgmap v2599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:46:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:45:59 smithi171 ceph-mon[37789]: pgmap v2599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:46:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3905.660408 secs 2024-07-26T09:46:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:01 smithi044 ceph-mon[32404]: pgmap v2600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3905.660408 secs 2024-07-26T09:46:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:01 smithi116 ceph-mon[37097]: pgmap v2600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3905.660408 secs 2024-07-26T09:46:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:01 smithi171 ceph-mon[37789]: pgmap v2600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:03 smithi044 ceph-mon[32404]: pgmap v2601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:03 smithi116 ceph-mon[37097]: pgmap v2601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:03 smithi171 ceph-mon[37789]: pgmap v2601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:05 smithi044 ceph-mon[32404]: pgmap v2602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:05 smithi116 ceph-mon[37097]: pgmap v2602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:05 smithi171 ceph-mon[37789]: pgmap v2602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:06.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3910.660545 secs 2024-07-26T09:46:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3910.660545 secs 2024-07-26T09:46:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3910.660545 secs 2024-07-26T09:46:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:07 smithi044 ceph-mon[32404]: pgmap v2603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:07 smithi116 ceph-mon[37097]: pgmap v2603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:07 smithi171 ceph-mon[37789]: pgmap v2603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:46:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:09 smithi044 ceph-mon[32404]: pgmap v2604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:09 smithi116 ceph-mon[37097]: pgmap v2604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:09 smithi171 ceph-mon[37789]: pgmap v2604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:46:11.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:11.450+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:46:11.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:11.450+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:46:11.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:11.450+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:11.705 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3915.660697 secs 2024-07-26T09:46:11.706 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:11 smithi044 ceph-mon[32404]: pgmap v2605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3915.660697 secs 2024-07-26T09:46:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:11 smithi116 ceph-mon[37097]: pgmap v2605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3915.660697 secs 2024-07-26T09:46:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:11 smithi171 ceph-mon[37789]: pgmap v2605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:14 smithi171 ceph-mon[37789]: pgmap v2606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:14 smithi044 ceph-mon[32404]: pgmap v2606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:14 smithi116 ceph-mon[37097]: pgmap v2606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:15 smithi171 ceph-mon[37789]: pgmap v2607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:15 smithi171 ceph-mon[37789]: pgmap v2608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:15 smithi044 ceph-mon[32404]: pgmap v2607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:15 smithi044 ceph-mon[32404]: pgmap v2608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:15 smithi116 ceph-mon[37097]: pgmap v2607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:15 smithi116 ceph-mon[37097]: pgmap v2608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:46:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3920.660832 secs 2024-07-26T09:46:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3920.660832 secs 2024-07-26T09:46:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3920.660832 secs 2024-07-26T09:46:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:17 smithi171 ceph-mon[37789]: pgmap v2609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:46:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:17 smithi044 ceph-mon[32404]: pgmap v2609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:46:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:17 smithi116 ceph-mon[37097]: pgmap v2609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T09:46:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:19 smithi044 ceph-mon[32404]: pgmap v2610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:19 smithi116 ceph-mon[37097]: pgmap v2610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:19 smithi171 ceph-mon[37789]: pgmap v2610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3925.660967 secs 2024-07-26T09:46:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:21 smithi044 ceph-mon[32404]: pgmap v2611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3925.660967 secs 2024-07-26T09:46:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:21 smithi116 ceph-mon[37097]: pgmap v2611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3925.660967 secs 2024-07-26T09:46:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:21 smithi171 ceph-mon[37789]: pgmap v2611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:46:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:23 smithi044 ceph-mon[32404]: pgmap v2612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:46:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:23 smithi116 ceph-mon[37097]: pgmap v2612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:46:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:23 smithi171 ceph-mon[37789]: pgmap v2612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:46:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:25 smithi044 ceph-mon[32404]: pgmap v2613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:46:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:25 smithi116 ceph-mon[37097]: pgmap v2613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:46:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:25 smithi171 ceph-mon[37789]: pgmap v2613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:46:26.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:26.458+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:46:26.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:26.458+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:26.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:26.458+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:46:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3930.661129 secs 2024-07-26T09:46:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3930.661129 secs 2024-07-26T09:46:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3930.661129 secs 2024-07-26T09:46:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:27 smithi044 ceph-mon[32404]: pgmap v2614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:27 smithi116 ceph-mon[37097]: pgmap v2614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:27 smithi171 ceph-mon[37789]: pgmap v2614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:29 smithi044 ceph-mon[32404]: pgmap v2615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:29 smithi116 ceph-mon[37097]: pgmap v2615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:29 smithi171 ceph-mon[37789]: pgmap v2615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3935.661296 secs 2024-07-26T09:46:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:31 smithi044 ceph-mon[32404]: pgmap v2616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3935.661296 secs 2024-07-26T09:46:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:31 smithi116 ceph-mon[37097]: pgmap v2616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3935.661296 secs 2024-07-26T09:46:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:31 smithi171 ceph-mon[37789]: pgmap v2616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:33 smithi044 ceph-mon[32404]: pgmap v2617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:33 smithi116 ceph-mon[37097]: pgmap v2617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:33 smithi171 ceph-mon[37789]: pgmap v2617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:35 smithi044 ceph-mon[32404]: pgmap v2618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:35 smithi116 ceph-mon[37097]: pgmap v2618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:35 smithi171 ceph-mon[37789]: pgmap v2618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:36.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3940.661463 secs 2024-07-26T09:46:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3940.661463 secs 2024-07-26T09:46:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3940.661463 secs 2024-07-26T09:46:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:37 smithi044 ceph-mon[32404]: pgmap v2619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:46:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:37 smithi116 ceph-mon[37097]: pgmap v2619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:46:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:37 smithi171 ceph-mon[37789]: pgmap v2619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:46:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:38 smithi044 ceph-mon[32404]: pgmap v2620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:38 smithi116 ceph-mon[37097]: pgmap v2620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:38 smithi171 ceph-mon[37789]: pgmap v2620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:41.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:41.462+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:46:41.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:41.462+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:41.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:41.462+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:46:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3945.661675 secs 2024-07-26T09:46:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:41 smithi044 ceph-mon[32404]: pgmap v2621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3945.661675 secs 2024-07-26T09:46:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:41 smithi116 ceph-mon[37097]: pgmap v2621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3945.661675 secs 2024-07-26T09:46:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:41 smithi171 ceph-mon[37789]: pgmap v2621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:43 smithi044 ceph-mon[32404]: pgmap v2622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:43 smithi116 ceph-mon[37097]: pgmap v2622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:43 smithi171 ceph-mon[37789]: pgmap v2622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:45 smithi044 ceph-mon[32404]: pgmap v2623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:45 smithi116 ceph-mon[37097]: pgmap v2623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:45 smithi171 ceph-mon[37789]: pgmap v2623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3950.661781 secs 2024-07-26T09:46:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3950.661781 secs 2024-07-26T09:46:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3950.661781 secs 2024-07-26T09:46:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:47 smithi044 ceph-mon[32404]: pgmap v2624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:47 smithi116 ceph-mon[37097]: pgmap v2624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:47 smithi171 ceph-mon[37789]: pgmap v2624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:49 smithi044 ceph-mon[32404]: pgmap v2625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:49 smithi116 ceph-mon[37097]: pgmap v2625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:49 smithi171 ceph-mon[37789]: pgmap v2625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3955.661883 secs 2024-07-26T09:46:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:51 smithi044 ceph-mon[32404]: pgmap v2626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3955.661883 secs 2024-07-26T09:46:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:51 smithi116 ceph-mon[37097]: pgmap v2626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3955.661883 secs 2024-07-26T09:46:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:51 smithi171 ceph-mon[37789]: pgmap v2626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:46:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:53 smithi044 ceph-mon[32404]: pgmap v2627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:53 smithi116 ceph-mon[37097]: pgmap v2627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:53 smithi171 ceph-mon[37789]: pgmap v2627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:46:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:55 smithi044 ceph-mon[32404]: pgmap v2628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:55 smithi116 ceph-mon[37097]: pgmap v2628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:55 smithi171 ceph-mon[37789]: pgmap v2628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:46:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:56.470+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:46:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:56.470+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:46:56.470+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:46:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3960.661979 secs 2024-07-26T09:46:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3960.661979 secs 2024-07-26T09:46:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3960.661979 secs 2024-07-26T09:46:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:57 smithi044 ceph-mon[32404]: pgmap v2629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:46:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:57 smithi116 ceph-mon[37097]: pgmap v2629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:46:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:57 smithi171 ceph-mon[37789]: pgmap v2629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:46:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:46:58 smithi044 ceph-mon[32404]: pgmap v2630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:46:58 smithi116 ceph-mon[37097]: pgmap v2630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:46:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:46:58 smithi171 ceph-mon[37789]: pgmap v2630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3965.662140 secs 2024-07-26T09:47:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:01 smithi044 ceph-mon[32404]: pgmap v2631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3965.662140 secs 2024-07-26T09:47:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:01 smithi116 ceph-mon[37097]: pgmap v2631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:01.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3965.662140 secs 2024-07-26T09:47:01.781 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:01 smithi171 ceph-mon[37789]: pgmap v2631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:03 smithi044 ceph-mon[32404]: pgmap v2632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:03 smithi116 ceph-mon[37097]: pgmap v2632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:03 smithi171 ceph-mon[37789]: pgmap v2632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:05 smithi044 ceph-mon[32404]: pgmap v2633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:05 smithi116 ceph-mon[37097]: pgmap v2633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:05 smithi171 ceph-mon[37789]: pgmap v2633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:06.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3970.662285 secs 2024-07-26T09:47:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3970.662285 secs 2024-07-26T09:47:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3970.662285 secs 2024-07-26T09:47:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:07 smithi044 ceph-mon[32404]: pgmap v2634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:07 smithi116 ceph-mon[37097]: pgmap v2634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:07 smithi171 ceph-mon[37789]: pgmap v2634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:09 smithi044 ceph-mon[32404]: pgmap v2635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:09 smithi116 ceph-mon[37097]: pgmap v2635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:09 smithi171 ceph-mon[37789]: pgmap v2635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:11.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:11.474+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:47:11.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:11.474+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:47:11.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:11.474+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3975.662451 secs 2024-07-26T09:47:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:11 smithi044 ceph-mon[32404]: pgmap v2636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3975.662451 secs 2024-07-26T09:47:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:11 smithi116 ceph-mon[37097]: pgmap v2636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3975.662451 secs 2024-07-26T09:47:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:11 smithi171 ceph-mon[37789]: pgmap v2636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:13 smithi044 ceph-mon[32404]: pgmap v2637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:13 smithi116 ceph-mon[37097]: pgmap v2637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:13 smithi171 ceph-mon[37789]: pgmap v2637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.372+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.372+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.373+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.373+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.373+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.373+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.373+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:47:15.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:15.373+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:47:15.661 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:15 smithi044 ceph-mon[32404]: pgmap v2638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:15.661 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:15.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:15 smithi171 ceph-mon[37789]: pgmap v2638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:15.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:15.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:15 smithi116 ceph-mon[37097]: pgmap v2638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3980.662527 secs 2024-07-26T09:47:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:16 smithi044 ceph-mon[32404]: pgmap v2639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3980.662527 secs 2024-07-26T09:47:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:16 smithi116 ceph-mon[37097]: pgmap v2639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3980.662527 secs 2024-07-26T09:47:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:16 smithi171 ceph-mon[37789]: pgmap v2639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:18 smithi044 ceph-mon[32404]: pgmap v2640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:18 smithi116 ceph-mon[37097]: pgmap v2640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:19.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:18 smithi171 ceph-mon[37789]: pgmap v2640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:20.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:20 smithi171 ceph-mon[37789]: pgmap v2641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:20.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:20 smithi171 ceph-mon[37789]: pgmap v2642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:20.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:20 smithi044 ceph-mon[32404]: pgmap v2641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:20 smithi044 ceph-mon[32404]: pgmap v2642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:20 smithi116 ceph-mon[37097]: pgmap v2641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:47:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:20 smithi116 ceph-mon[37097]: pgmap v2642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:47:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3985.662669 secs 2024-07-26T09:47:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3985.662669 secs 2024-07-26T09:47:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3985.662669 secs 2024-07-26T09:47:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:22 smithi044 ceph-mon[32404]: pgmap v2643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:47:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:22 smithi116 ceph-mon[37097]: pgmap v2643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:47:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:22 smithi171 ceph-mon[37789]: pgmap v2643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:47:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:24 smithi044 ceph-mon[32404]: pgmap v2644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:24 smithi116 ceph-mon[37097]: pgmap v2644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:24 smithi171 ceph-mon[37789]: pgmap v2644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:26.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:26.483+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:47:26.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:26.483+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:47:26.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:26.483+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:26.486 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:26 smithi044 ceph-mon[32404]: pgmap v2645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:26.486 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3990.662834 secs 2024-07-26T09:47:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:26 smithi116 ceph-mon[37097]: pgmap v2645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3990.662834 secs 2024-07-26T09:47:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:26 smithi171 ceph-mon[37789]: pgmap v2645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:47:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3990.662834 secs 2024-07-26T09:47:28.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:28 smithi044 ceph-mon[32404]: pgmap v2646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:28 smithi116 ceph-mon[37097]: pgmap v2646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:28 smithi171 ceph-mon[37789]: pgmap v2646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:30 smithi044 ceph-mon[32404]: pgmap v2647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:47:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:30 smithi116 ceph-mon[37097]: pgmap v2647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:47:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:30 smithi171 ceph-mon[37789]: pgmap v2647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:47:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 3995.662939 secs 2024-07-26T09:47:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 3995.662939 secs 2024-07-26T09:47:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 3995.662939 secs 2024-07-26T09:47:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:32 smithi044 ceph-mon[32404]: pgmap v2648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:32 smithi116 ceph-mon[37097]: pgmap v2648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:32 smithi171 ceph-mon[37789]: pgmap v2648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:34 smithi044 ceph-mon[32404]: pgmap v2649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:47:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:34 smithi116 ceph-mon[37097]: pgmap v2649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:47:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:34 smithi171 ceph-mon[37789]: pgmap v2649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:47:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:36 smithi044 ceph-mon[32404]: pgmap v2650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4000.663103 secs 2024-07-26T09:47:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:36 smithi116 ceph-mon[37097]: pgmap v2650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4000.663103 secs 2024-07-26T09:47:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:36 smithi171 ceph-mon[37789]: pgmap v2650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4000.663103 secs 2024-07-26T09:47:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:37 smithi044 ceph-mon[32404]: pgmap v2651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:37 smithi116 ceph-mon[37097]: pgmap v2651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:37 smithi171 ceph-mon[37789]: pgmap v2651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:40 smithi044 ceph-mon[32404]: pgmap v2652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:40 smithi116 ceph-mon[37097]: pgmap v2652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:40 smithi171 ceph-mon[37789]: pgmap v2652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:47:41.489 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4005.663257 secs 2024-07-26T09:47:41.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:41.486+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:47:41.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:41.486+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:47:41.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:41.486+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4005.663257 secs 2024-07-26T09:47:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4005.663257 secs 2024-07-26T09:47:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:42 smithi044 ceph-mon[32404]: pgmap v2653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:42 smithi116 ceph-mon[37097]: pgmap v2653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:42 smithi171 ceph-mon[37789]: pgmap v2653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:44 smithi044 ceph-mon[32404]: pgmap v2654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:44 smithi116 ceph-mon[37097]: pgmap v2654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:44 smithi171 ceph-mon[37789]: pgmap v2654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:47:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:46 smithi044 ceph-mon[32404]: pgmap v2655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4010.663399 secs 2024-07-26T09:47:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:46 smithi116 ceph-mon[37097]: pgmap v2655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4010.663399 secs 2024-07-26T09:47:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:46 smithi171 ceph-mon[37789]: pgmap v2655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:47:46.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4010.663399 secs 2024-07-26T09:47:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:48 smithi044 ceph-mon[32404]: pgmap v2656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:48 smithi116 ceph-mon[37097]: pgmap v2656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:48 smithi171 ceph-mon[37789]: pgmap v2656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:50 smithi044 ceph-mon[32404]: pgmap v2657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:50 smithi116 ceph-mon[37097]: pgmap v2657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:50 smithi171 ceph-mon[37789]: pgmap v2657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4015.663565 secs 2024-07-26T09:47:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4015.663565 secs 2024-07-26T09:47:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4015.663565 secs 2024-07-26T09:47:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:52 smithi044 ceph-mon[32404]: pgmap v2658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:52 smithi116 ceph-mon[37097]: pgmap v2658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:52 smithi171 ceph-mon[37789]: pgmap v2658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:54 smithi044 ceph-mon[32404]: pgmap v2659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:47:54.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:54 smithi116 ceph-mon[37097]: pgmap v2659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:47:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:54 smithi171 ceph-mon[37789]: pgmap v2659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:47:56.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:56.492+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:47:56.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:56.492+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:47:56.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:47:56.492+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:56.495 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:56 smithi044 ceph-mon[32404]: pgmap v2660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:56.496 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4020.663701 secs 2024-07-26T09:47:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:56 smithi116 ceph-mon[37097]: pgmap v2660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4020.663701 secs 2024-07-26T09:47:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:56 smithi171 ceph-mon[37789]: pgmap v2660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:47:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4020.663701 secs 2024-07-26T09:47:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:47:58 smithi044 ceph-mon[32404]: pgmap v2661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:47:58 smithi116 ceph-mon[37097]: pgmap v2661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:47:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:47:58 smithi171 ceph-mon[37789]: pgmap v2661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:48:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:00 smithi044 ceph-mon[32404]: pgmap v2662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:00 smithi116 ceph-mon[37097]: pgmap v2662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:00 smithi171 ceph-mon[37789]: pgmap v2662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4025.663852 secs 2024-07-26T09:48:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4025.663852 secs 2024-07-26T09:48:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4025.663852 secs 2024-07-26T09:48:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:02 smithi044 ceph-mon[32404]: pgmap v2663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:02 smithi116 ceph-mon[37097]: pgmap v2663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:02 smithi171 ceph-mon[37789]: pgmap v2663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:04 smithi044 ceph-mon[32404]: pgmap v2664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:04 smithi116 ceph-mon[37097]: pgmap v2664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:04 smithi171 ceph-mon[37789]: pgmap v2664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:06 smithi044 ceph-mon[32404]: pgmap v2665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4030.663993 secs 2024-07-26T09:48:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:06 smithi116 ceph-mon[37097]: pgmap v2665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4030.663993 secs 2024-07-26T09:48:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:06 smithi171 ceph-mon[37789]: pgmap v2665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4030.663993 secs 2024-07-26T09:48:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:08 smithi116 ceph-mon[37097]: pgmap v2666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:08 smithi171 ceph-mon[37789]: pgmap v2666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:08.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:08 smithi044 ceph-mon[32404]: pgmap v2666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:10 smithi171 ceph-mon[37789]: pgmap v2667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:10 smithi044 ceph-mon[32404]: pgmap v2667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:10 smithi116 ceph-mon[37097]: pgmap v2667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:11.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:11.496+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:48:11.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:11.496+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:11.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:11.496+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:48:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4035.664119 secs 2024-07-26T09:48:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4035.664119 secs 2024-07-26T09:48:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4035.664119 secs 2024-07-26T09:48:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:12 smithi171 ceph-mon[37789]: pgmap v2668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:48:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:12 smithi044 ceph-mon[32404]: pgmap v2668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:48:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:12 smithi116 ceph-mon[37097]: pgmap v2668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:48:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:14 smithi171 ceph-mon[37789]: pgmap v2669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:14 smithi044 ceph-mon[32404]: pgmap v2669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:14 smithi116 ceph-mon[37097]: pgmap v2669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:16 smithi171 ceph-mon[37789]: pgmap v2670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:48:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4040.664266 secs 2024-07-26T09:48:16.675 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:16 smithi044 ceph-mon[32404]: pgmap v2670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:48:16.675 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4040.664266 secs 2024-07-26T09:48:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:16 smithi116 ceph-mon[37097]: pgmap v2670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:48:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4040.664266 secs 2024-07-26T09:48:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:18 smithi171 ceph-mon[37789]: pgmap v2671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:18 smithi044 ceph-mon[32404]: pgmap v2671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:18 smithi116 ceph-mon[37097]: pgmap v2671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:20 smithi171 ceph-mon[37789]: pgmap v2672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:20 smithi044 ceph-mon[32404]: pgmap v2672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:20 smithi116 ceph-mon[37097]: pgmap v2672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4045.664395 secs 2024-07-26T09:48:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:21 smithi171 ceph-mon[37789]: pgmap v2673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:48:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:21 smithi171 ceph-mon[37789]: pgmap v2674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:48:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4045.664395 secs 2024-07-26T09:48:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:21 smithi044 ceph-mon[32404]: pgmap v2673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:48:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:21 smithi044 ceph-mon[32404]: pgmap v2674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:48:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4045.664395 secs 2024-07-26T09:48:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:21 smithi116 ceph-mon[37097]: pgmap v2673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:48:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:21 smithi116 ceph-mon[37097]: pgmap v2674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:48:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:23 smithi044 ceph-mon[32404]: pgmap v2675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:48:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:23 smithi116 ceph-mon[37097]: pgmap v2675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:48:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:23 smithi171 ceph-mon[37789]: pgmap v2675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:48:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:25 smithi044 ceph-mon[32404]: pgmap v2676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:25 smithi116 ceph-mon[37097]: pgmap v2676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:25 smithi171 ceph-mon[37789]: pgmap v2676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:26.504+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:48:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:26.504+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:48:26.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:26.504+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4050.664540 secs 2024-07-26T09:48:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4050.664540 secs 2024-07-26T09:48:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4050.664540 secs 2024-07-26T09:48:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:27 smithi044 ceph-mon[32404]: pgmap v2677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:48:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:27 smithi116 ceph-mon[37097]: pgmap v2677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:48:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:27 smithi171 ceph-mon[37789]: pgmap v2677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:48:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:29 smithi044 ceph-mon[32404]: pgmap v2678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:29 smithi116 ceph-mon[37097]: pgmap v2678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:29 smithi171 ceph-mon[37789]: pgmap v2678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:48:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4055.664686 secs 2024-07-26T09:48:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:31 smithi044 ceph-mon[32404]: pgmap v2679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:48:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4055.664686 secs 2024-07-26T09:48:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:31 smithi116 ceph-mon[37097]: pgmap v2679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:48:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4055.664686 secs 2024-07-26T09:48:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:31 smithi171 ceph-mon[37789]: pgmap v2679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:48:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:33 smithi044 ceph-mon[32404]: pgmap v2680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:33 smithi116 ceph-mon[37097]: pgmap v2680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:33 smithi171 ceph-mon[37789]: pgmap v2680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:35.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:35 smithi044 ceph-mon[32404]: pgmap v2681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:35 smithi116 ceph-mon[37097]: pgmap v2681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:35 smithi171 ceph-mon[37789]: pgmap v2681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4060.664836 secs 2024-07-26T09:48:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4060.664836 secs 2024-07-26T09:48:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4060.664836 secs 2024-07-26T09:48:37.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:37 smithi044 ceph-mon[32404]: pgmap v2682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:37 smithi116 ceph-mon[37097]: pgmap v2682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:37 smithi171 ceph-mon[37789]: pgmap v2682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:39 smithi044 ceph-mon[32404]: pgmap v2683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:39 smithi116 ceph-mon[37097]: pgmap v2683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:39 smithi171 ceph-mon[37789]: pgmap v2683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:41.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:41.507+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:48:41.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:41.507+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:48:41.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:41.507+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4065.664990 secs 2024-07-26T09:48:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:41 smithi044 ceph-mon[32404]: pgmap v2684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4065.664990 secs 2024-07-26T09:48:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:41 smithi116 ceph-mon[37097]: pgmap v2684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4065.664990 secs 2024-07-26T09:48:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:41 smithi171 ceph-mon[37789]: pgmap v2684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:43 smithi044 ceph-mon[32404]: pgmap v2685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:43 smithi116 ceph-mon[37097]: pgmap v2685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:43 smithi171 ceph-mon[37789]: pgmap v2685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:44 smithi171 ceph-mon[37789]: pgmap v2686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:44 smithi044 ceph-mon[32404]: pgmap v2686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:45.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:44 smithi116 ceph-mon[37097]: pgmap v2686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4070.665141 secs 2024-07-26T09:48:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4070.665141 secs 2024-07-26T09:48:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4070.665141 secs 2024-07-26T09:48:47.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:47 smithi044 ceph-mon[32404]: pgmap v2687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:47.596 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:47 smithi171 ceph-mon[37789]: pgmap v2687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:47.596 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:47 smithi116 ceph-mon[37097]: pgmap v2687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:49 smithi044 ceph-mon[32404]: pgmap v2688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:49 smithi116 ceph-mon[37097]: pgmap v2688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:49 smithi171 ceph-mon[37789]: pgmap v2688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:51.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4075.665284 secs 2024-07-26T09:48:51.770 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:51 smithi044 ceph-mon[32404]: pgmap v2689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4075.665284 secs 2024-07-26T09:48:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:51 smithi116 ceph-mon[37097]: pgmap v2689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4075.665284 secs 2024-07-26T09:48:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:51 smithi171 ceph-mon[37789]: pgmap v2689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:53 smithi044 ceph-mon[32404]: pgmap v2690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:53 smithi116 ceph-mon[37097]: pgmap v2690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:53 smithi171 ceph-mon[37789]: pgmap v2690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:48:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:55 smithi044 ceph-mon[32404]: pgmap v2691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:55 smithi116 ceph-mon[37097]: pgmap v2691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:55 smithi171 ceph-mon[37789]: pgmap v2691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:48:56.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:56.515+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:48:56.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:56.515+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:56.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:48:56.515+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:48:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4080.665427 secs 2024-07-26T09:48:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4080.665427 secs 2024-07-26T09:48:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4080.665427 secs 2024-07-26T09:48:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:57 smithi044 ceph-mon[32404]: pgmap v2692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:57 smithi116 ceph-mon[37097]: pgmap v2692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:57 smithi171 ceph-mon[37789]: pgmap v2692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:48:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:48:59 smithi044 ceph-mon[32404]: pgmap v2693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:48:59 smithi116 ceph-mon[37097]: pgmap v2693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:48:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:48:59 smithi171 ceph-mon[37789]: pgmap v2693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4085.665589 secs 2024-07-26T09:49:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:01 smithi044 ceph-mon[32404]: pgmap v2694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4085.665589 secs 2024-07-26T09:49:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:01 smithi116 ceph-mon[37097]: pgmap v2694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4085.665589 secs 2024-07-26T09:49:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:01 smithi171 ceph-mon[37789]: pgmap v2694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:03 smithi044 ceph-mon[32404]: pgmap v2695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:03 smithi116 ceph-mon[37097]: pgmap v2695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:03 smithi171 ceph-mon[37789]: pgmap v2695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:05 smithi044 ceph-mon[32404]: pgmap v2696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:05 smithi116 ceph-mon[37097]: pgmap v2696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:05 smithi171 ceph-mon[37789]: pgmap v2696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4090.665747 secs 2024-07-26T09:49:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4090.665747 secs 2024-07-26T09:49:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4090.665747 secs 2024-07-26T09:49:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:07 smithi116 ceph-mon[37097]: pgmap v2697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:07 smithi171 ceph-mon[37789]: pgmap v2697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:07 smithi044 ceph-mon[32404]: pgmap v2697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:08 smithi171 ceph-mon[37789]: pgmap v2698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:08 smithi044 ceph-mon[32404]: pgmap v2698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:08 smithi116 ceph-mon[37097]: pgmap v2698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:11.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:11.519+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:49:11.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:11.519+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:49:11.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:11.519+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4095.665893 secs 2024-07-26T09:49:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:11 smithi044 ceph-mon[32404]: pgmap v2699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4095.665893 secs 2024-07-26T09:49:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:11 smithi116 ceph-mon[37097]: pgmap v2699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4095.665893 secs 2024-07-26T09:49:11.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:11 smithi171 ceph-mon[37789]: pgmap v2699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:13 smithi044 ceph-mon[32404]: pgmap v2700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:13 smithi116 ceph-mon[37097]: pgmap v2700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:13 smithi171 ceph-mon[37789]: pgmap v2700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:15 smithi044 ceph-mon[32404]: pgmap v2701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:15 smithi116 ceph-mon[37097]: pgmap v2701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:15 smithi171 ceph-mon[37789]: pgmap v2701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4100.666052 secs 2024-07-26T09:49:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4100.666052 secs 2024-07-26T09:49:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4100.666052 secs 2024-07-26T09:49:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:17 smithi044 ceph-mon[32404]: pgmap v2702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:17 smithi116 ceph-mon[37097]: pgmap v2702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:17 smithi171 ceph-mon[37789]: pgmap v2702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:19 smithi044 ceph-mon[32404]: pgmap v2703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:19 smithi116 ceph-mon[37097]: pgmap v2703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:19 smithi171 ceph-mon[37789]: pgmap v2703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4105.666218 secs 2024-07-26T09:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:21 smithi044 ceph-mon[32404]: pgmap v2704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4105.666218 secs 2024-07-26T09:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:21 smithi116 ceph-mon[37097]: pgmap v2704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4105.666218 secs 2024-07-26T09:49:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:21 smithi171 ceph-mon[37789]: pgmap v2704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:49:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:49:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:49:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:23 smithi116 ceph-mon[37097]: pgmap v2705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:23 smithi116 ceph-mon[37097]: pgmap v2706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:49:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:23 smithi171 ceph-mon[37789]: pgmap v2705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:23 smithi171 ceph-mon[37789]: pgmap v2706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:49:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:23 smithi044 ceph-mon[32404]: pgmap v2705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:23 smithi044 ceph-mon[32404]: pgmap v2706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T09:49:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:24 smithi171 ceph-mon[37789]: pgmap v2707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:24 smithi044 ceph-mon[32404]: pgmap v2707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:24 smithi116 ceph-mon[37097]: pgmap v2707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:26.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:26.521+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:49:26.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:26.521+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:49:26.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:26.521+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:26.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4110.666381 secs 2024-07-26T09:49:26.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:26 smithi044 ceph-mon[32404]: pgmap v2708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4110.666381 secs 2024-07-26T09:49:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:26 smithi116 ceph-mon[37097]: pgmap v2708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4110.666381 secs 2024-07-26T09:49:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:26 smithi171 ceph-mon[37789]: pgmap v2708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:28 smithi044 ceph-mon[32404]: pgmap v2709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:49:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:28 smithi116 ceph-mon[37097]: pgmap v2709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:49:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:28 smithi171 ceph-mon[37789]: pgmap v2709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:49:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:30 smithi044 ceph-mon[32404]: pgmap v2710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:30 smithi116 ceph-mon[37097]: pgmap v2710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:30 smithi171 ceph-mon[37789]: pgmap v2710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:49:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4115.666550 secs 2024-07-26T09:49:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4115.666550 secs 2024-07-26T09:49:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4115.666550 secs 2024-07-26T09:49:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:32 smithi044 ceph-mon[32404]: pgmap v2711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:32 smithi116 ceph-mon[37097]: pgmap v2711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:32 smithi171 ceph-mon[37789]: pgmap v2711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:49:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:34 smithi116 ceph-mon[37097]: pgmap v2712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:34 smithi171 ceph-mon[37789]: pgmap v2712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:34 smithi044 ceph-mon[32404]: pgmap v2712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4120.666707 secs 2024-07-26T09:49:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:36 smithi171 ceph-mon[37789]: pgmap v2713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4120.666707 secs 2024-07-26T09:49:37.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:36 smithi044 ceph-mon[32404]: pgmap v2713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4120.666707 secs 2024-07-26T09:49:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:36 smithi116 ceph-mon[37097]: pgmap v2713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:38 smithi171 ceph-mon[37789]: pgmap v2714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:38 smithi044 ceph-mon[32404]: pgmap v2714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:38 smithi116 ceph-mon[37097]: pgmap v2714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:40 smithi171 ceph-mon[37789]: pgmap v2715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:40 smithi044 ceph-mon[32404]: pgmap v2715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:40 smithi116 ceph-mon[37097]: pgmap v2715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:41.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:41.523+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:49:41.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:41.523+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:49:41.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:41.523+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4125.666881 secs 2024-07-26T09:49:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4125.666881 secs 2024-07-26T09:49:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4125.666881 secs 2024-07-26T09:49:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:42 smithi171 ceph-mon[37789]: pgmap v2716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:42 smithi044 ceph-mon[32404]: pgmap v2716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:42 smithi116 ceph-mon[37097]: pgmap v2716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:44 smithi171 ceph-mon[37789]: pgmap v2717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:44 smithi044 ceph-mon[32404]: pgmap v2717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:44 smithi116 ceph-mon[37097]: pgmap v2717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:47.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4130.667003 secs 2024-07-26T09:49:47.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:46 smithi171 ceph-mon[37789]: pgmap v2718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4130.667003 secs 2024-07-26T09:49:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:46 smithi044 ceph-mon[32404]: pgmap v2718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4130.667003 secs 2024-07-26T09:49:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:46 smithi116 ceph-mon[37097]: pgmap v2718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:48 smithi171 ceph-mon[37789]: pgmap v2719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:49.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:48 smithi044 ceph-mon[32404]: pgmap v2719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:48 smithi116 ceph-mon[37097]: pgmap v2719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:50 smithi171 ceph-mon[37789]: pgmap v2720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:50 smithi044 ceph-mon[32404]: pgmap v2720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:50 smithi116 ceph-mon[37097]: pgmap v2720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:49:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4135.667167 secs 2024-07-26T09:49:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4135.667167 secs 2024-07-26T09:49:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4135.667167 secs 2024-07-26T09:49:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:52 smithi044 ceph-mon[32404]: pgmap v2721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:52 smithi116 ceph-mon[37097]: pgmap v2721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:52 smithi171 ceph-mon[37789]: pgmap v2721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:49:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:54 smithi044 ceph-mon[32404]: pgmap v2722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:54 smithi116 ceph-mon[37097]: pgmap v2722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:54 smithi171 ceph-mon[37789]: pgmap v2722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:49:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:56.532+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:49:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:56.532+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:49:56.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:49:56.532+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4140.667331 secs 2024-07-26T09:49:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:56 smithi044 ceph-mon[32404]: pgmap v2723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4140.667331 secs 2024-07-26T09:49:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:56 smithi116 ceph-mon[37097]: pgmap v2723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4140.667331 secs 2024-07-26T09:49:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:56 smithi171 ceph-mon[37789]: pgmap v2723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:49:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:49:58 smithi044 ceph-mon[32404]: pgmap v2724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:49:58 smithi116 ceph-mon[37097]: pgmap v2724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:49:59.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:49:58 smithi171 ceph-mon[37789]: pgmap v2724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:00 smithi044 ceph-mon[32404]: pgmap v2725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:50:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:00 smithi116 ceph-mon[37097]: pgmap v2725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:50:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:00 smithi171 ceph-mon[37789]: pgmap v2725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T09:50:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4145.667496 secs 2024-07-26T09:50:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4145.667496 secs 2024-07-26T09:50:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4145.667496 secs 2024-07-26T09:50:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:02 smithi044 ceph-mon[32404]: pgmap v2726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:02 smithi116 ceph-mon[37097]: pgmap v2726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:02 smithi171 ceph-mon[37789]: pgmap v2726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:04 smithi044 ceph-mon[32404]: pgmap v2727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:04 smithi116 ceph-mon[37097]: pgmap v2727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:04 smithi171 ceph-mon[37789]: pgmap v2727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4150.667661 secs 2024-07-26T09:50:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:06 smithi044 ceph-mon[32404]: pgmap v2728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4150.667661 secs 2024-07-26T09:50:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:06 smithi116 ceph-mon[37097]: pgmap v2728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4150.667661 secs 2024-07-26T09:50:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:06 smithi171 ceph-mon[37789]: pgmap v2728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:09.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:08 smithi044 ceph-mon[32404]: pgmap v2729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:08 smithi116 ceph-mon[37097]: pgmap v2729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:08 smithi171 ceph-mon[37789]: pgmap v2729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:10 smithi044 ceph-mon[32404]: pgmap v2730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:10 smithi116 ceph-mon[37097]: pgmap v2730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:10 smithi171 ceph-mon[37789]: pgmap v2730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:11.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:11.536+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:50:11.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:11.536+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:11.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:11.536+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:50:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4155.667796 secs 2024-07-26T09:50:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4155.667796 secs 2024-07-26T09:50:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4155.667796 secs 2024-07-26T09:50:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:12 smithi044 ceph-mon[32404]: pgmap v2731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:12 smithi116 ceph-mon[37097]: pgmap v2731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:12 smithi171 ceph-mon[37789]: pgmap v2731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:14 smithi044 ceph-mon[32404]: pgmap v2732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:15.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:14 smithi116 ceph-mon[37097]: pgmap v2732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:14 smithi171 ceph-mon[37789]: pgmap v2732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:15.405+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:50:15.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:15.405+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:15.406+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:15.407+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:50:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4160.667944 secs 2024-07-26T09:50:17.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:16 smithi044 ceph-mon[32404]: pgmap v2733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4160.667944 secs 2024-07-26T09:50:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:16 smithi116 ceph-mon[37097]: pgmap v2733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4160.667944 secs 2024-07-26T09:50:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:16 smithi171 ceph-mon[37789]: pgmap v2733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:19.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:18 smithi044 ceph-mon[32404]: pgmap v2734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:18 smithi116 ceph-mon[37097]: pgmap v2734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:18 smithi171 ceph-mon[37789]: pgmap v2734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:20 smithi044 ceph-mon[32404]: pgmap v2735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:20 smithi116 ceph-mon[37097]: pgmap v2735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:20 smithi171 ceph-mon[37789]: pgmap v2735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4165.668106 secs 2024-07-26T09:50:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4165.668106 secs 2024-07-26T09:50:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4165.668106 secs 2024-07-26T09:50:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:22 smithi044 ceph-mon[32404]: pgmap v2736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:22 smithi116 ceph-mon[37097]: pgmap v2736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:22 smithi171 ceph-mon[37789]: pgmap v2736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:50:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:23 smithi044 ceph-mon[32404]: pgmap v2737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:50:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:23 smithi044 ceph-mon[32404]: pgmap v2738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:50:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:50:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:23 smithi116 ceph-mon[37097]: pgmap v2737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:50:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:23 smithi116 ceph-mon[37097]: pgmap v2738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:50:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:50:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:23 smithi171 ceph-mon[37789]: pgmap v2737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:50:24.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:23 smithi171 ceph-mon[37789]: pgmap v2738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:50:24.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:50:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:26 smithi171 ceph-mon[37789]: pgmap v2739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:26 smithi044 ceph-mon[32404]: pgmap v2739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:26 smithi116 ceph-mon[37097]: pgmap v2739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:26.544+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:50:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:26.544+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:26.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:26.544+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:50:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4170.668253 secs 2024-07-26T09:50:27.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4170.668253 secs 2024-07-26T09:50:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4170.668253 secs 2024-07-26T09:50:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:28 smithi171 ceph-mon[37789]: pgmap v2740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:28 smithi044 ceph-mon[32404]: pgmap v2740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:28 smithi116 ceph-mon[37097]: pgmap v2740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:30 smithi171 ceph-mon[37789]: pgmap v2741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:30 smithi044 ceph-mon[32404]: pgmap v2741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:30 smithi116 ceph-mon[37097]: pgmap v2741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:50:32.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4175.668408 secs 2024-07-26T09:50:32.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:32 smithi116 ceph-mon[37097]: pgmap v2742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:50:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4175.668408 secs 2024-07-26T09:50:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:32 smithi171 ceph-mon[37789]: pgmap v2742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:50:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4175.668408 secs 2024-07-26T09:50:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:32 smithi044 ceph-mon[32404]: pgmap v2742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:34 smithi171 ceph-mon[37789]: pgmap v2743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:50:34.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:34 smithi044 ceph-mon[32404]: pgmap v2743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:50:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:34 smithi116 ceph-mon[37097]: pgmap v2743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:50:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:36 smithi171 ceph-mon[37789]: pgmap v2744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:36 smithi044 ceph-mon[32404]: pgmap v2744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:36 smithi116 ceph-mon[37097]: pgmap v2744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:50:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4180.668572 secs 2024-07-26T09:50:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4180.668572 secs 2024-07-26T09:50:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4180.668572 secs 2024-07-26T09:50:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:38 smithi171 ceph-mon[37789]: pgmap v2745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:38 smithi044 ceph-mon[32404]: pgmap v2745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:38 smithi116 ceph-mon[37097]: pgmap v2745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:40 smithi171 ceph-mon[37789]: pgmap v2746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:40 smithi044 ceph-mon[32404]: pgmap v2746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:40 smithi116 ceph-mon[37097]: pgmap v2746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:41.548+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:50:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:41.548+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:41.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:41.548+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:50:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4185.668736 secs 2024-07-26T09:50:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:42 smithi044 ceph-mon[32404]: pgmap v2747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4185.668736 secs 2024-07-26T09:50:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:42 smithi116 ceph-mon[37097]: pgmap v2747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4185.668736 secs 2024-07-26T09:50:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:42 smithi171 ceph-mon[37789]: pgmap v2747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:44.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:44 smithi044 ceph-mon[32404]: pgmap v2748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:50:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:44 smithi116 ceph-mon[37097]: pgmap v2748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:50:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:44 smithi171 ceph-mon[37789]: pgmap v2748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:50:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:46 smithi044 ceph-mon[32404]: pgmap v2749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4190.668900 secs 2024-07-26T09:50:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:46 smithi116 ceph-mon[37097]: pgmap v2749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4190.668900 secs 2024-07-26T09:50:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:46 smithi171 ceph-mon[37789]: pgmap v2749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:46.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4190.668900 secs 2024-07-26T09:50:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:48 smithi044 ceph-mon[32404]: pgmap v2750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:48 smithi116 ceph-mon[37097]: pgmap v2750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:48 smithi171 ceph-mon[37789]: pgmap v2750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:50 smithi044 ceph-mon[32404]: pgmap v2751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:50 smithi116 ceph-mon[37097]: pgmap v2751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:50 smithi171 ceph-mon[37789]: pgmap v2751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4195.669072 secs 2024-07-26T09:50:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4195.669072 secs 2024-07-26T09:50:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4195.669072 secs 2024-07-26T09:50:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:52 smithi044 ceph-mon[32404]: pgmap v2752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:52 smithi116 ceph-mon[37097]: pgmap v2752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:52 smithi171 ceph-mon[37789]: pgmap v2752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:54 smithi044 ceph-mon[32404]: pgmap v2753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:50:54.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:54 smithi116 ceph-mon[37097]: pgmap v2753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:50:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:54 smithi171 ceph-mon[37789]: pgmap v2753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:50:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:56 smithi044 ceph-mon[32404]: pgmap v2754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4200.669208 secs 2024-07-26T09:50:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:56 smithi116 ceph-mon[37097]: pgmap v2754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4200.669208 secs 2024-07-26T09:50:56.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:56.557+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:50:56.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:56.557+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:56.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:50:56.557+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:50:56.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:56 smithi171 ceph-mon[37789]: pgmap v2754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:50:56.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4200.669208 secs 2024-07-26T09:50:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:50:58 smithi044 ceph-mon[32404]: pgmap v2755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:50:58 smithi116 ceph-mon[37097]: pgmap v2755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:50:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:50:58 smithi171 ceph-mon[37789]: pgmap v2755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:00 smithi044 ceph-mon[32404]: pgmap v2756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:00 smithi116 ceph-mon[37097]: pgmap v2756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:00 smithi171 ceph-mon[37789]: pgmap v2756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4205.669367 secs 2024-07-26T09:51:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4205.669367 secs 2024-07-26T09:51:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4205.669367 secs 2024-07-26T09:51:02.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:02 smithi116 ceph-mon[37097]: pgmap v2757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:02 smithi044 ceph-mon[32404]: pgmap v2757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:02 smithi171 ceph-mon[37789]: pgmap v2757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:04 smithi044 ceph-mon[32404]: pgmap v2758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:04 smithi116 ceph-mon[37097]: pgmap v2758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:04 smithi171 ceph-mon[37789]: pgmap v2758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:06 smithi044 ceph-mon[32404]: pgmap v2759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4210.669520 secs 2024-07-26T09:51:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:06 smithi116 ceph-mon[37097]: pgmap v2759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4210.669520 secs 2024-07-26T09:51:06.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:06 smithi171 ceph-mon[37789]: pgmap v2759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:06.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4210.669520 secs 2024-07-26T09:51:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:08 smithi044 ceph-mon[32404]: pgmap v2760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:08 smithi116 ceph-mon[37097]: pgmap v2760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:08 smithi171 ceph-mon[37789]: pgmap v2760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:10 smithi044 ceph-mon[32404]: pgmap v2761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:10 smithi116 ceph-mon[37097]: pgmap v2761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:10 smithi171 ceph-mon[37789]: pgmap v2761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4215.669692 secs 2024-07-26T09:51:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4215.669692 secs 2024-07-26T09:51:11.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:11.560+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:51:11.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:11.560+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:11.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:11.560+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:51:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4215.669692 secs 2024-07-26T09:51:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:12 smithi044 ceph-mon[32404]: pgmap v2762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:12 smithi116 ceph-mon[37097]: pgmap v2762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:12 smithi171 ceph-mon[37789]: pgmap v2762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:14 smithi044 ceph-mon[32404]: pgmap v2763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:14 smithi116 ceph-mon[37097]: pgmap v2763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:14 smithi171 ceph-mon[37789]: pgmap v2763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:16 smithi044 ceph-mon[32404]: pgmap v2764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4220.669862 secs 2024-07-26T09:51:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:16 smithi116 ceph-mon[37097]: pgmap v2764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4220.669862 secs 2024-07-26T09:51:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:16 smithi171 ceph-mon[37789]: pgmap v2764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4220.669862 secs 2024-07-26T09:51:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:18 smithi044 ceph-mon[32404]: pgmap v2765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:18 smithi116 ceph-mon[37097]: pgmap v2765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:18 smithi171 ceph-mon[37789]: pgmap v2765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:20 smithi044 ceph-mon[32404]: pgmap v2766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:20 smithi116 ceph-mon[37097]: pgmap v2766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:20 smithi171 ceph-mon[37789]: pgmap v2766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4225.670010 secs 2024-07-26T09:51:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4225.670010 secs 2024-07-26T09:51:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4225.670010 secs 2024-07-26T09:51:22.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:22 smithi044 ceph-mon[32404]: pgmap v2767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:22 smithi116 ceph-mon[37097]: pgmap v2767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:22 smithi171 ceph-mon[37789]: pgmap v2767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:24.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:24 smithi044 ceph-mon[32404]: pgmap v2768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:24.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:24.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:24.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:51:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:24 smithi116 ceph-mon[37097]: pgmap v2768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:24.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:51:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:24 smithi171 ceph-mon[37789]: pgmap v2768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:24.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:51:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:25 smithi044 ceph-mon[32404]: pgmap v2769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:51:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:25 smithi044 ceph-mon[32404]: pgmap v2770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:25 smithi116 ceph-mon[37097]: pgmap v2769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:51:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:25 smithi116 ceph-mon[37097]: pgmap v2770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:25 smithi171 ceph-mon[37789]: pgmap v2769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:51:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:25 smithi171 ceph-mon[37789]: pgmap v2770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:26.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:26.569+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:51:26.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:26.569+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:26.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:26.569+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:51:26.572 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4230.670160 secs 2024-07-26T09:51:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4230.670160 secs 2024-07-26T09:51:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4230.670160 secs 2024-07-26T09:51:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:27 smithi171 ceph-mon[37789]: pgmap v2771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:27 smithi044 ceph-mon[32404]: pgmap v2771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:27 smithi116 ceph-mon[37097]: pgmap v2771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:29 smithi171 ceph-mon[37789]: pgmap v2772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:29 smithi044 ceph-mon[32404]: pgmap v2772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:29 smithi116 ceph-mon[37097]: pgmap v2772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:31 smithi171 ceph-mon[37789]: pgmap v2773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4235.670320 secs 2024-07-26T09:51:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:31 smithi044 ceph-mon[32404]: pgmap v2773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4235.670320 secs 2024-07-26T09:51:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:31 smithi116 ceph-mon[37097]: pgmap v2773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:51:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4235.670320 secs 2024-07-26T09:51:33.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:33 smithi116 ceph-mon[37097]: pgmap v2774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:33 smithi171 ceph-mon[37789]: pgmap v2774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:33 smithi044 ceph-mon[32404]: pgmap v2774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:51:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:35 smithi171 ceph-mon[37789]: pgmap v2775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:51:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:35 smithi044 ceph-mon[32404]: pgmap v2775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:51:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:35 smithi116 ceph-mon[37097]: pgmap v2775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:51:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4240.670459 secs 2024-07-26T09:51:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4240.670459 secs 2024-07-26T09:51:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4240.670459 secs 2024-07-26T09:51:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:37 smithi171 ceph-mon[37789]: pgmap v2776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:37 smithi044 ceph-mon[32404]: pgmap v2776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:37 smithi116 ceph-mon[37097]: pgmap v2776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:39 smithi171 ceph-mon[37789]: pgmap v2777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:39 smithi044 ceph-mon[32404]: pgmap v2777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:39 smithi116 ceph-mon[37097]: pgmap v2777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:41.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:41.572+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:51:41.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:41.572+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:51:41.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:41.572+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:41 smithi171 ceph-mon[37789]: pgmap v2778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4245.670672 secs 2024-07-26T09:51:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:41 smithi044 ceph-mon[32404]: pgmap v2778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4245.670672 secs 2024-07-26T09:51:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:41 smithi116 ceph-mon[37097]: pgmap v2778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4245.670672 secs 2024-07-26T09:51:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:43 smithi171 ceph-mon[37789]: pgmap v2779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:43 smithi044 ceph-mon[32404]: pgmap v2779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:43 smithi116 ceph-mon[37097]: pgmap v2779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:45 smithi044 ceph-mon[32404]: pgmap v2780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:45 smithi116 ceph-mon[37097]: pgmap v2780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:45 smithi171 ceph-mon[37789]: pgmap v2780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4250.670818 secs 2024-07-26T09:51:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4250.670818 secs 2024-07-26T09:51:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4250.670818 secs 2024-07-26T09:51:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:47 smithi044 ceph-mon[32404]: pgmap v2781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:47 smithi116 ceph-mon[37097]: pgmap v2781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:47 smithi171 ceph-mon[37789]: pgmap v2781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:49 smithi044 ceph-mon[32404]: pgmap v2782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:49 smithi116 ceph-mon[37097]: pgmap v2782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:49 smithi171 ceph-mon[37789]: pgmap v2782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:51 smithi044 ceph-mon[32404]: pgmap v2783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4255.670993 secs 2024-07-26T09:51:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:51 smithi116 ceph-mon[37097]: pgmap v2783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4255.670993 secs 2024-07-26T09:51:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:51 smithi171 ceph-mon[37789]: pgmap v2783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4255.670993 secs 2024-07-26T09:51:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:53 smithi044 ceph-mon[32404]: pgmap v2784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:53 smithi116 ceph-mon[37097]: pgmap v2784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:53.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:53 smithi171 ceph-mon[37789]: pgmap v2784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:55 smithi044 ceph-mon[32404]: pgmap v2785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:55 smithi116 ceph-mon[37097]: pgmap v2785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:55 smithi171 ceph-mon[37789]: pgmap v2785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:51:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:56.581+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:51:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:56.581+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:56.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:51:56.581+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:51:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4260.671136 secs 2024-07-26T09:51:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4260.671136 secs 2024-07-26T09:51:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4260.671136 secs 2024-07-26T09:51:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:57 smithi044 ceph-mon[32404]: pgmap v2786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:57 smithi116 ceph-mon[37097]: pgmap v2786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:57 smithi171 ceph-mon[37789]: pgmap v2786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:51:59.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:51:59 smithi044 ceph-mon[32404]: pgmap v2787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:51:59 smithi116 ceph-mon[37097]: pgmap v2787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:51:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:51:59 smithi171 ceph-mon[37789]: pgmap v2787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:01 smithi044 ceph-mon[32404]: pgmap v2788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4265.671292 secs 2024-07-26T09:52:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:01 smithi116 ceph-mon[37097]: pgmap v2788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4265.671292 secs 2024-07-26T09:52:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:01 smithi171 ceph-mon[37789]: pgmap v2788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4265.671292 secs 2024-07-26T09:52:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:03 smithi044 ceph-mon[32404]: pgmap v2789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:03 smithi116 ceph-mon[37097]: pgmap v2789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:03 smithi171 ceph-mon[37789]: pgmap v2789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:05 smithi044 ceph-mon[32404]: pgmap v2790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:05 smithi116 ceph-mon[37097]: pgmap v2790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:05 smithi171 ceph-mon[37789]: pgmap v2790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:06.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4270.671458 secs 2024-07-26T09:52:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4270.671458 secs 2024-07-26T09:52:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4270.671458 secs 2024-07-26T09:52:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:07 smithi044 ceph-mon[32404]: pgmap v2791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:07 smithi116 ceph-mon[37097]: pgmap v2791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:07 smithi171 ceph-mon[37789]: pgmap v2791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:09 smithi044 ceph-mon[32404]: pgmap v2792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:09 smithi116 ceph-mon[37097]: pgmap v2792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:09 smithi171 ceph-mon[37789]: pgmap v2792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:11.584+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:52:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:11.584+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:11.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:11.584+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:52:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:11 smithi044 ceph-mon[32404]: pgmap v2793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4275.671665 secs 2024-07-26T09:52:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:11 smithi116 ceph-mon[37097]: pgmap v2793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4275.671665 secs 2024-07-26T09:52:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:11 smithi171 ceph-mon[37789]: pgmap v2793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4275.671665 secs 2024-07-26T09:52:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:13 smithi044 ceph-mon[32404]: pgmap v2794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:13 smithi116 ceph-mon[37097]: pgmap v2794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:13 smithi171 ceph-mon[37789]: pgmap v2794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:15 smithi044 ceph-mon[32404]: pgmap v2795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:15 smithi116 ceph-mon[37097]: pgmap v2795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:15 smithi171 ceph-mon[37789]: pgmap v2795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4280.671828 secs 2024-07-26T09:52:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4280.671828 secs 2024-07-26T09:52:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4280.671828 secs 2024-07-26T09:52:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:17 smithi044 ceph-mon[32404]: pgmap v2796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:17 smithi116 ceph-mon[37097]: pgmap v2796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:17 smithi171 ceph-mon[37789]: pgmap v2796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:19 smithi044 ceph-mon[32404]: pgmap v2797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:19 smithi116 ceph-mon[37097]: pgmap v2797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:19 smithi171 ceph-mon[37789]: pgmap v2797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:52:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:21 smithi044 ceph-mon[32404]: pgmap v2798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4285.671987 secs 2024-07-26T09:52:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:21 smithi116 ceph-mon[37097]: pgmap v2798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4285.671987 secs 2024-07-26T09:52:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:21 smithi171 ceph-mon[37789]: pgmap v2798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4285.671987 secs 2024-07-26T09:52:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:23 smithi044 ceph-mon[32404]: pgmap v2799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:23 smithi116 ceph-mon[37097]: pgmap v2799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:23 smithi171 ceph-mon[37789]: pgmap v2799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:25 smithi044 ceph-mon[32404]: pgmap v2800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:25 smithi116 ceph-mon[37097]: pgmap v2800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:25 smithi171 ceph-mon[37789]: pgmap v2800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:26.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:26.585+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:52:26.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:26.585+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:52:26.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:26.585+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4290.672089 secs 2024-07-26T09:52:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:26 smithi044 ceph-mon[32404]: pgmap v2801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:52:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:26 smithi044 ceph-mon[32404]: pgmap v2802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:52:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4290.672089 secs 2024-07-26T09:52:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:26 smithi116 ceph-mon[37097]: pgmap v2801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:52:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:26 smithi116 ceph-mon[37097]: pgmap v2802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:52:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4290.672089 secs 2024-07-26T09:52:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:26 smithi171 ceph-mon[37789]: pgmap v2801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:52:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:26 smithi171 ceph-mon[37789]: pgmap v2802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:52:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:28 smithi044 ceph-mon[32404]: pgmap v2803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:28 smithi116 ceph-mon[37097]: pgmap v2803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:28 smithi171 ceph-mon[37789]: pgmap v2803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:29 smithi044 ceph-mon[32404]: pgmap v2804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:29 smithi116 ceph-mon[37097]: pgmap v2804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:29 smithi171 ceph-mon[37789]: pgmap v2804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4295.672256 secs 2024-07-26T09:52:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4295.672256 secs 2024-07-26T09:52:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4295.672256 secs 2024-07-26T09:52:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:32 smithi044 ceph-mon[32404]: pgmap v2805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:32 smithi116 ceph-mon[37097]: pgmap v2805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:32 smithi171 ceph-mon[37789]: pgmap v2805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:52:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:33 smithi044 ceph-mon[32404]: pgmap v2806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:33 smithi116 ceph-mon[37097]: pgmap v2806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:33 smithi171 ceph-mon[37789]: pgmap v2806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:52:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:35 smithi044 ceph-mon[32404]: pgmap v2807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:52:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:35 smithi116 ceph-mon[37097]: pgmap v2807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:52:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:35 smithi171 ceph-mon[37789]: pgmap v2807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:52:36.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4300.672403 secs 2024-07-26T09:52:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4300.672403 secs 2024-07-26T09:52:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4300.672403 secs 2024-07-26T09:52:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:37 smithi044 ceph-mon[32404]: pgmap v2808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:38.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:37 smithi116 ceph-mon[37097]: pgmap v2808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:37 smithi171 ceph-mon[37789]: pgmap v2808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:39 smithi044 ceph-mon[32404]: pgmap v2809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:39 smithi116 ceph-mon[37097]: pgmap v2809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:39 smithi171 ceph-mon[37789]: pgmap v2809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4305.672570 secs 2024-07-26T09:52:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4305.672570 secs 2024-07-26T09:52:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4305.672570 secs 2024-07-26T09:52:41.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:41.589+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:52:41.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:41.589+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:52:41.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:41.589+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:42 smithi044 ceph-mon[32404]: pgmap v2810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:42 smithi116 ceph-mon[37097]: pgmap v2810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:42 smithi171 ceph-mon[37789]: pgmap v2810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:43 smithi044 ceph-mon[32404]: pgmap v2811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:52:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:43 smithi116 ceph-mon[37097]: pgmap v2811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:52:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:43 smithi171 ceph-mon[37789]: pgmap v2811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:52:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:45 smithi044 ceph-mon[32404]: pgmap v2812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:45 smithi116 ceph-mon[37097]: pgmap v2812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:45 smithi171 ceph-mon[37789]: pgmap v2812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4310.672734 secs 2024-07-26T09:52:47.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4310.672734 secs 2024-07-26T09:52:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4310.672734 secs 2024-07-26T09:52:48.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:47 smithi044 ceph-mon[32404]: pgmap v2813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:47 smithi116 ceph-mon[37097]: pgmap v2813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:47 smithi171 ceph-mon[37789]: pgmap v2813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:49 smithi044 ceph-mon[32404]: pgmap v2814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:49 smithi116 ceph-mon[37097]: pgmap v2814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:49 smithi171 ceph-mon[37789]: pgmap v2814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:52:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4315.672879 secs 2024-07-26T09:52:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4315.672879 secs 2024-07-26T09:52:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4315.672879 secs 2024-07-26T09:52:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:52 smithi044 ceph-mon[32404]: pgmap v2815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:52 smithi116 ceph-mon[37097]: pgmap v2815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:52 smithi171 ceph-mon[37789]: pgmap v2815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:54.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:53 smithi044 ceph-mon[32404]: pgmap v2816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:52:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:53 smithi116 ceph-mon[37097]: pgmap v2816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:52:54.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:53 smithi171 ceph-mon[37789]: pgmap v2816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:52:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:55 smithi044 ceph-mon[32404]: pgmap v2817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:55 smithi116 ceph-mon[37097]: pgmap v2817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:55 smithi171 ceph-mon[37789]: pgmap v2817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:52:56.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:56.598+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:52:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:56.598+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:52:56.598+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:52:56.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4320.673022 secs 2024-07-26T09:52:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4320.673022 secs 2024-07-26T09:52:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4320.673022 secs 2024-07-26T09:52:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:57 smithi044 ceph-mon[32404]: pgmap v2818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:57 smithi116 ceph-mon[37097]: pgmap v2818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:52:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:57 smithi171 ceph-mon[37789]: pgmap v2818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:52:59 smithi044 ceph-mon[32404]: pgmap v2819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:52:59 smithi116 ceph-mon[37097]: pgmap v2819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:52:59 smithi171 ceph-mon[37789]: pgmap v2819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4325.673191 secs 2024-07-26T09:53:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4325.673191 secs 2024-07-26T09:53:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4325.673191 secs 2024-07-26T09:53:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:02 smithi044 ceph-mon[32404]: pgmap v2820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:02 smithi116 ceph-mon[37097]: pgmap v2820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:02 smithi171 ceph-mon[37789]: pgmap v2820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:03 smithi044 ceph-mon[32404]: pgmap v2821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:03 smithi116 ceph-mon[37097]: pgmap v2821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:03 smithi171 ceph-mon[37789]: pgmap v2821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:05 smithi044 ceph-mon[32404]: pgmap v2822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:05 smithi116 ceph-mon[37097]: pgmap v2822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:05 smithi171 ceph-mon[37789]: pgmap v2822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:06.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4330.673330 secs 2024-07-26T09:53:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4330.673330 secs 2024-07-26T09:53:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4330.673330 secs 2024-07-26T09:53:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:07 smithi044 ceph-mon[32404]: pgmap v2823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:07 smithi116 ceph-mon[37097]: pgmap v2823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:07 smithi171 ceph-mon[37789]: pgmap v2823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:09 smithi044 ceph-mon[32404]: pgmap v2824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:09 smithi116 ceph-mon[37097]: pgmap v2824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:09 smithi171 ceph-mon[37789]: pgmap v2824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4335.673502 secs 2024-07-26T09:53:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4335.673502 secs 2024-07-26T09:53:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4335.673502 secs 2024-07-26T09:53:11.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:11.600+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:53:11.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:11.601+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:11.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:11.601+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:53:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:12 smithi044 ceph-mon[32404]: pgmap v2825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:12 smithi116 ceph-mon[37097]: pgmap v2825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:12 smithi171 ceph-mon[37789]: pgmap v2825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:13 smithi044 ceph-mon[32404]: pgmap v2826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:13 smithi116 ceph-mon[37097]: pgmap v2826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:13 smithi171 ceph-mon[37789]: pgmap v2826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:15.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:15.438+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:53:15.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:15.438+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:53:15.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:15.438+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:53:15.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:15.439+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:53:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:15 smithi044 ceph-mon[32404]: pgmap v2827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:15 smithi116 ceph-mon[37097]: pgmap v2827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:15 smithi171 ceph-mon[37789]: pgmap v2827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4340.673696 secs 2024-07-26T09:53:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4340.673696 secs 2024-07-26T09:53:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4340.673696 secs 2024-07-26T09:53:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:17 smithi044 ceph-mon[32404]: pgmap v2828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:17 smithi116 ceph-mon[37097]: pgmap v2828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:17 smithi171 ceph-mon[37789]: pgmap v2828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:19 smithi044 ceph-mon[32404]: pgmap v2829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:19 smithi116 ceph-mon[37097]: pgmap v2829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:19 smithi171 ceph-mon[37789]: pgmap v2829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4345.673861 secs 2024-07-26T09:53:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4345.673861 secs 2024-07-26T09:53:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4345.673861 secs 2024-07-26T09:53:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:22 smithi044 ceph-mon[32404]: pgmap v2830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:22 smithi116 ceph-mon[37097]: pgmap v2830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:22 smithi171 ceph-mon[37789]: pgmap v2830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:23 smithi044 ceph-mon[32404]: pgmap v2831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:23 smithi116 ceph-mon[37097]: pgmap v2831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:23 smithi171 ceph-mon[37789]: pgmap v2831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T09:53:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:25 smithi044 ceph-mon[32404]: pgmap v2832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:25 smithi116 ceph-mon[37097]: pgmap v2832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:25 smithi171 ceph-mon[37789]: pgmap v2832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:26.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:26.610+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:53:26.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:26.610+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:53:26.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:26.610+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:26.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4350.674025 secs 2024-07-26T09:53:26.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4350.674025 secs 2024-07-26T09:53:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4350.674025 secs 2024-07-26T09:53:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:27.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: pgmap v2833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: pgmap v2834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:27.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: pgmap v2833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: pgmap v2834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:28.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:53:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: pgmap v2833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: pgmap v2834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:28.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:53:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:29 smithi044 ceph-mon[32404]: pgmap v2835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:29 smithi116 ceph-mon[37097]: pgmap v2835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:29 smithi171 ceph-mon[37789]: pgmap v2835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4355.674193 secs 2024-07-26T09:53:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:31 smithi044 ceph-mon[32404]: pgmap v2836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4355.674193 secs 2024-07-26T09:53:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:31 smithi116 ceph-mon[37097]: pgmap v2836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4355.674193 secs 2024-07-26T09:53:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:31 smithi171 ceph-mon[37789]: pgmap v2836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:53:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:33 smithi044 ceph-mon[32404]: pgmap v2837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:53:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:33 smithi116 ceph-mon[37097]: pgmap v2837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:53:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:33 smithi171 ceph-mon[37789]: pgmap v2837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:53:36.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:35 smithi171 ceph-mon[37789]: pgmap v2838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:35 smithi044 ceph-mon[32404]: pgmap v2838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:35 smithi116 ceph-mon[37097]: pgmap v2838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4360.674345 secs 2024-07-26T09:53:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4360.674345 secs 2024-07-26T09:53:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4360.674345 secs 2024-07-26T09:53:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:37 smithi044 ceph-mon[32404]: pgmap v2839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:37 smithi116 ceph-mon[37097]: pgmap v2839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:37 smithi171 ceph-mon[37789]: pgmap v2839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T09:53:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:39 smithi044 ceph-mon[32404]: pgmap v2840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:39 smithi116 ceph-mon[37097]: pgmap v2840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:39 smithi171 ceph-mon[37789]: pgmap v2840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:41.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:41.614+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:53:41.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:41.614+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:53:41.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:41.614+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4365.674494 secs 2024-07-26T09:53:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:41 smithi044 ceph-mon[32404]: pgmap v2841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:53:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4365.674494 secs 2024-07-26T09:53:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:41 smithi116 ceph-mon[37097]: pgmap v2841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:53:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4365.674494 secs 2024-07-26T09:53:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:41 smithi171 ceph-mon[37789]: pgmap v2841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:53:44.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:43 smithi044 ceph-mon[32404]: pgmap v2842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:43 smithi116 ceph-mon[37097]: pgmap v2842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:43 smithi171 ceph-mon[37789]: pgmap v2842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:53:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:45 smithi044 ceph-mon[32404]: pgmap v2843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:45 smithi116 ceph-mon[37097]: pgmap v2843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:45 smithi171 ceph-mon[37789]: pgmap v2843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4370.674665 secs 2024-07-26T09:53:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4370.674665 secs 2024-07-26T09:53:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4370.674665 secs 2024-07-26T09:53:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:47 smithi044 ceph-mon[32404]: pgmap v2844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:47 smithi116 ceph-mon[37097]: pgmap v2844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:47 smithi171 ceph-mon[37789]: pgmap v2844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:49 smithi044 ceph-mon[32404]: pgmap v2845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:49 smithi116 ceph-mon[37097]: pgmap v2845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:49 smithi171 ceph-mon[37789]: pgmap v2845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4375.674832 secs 2024-07-26T09:53:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:51 smithi044 ceph-mon[32404]: pgmap v2846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4375.674832 secs 2024-07-26T09:53:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:51 smithi116 ceph-mon[37097]: pgmap v2846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4375.674832 secs 2024-07-26T09:53:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:51 smithi171 ceph-mon[37789]: pgmap v2846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:53:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:53 smithi044 ceph-mon[32404]: pgmap v2847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:53 smithi116 ceph-mon[37097]: pgmap v2847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:53 smithi171 ceph-mon[37789]: pgmap v2847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:55 smithi044 ceph-mon[32404]: pgmap v2848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:55 smithi116 ceph-mon[37097]: pgmap v2848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:55 smithi171 ceph-mon[37789]: pgmap v2848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:53:56.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:56.623+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:53:56.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:56.623+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:53:56.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:53:56.623+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4380.674973 secs 2024-07-26T09:53:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4380.674973 secs 2024-07-26T09:53:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4380.674973 secs 2024-07-26T09:53:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:57 smithi044 ceph-mon[32404]: pgmap v2849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:57 smithi116 ceph-mon[37097]: pgmap v2849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:53:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:57 smithi171 ceph-mon[37789]: pgmap v2849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:00.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:53:59 smithi044 ceph-mon[32404]: pgmap v2850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:53:59 smithi116 ceph-mon[37097]: pgmap v2850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:53:59 smithi171 ceph-mon[37789]: pgmap v2850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4385.675086 secs 2024-07-26T09:54:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:01 smithi044 ceph-mon[32404]: pgmap v2851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:54:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4385.675086 secs 2024-07-26T09:54:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:01 smithi116 ceph-mon[37097]: pgmap v2851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:54:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4385.675086 secs 2024-07-26T09:54:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:01 smithi171 ceph-mon[37789]: pgmap v2851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:54:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:04 smithi044 ceph-mon[32404]: pgmap v2852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:04 smithi116 ceph-mon[37097]: pgmap v2852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:04 smithi171 ceph-mon[37789]: pgmap v2852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:06 smithi044 ceph-mon[32404]: pgmap v2853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:06 smithi116 ceph-mon[37097]: pgmap v2853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:06 smithi171 ceph-mon[37789]: pgmap v2853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4390.675258 secs 2024-07-26T09:54:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4390.675258 secs 2024-07-26T09:54:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4390.675258 secs 2024-07-26T09:54:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:08 smithi171 ceph-mon[37789]: pgmap v2854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:08 smithi044 ceph-mon[32404]: pgmap v2854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:08 smithi116 ceph-mon[37097]: pgmap v2854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:10 smithi171 ceph-mon[37789]: pgmap v2855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:10 smithi044 ceph-mon[32404]: pgmap v2855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:10 smithi116 ceph-mon[37097]: pgmap v2855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:11.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:11.626+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:54:11.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:11.626+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:11.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:11.626+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:54:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4395.675430 secs 2024-07-26T09:54:12.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:12 smithi171 ceph-mon[37789]: pgmap v2856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:54:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4395.675430 secs 2024-07-26T09:54:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:12 smithi044 ceph-mon[32404]: pgmap v2856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:54:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4395.675430 secs 2024-07-26T09:54:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:12 smithi116 ceph-mon[37097]: pgmap v2856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:54:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:14 smithi171 ceph-mon[37789]: pgmap v2857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:14 smithi044 ceph-mon[32404]: pgmap v2857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:14 smithi116 ceph-mon[37097]: pgmap v2857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T09:54:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:16 smithi171 ceph-mon[37789]: pgmap v2858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:16 smithi044 ceph-mon[32404]: pgmap v2858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:16 smithi116 ceph-mon[37097]: pgmap v2858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4400.675571 secs 2024-07-26T09:54:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4400.675571 secs 2024-07-26T09:54:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4400.675571 secs 2024-07-26T09:54:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:18 smithi171 ceph-mon[37789]: pgmap v2859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:18 smithi044 ceph-mon[32404]: pgmap v2859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:18 smithi116 ceph-mon[37097]: pgmap v2859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:20 smithi171 ceph-mon[37789]: pgmap v2860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:20 smithi044 ceph-mon[32404]: pgmap v2860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:20 smithi116 ceph-mon[37097]: pgmap v2860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4405.675711 secs 2024-07-26T09:54:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:22 smithi171 ceph-mon[37789]: pgmap v2861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4405.675711 secs 2024-07-26T09:54:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:22 smithi044 ceph-mon[32404]: pgmap v2861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4405.675711 secs 2024-07-26T09:54:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:22 smithi116 ceph-mon[37097]: pgmap v2861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:24 smithi171 ceph-mon[37789]: pgmap v2862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:24 smithi044 ceph-mon[32404]: pgmap v2862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:24 smithi116 ceph-mon[37097]: pgmap v2862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:26 smithi171 ceph-mon[37789]: pgmap v2863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:26 smithi044 ceph-mon[32404]: pgmap v2863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:26 smithi116 ceph-mon[37097]: pgmap v2863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:54:26.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:26.633+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:54:26.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:26.633+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:26.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:26.633+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:54:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4410.675833 secs 2024-07-26T09:54:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4410.675833 secs 2024-07-26T09:54:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4410.675833 secs 2024-07-26T09:54:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: pgmap v2864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:54:28.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:28 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: pgmap v2864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:54:28.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:28 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: pgmap v2864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:54:28.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:28 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T09:54:29.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:29.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:29.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:29 smithi044 ceph-mon[32404]: pgmap v2865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:54:29.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:29 smithi044 ceph-mon[32404]: pgmap v2866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:29.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:54:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:29 smithi116 ceph-mon[37097]: pgmap v2865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:54:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:29 smithi116 ceph-mon[37097]: pgmap v2866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:54:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:29 smithi171 ceph-mon[37789]: pgmap v2865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:54:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:29 smithi171 ceph-mon[37789]: pgmap v2866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:54:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:31 smithi044 ceph-mon[32404]: pgmap v2867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4415.675985 secs 2024-07-26T09:54:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:31 smithi116 ceph-mon[37097]: pgmap v2867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4415.675985 secs 2024-07-26T09:54:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:31 smithi171 ceph-mon[37789]: pgmap v2867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4415.675985 secs 2024-07-26T09:54:33.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:33 smithi044 ceph-mon[32404]: pgmap v2868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:33 smithi116 ceph-mon[37097]: pgmap v2868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:33 smithi171 ceph-mon[37789]: pgmap v2868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:35.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:35 smithi044 ceph-mon[32404]: pgmap v2869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:35 smithi116 ceph-mon[37097]: pgmap v2869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:35 smithi171 ceph-mon[37789]: pgmap v2869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:54:36.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4420.676092 secs 2024-07-26T09:54:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4420.676092 secs 2024-07-26T09:54:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4420.676092 secs 2024-07-26T09:54:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:37 smithi044 ceph-mon[32404]: pgmap v2870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:54:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:37 smithi116 ceph-mon[37097]: pgmap v2870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:54:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:37 smithi171 ceph-mon[37789]: pgmap v2870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:54:39.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:39 smithi044 ceph-mon[32404]: pgmap v2871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:39 smithi116 ceph-mon[37097]: pgmap v2871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:39 smithi171 ceph-mon[37789]: pgmap v2871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:54:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:41 smithi044 ceph-mon[32404]: pgmap v2872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4425.676255 secs 2024-07-26T09:54:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:41 smithi116 ceph-mon[37097]: pgmap v2872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4425.676255 secs 2024-07-26T09:54:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:41 smithi171 ceph-mon[37789]: pgmap v2872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4425.676255 secs 2024-07-26T09:54:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:41.636+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:54:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:41.636+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:54:41.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:41.636+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:43 smithi044 ceph-mon[32404]: pgmap v2873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:43 smithi116 ceph-mon[37097]: pgmap v2873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:43 smithi171 ceph-mon[37789]: pgmap v2873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:45 smithi044 ceph-mon[32404]: pgmap v2874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:45 smithi116 ceph-mon[37097]: pgmap v2874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:45 smithi171 ceph-mon[37789]: pgmap v2874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:54:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4430.676423 secs 2024-07-26T09:54:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4430.676423 secs 2024-07-26T09:54:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4430.676423 secs 2024-07-26T09:54:47.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:47 smithi044 ceph-mon[32404]: pgmap v2875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:47 smithi116 ceph-mon[37097]: pgmap v2875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:47 smithi171 ceph-mon[37789]: pgmap v2875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:49 smithi044 ceph-mon[32404]: pgmap v2876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:54:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:49 smithi116 ceph-mon[37097]: pgmap v2876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:54:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:49 smithi171 ceph-mon[37789]: pgmap v2876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:54:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:51 smithi044 ceph-mon[32404]: pgmap v2877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4435.676599 secs 2024-07-26T09:54:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:51 smithi116 ceph-mon[37097]: pgmap v2877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4435.676599 secs 2024-07-26T09:54:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:51 smithi171 ceph-mon[37789]: pgmap v2877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:54:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4435.676599 secs 2024-07-26T09:54:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:53 smithi044 ceph-mon[32404]: pgmap v2878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:53 smithi116 ceph-mon[37097]: pgmap v2878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:53 smithi171 ceph-mon[37789]: pgmap v2878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:55 smithi116 ceph-mon[37097]: pgmap v2879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:55 smithi171 ceph-mon[37789]: pgmap v2879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:55 smithi044 ceph-mon[32404]: pgmap v2879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:54:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4440.676738 secs 2024-07-26T09:54:56.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:56.644+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:54:56.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:56.644+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:54:56.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:54:56.644+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:56.647 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4440.676738 secs 2024-07-26T09:54:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4440.676738 secs 2024-07-26T09:54:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:57 smithi171 ceph-mon[37789]: pgmap v2880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:54:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:57 smithi044 ceph-mon[32404]: pgmap v2880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:54:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:57 smithi116 ceph-mon[37097]: pgmap v2880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:54:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:54:59 smithi171 ceph-mon[37789]: pgmap v2881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:54:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:54:59 smithi044 ceph-mon[32404]: pgmap v2881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:54:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:54:59 smithi116 ceph-mon[37097]: pgmap v2881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:55:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:01 smithi171 ceph-mon[37789]: pgmap v2882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:55:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4445.676909 secs 2024-07-26T09:55:01.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:01 smithi044 ceph-mon[32404]: pgmap v2882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:55:01.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4445.676909 secs 2024-07-26T09:55:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:01 smithi116 ceph-mon[37097]: pgmap v2882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:55:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4445.676909 secs 2024-07-26T09:55:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:03 smithi171 ceph-mon[37789]: pgmap v2883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:55:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:03 smithi044 ceph-mon[32404]: pgmap v2883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:55:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:03 smithi116 ceph-mon[37097]: pgmap v2883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:55:05.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:05 smithi116 ceph-mon[37097]: pgmap v2884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:05 smithi171 ceph-mon[37789]: pgmap v2884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:05 smithi044 ceph-mon[32404]: pgmap v2884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4450.677077 secs 2024-07-26T09:55:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4450.677077 secs 2024-07-26T09:55:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4450.677077 secs 2024-07-26T09:55:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:07 smithi171 ceph-mon[37789]: pgmap v2885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:07 smithi044 ceph-mon[32404]: pgmap v2885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:07 smithi116 ceph-mon[37097]: pgmap v2885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:09 smithi171 ceph-mon[37789]: pgmap v2886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:09 smithi044 ceph-mon[32404]: pgmap v2886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:09 smithi116 ceph-mon[37097]: pgmap v2886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:11 smithi171 ceph-mon[37789]: pgmap v2887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:11.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4455.677179 secs 2024-07-26T09:55:11.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:11.648+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:55:11.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:11.648+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:55:11.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:11.648+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:11.651 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:11 smithi044 ceph-mon[32404]: pgmap v2887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:11.651 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4455.677179 secs 2024-07-26T09:55:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:11 smithi116 ceph-mon[37097]: pgmap v2887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4455.677179 secs 2024-07-26T09:55:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:13 smithi044 ceph-mon[32404]: pgmap v2888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:13 smithi116 ceph-mon[37097]: pgmap v2888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:13.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:13 smithi171 ceph-mon[37789]: pgmap v2888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:15 smithi044 ceph-mon[32404]: pgmap v2889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:15 smithi116 ceph-mon[37097]: pgmap v2889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:15 smithi171 ceph-mon[37789]: pgmap v2889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4460.677319 secs 2024-07-26T09:55:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4460.677319 secs 2024-07-26T09:55:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4460.677319 secs 2024-07-26T09:55:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:17 smithi044 ceph-mon[32404]: pgmap v2890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:17 smithi116 ceph-mon[37097]: pgmap v2890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:17 smithi171 ceph-mon[37789]: pgmap v2890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:19 smithi044 ceph-mon[32404]: pgmap v2891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:19 smithi116 ceph-mon[37097]: pgmap v2891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:19 smithi171 ceph-mon[37789]: pgmap v2891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:21 smithi044 ceph-mon[32404]: pgmap v2892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4465.677458 secs 2024-07-26T09:55:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:21 smithi116 ceph-mon[37097]: pgmap v2892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4465.677458 secs 2024-07-26T09:55:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:21 smithi171 ceph-mon[37789]: pgmap v2892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4465.677458 secs 2024-07-26T09:55:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:23 smithi044 ceph-mon[32404]: pgmap v2893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:23 smithi116 ceph-mon[37097]: pgmap v2893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:23 smithi171 ceph-mon[37789]: pgmap v2893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:25 smithi044 ceph-mon[32404]: pgmap v2894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:25 smithi116 ceph-mon[37097]: pgmap v2894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:25 smithi171 ceph-mon[37789]: pgmap v2894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:26.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:26.655+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:55:26.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:26.655+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:55:26.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:26.655+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:26.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4470.677724 secs 2024-07-26T09:55:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4470.677724 secs 2024-07-26T09:55:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4470.677724 secs 2024-07-26T09:55:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:27 smithi044 ceph-mon[32404]: pgmap v2895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:27 smithi116 ceph-mon[37097]: pgmap v2895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:27 smithi171 ceph-mon[37789]: pgmap v2895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:29 smithi044 ceph-mon[32404]: pgmap v2896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:29 smithi116 ceph-mon[37097]: pgmap v2896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:55:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:29 smithi171 ceph-mon[37789]: pgmap v2896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:55:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:30.462+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:55:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:30.462+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:55:30.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:30.463+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:55:30.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:30.463+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:55:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:30 smithi044 ceph-mon[32404]: pgmap v2897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:55:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:30 smithi044 ceph-mon[32404]: pgmap v2898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:30 smithi116 ceph-mon[37097]: pgmap v2897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:55:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:30 smithi116 ceph-mon[37097]: pgmap v2898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:30 smithi171 ceph-mon[37789]: pgmap v2897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:55:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:30 smithi171 ceph-mon[37789]: pgmap v2898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4475.677759 secs 2024-07-26T09:55:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4475.677759 secs 2024-07-26T09:55:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4475.677759 secs 2024-07-26T09:55:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:32 smithi044 ceph-mon[32404]: pgmap v2899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:32 smithi116 ceph-mon[37097]: pgmap v2899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:32 smithi171 ceph-mon[37789]: pgmap v2899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:34 smithi044 ceph-mon[32404]: pgmap v2900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:34 smithi116 ceph-mon[37097]: pgmap v2900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:34 smithi171 ceph-mon[37789]: pgmap v2900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4480.677922 secs 2024-07-26T09:55:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:36 smithi044 ceph-mon[32404]: pgmap v2901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4480.677922 secs 2024-07-26T09:55:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:36 smithi116 ceph-mon[37097]: pgmap v2901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4480.677922 secs 2024-07-26T09:55:36.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:36 smithi171 ceph-mon[37789]: pgmap v2901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T09:55:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:38 smithi044 ceph-mon[32404]: pgmap v2902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:38 smithi116 ceph-mon[37097]: pgmap v2902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:38 smithi171 ceph-mon[37789]: pgmap v2902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:40 smithi044 ceph-mon[32404]: pgmap v2903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:55:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:40 smithi116 ceph-mon[37097]: pgmap v2903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:55:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:40 smithi171 ceph-mon[37789]: pgmap v2903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:55:41.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:41.659+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:55:41.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:41.659+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:41.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:41.659+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:55:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4485.678074 secs 2024-07-26T09:55:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4485.678074 secs 2024-07-26T09:55:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4485.678074 secs 2024-07-26T09:55:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:42 smithi044 ceph-mon[32404]: pgmap v2904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:42 smithi116 ceph-mon[37097]: pgmap v2904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:42 smithi171 ceph-mon[37789]: pgmap v2904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:44.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:44 smithi044 ceph-mon[32404]: pgmap v2905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:44 smithi116 ceph-mon[37097]: pgmap v2905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:44 smithi171 ceph-mon[37789]: pgmap v2905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4490.678177 secs 2024-07-26T09:55:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:46 smithi044 ceph-mon[32404]: pgmap v2906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4490.678177 secs 2024-07-26T09:55:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:46 smithi116 ceph-mon[37097]: pgmap v2906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4490.678177 secs 2024-07-26T09:55:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:46 smithi171 ceph-mon[37789]: pgmap v2906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:48 smithi044 ceph-mon[32404]: pgmap v2907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:48 smithi116 ceph-mon[37097]: pgmap v2907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:48 smithi171 ceph-mon[37789]: pgmap v2907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:50 smithi044 ceph-mon[32404]: pgmap v2908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:50 smithi116 ceph-mon[37097]: pgmap v2908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:50 smithi171 ceph-mon[37789]: pgmap v2908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:55:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4495.678341 secs 2024-07-26T09:55:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4495.678341 secs 2024-07-26T09:55:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4495.678341 secs 2024-07-26T09:55:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:52 smithi044 ceph-mon[32404]: pgmap v2909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:52 smithi116 ceph-mon[37097]: pgmap v2909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:52 smithi171 ceph-mon[37789]: pgmap v2909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:54 smithi044 ceph-mon[32404]: pgmap v2910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:54 smithi116 ceph-mon[37097]: pgmap v2910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:54 smithi171 ceph-mon[37789]: pgmap v2910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:55:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:56.668+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:55:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:56.668+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:56.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:55:56.668+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:55:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4500.678513 secs 2024-07-26T09:55:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:56 smithi044 ceph-mon[32404]: pgmap v2911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4500.678513 secs 2024-07-26T09:55:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:56 smithi116 ceph-mon[37097]: pgmap v2911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4500.678513 secs 2024-07-26T09:55:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:56 smithi171 ceph-mon[37789]: pgmap v2911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:55:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:55:58 smithi044 ceph-mon[32404]: pgmap v2912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:55:58 smithi116 ceph-mon[37097]: pgmap v2912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:55:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:55:58 smithi171 ceph-mon[37789]: pgmap v2912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:00 smithi044 ceph-mon[32404]: pgmap v2913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:00 smithi116 ceph-mon[37097]: pgmap v2913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:00 smithi171 ceph-mon[37789]: pgmap v2913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4505.678685 secs 2024-07-26T09:56:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4505.678685 secs 2024-07-26T09:56:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4505.678685 secs 2024-07-26T09:56:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:02 smithi044 ceph-mon[32404]: pgmap v2914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:02 smithi116 ceph-mon[37097]: pgmap v2914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:02 smithi171 ceph-mon[37789]: pgmap v2914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:04 smithi044 ceph-mon[32404]: pgmap v2915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:04 smithi116 ceph-mon[37097]: pgmap v2915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:04 smithi171 ceph-mon[37789]: pgmap v2915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4510.678853 secs 2024-07-26T09:56:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:06 smithi044 ceph-mon[32404]: pgmap v2916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4510.678853 secs 2024-07-26T09:56:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:06 smithi116 ceph-mon[37097]: pgmap v2916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4510.678853 secs 2024-07-26T09:56:06.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:06 smithi171 ceph-mon[37789]: pgmap v2916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:08 smithi116 ceph-mon[37097]: pgmap v2917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:08 smithi171 ceph-mon[37789]: pgmap v2917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:08 smithi044 ceph-mon[32404]: pgmap v2917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:10 smithi171 ceph-mon[37789]: pgmap v2918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:10 smithi044 ceph-mon[32404]: pgmap v2918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:10 smithi116 ceph-mon[37097]: pgmap v2918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:11.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:11.672+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:56:11.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:11.672+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:11.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:11.672+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:56:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4515.678980 secs 2024-07-26T09:56:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4515.678980 secs 2024-07-26T09:56:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4515.678980 secs 2024-07-26T09:56:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:12 smithi171 ceph-mon[37789]: pgmap v2919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:12 smithi044 ceph-mon[32404]: pgmap v2919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:12 smithi116 ceph-mon[37097]: pgmap v2919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:14.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b4730 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcf5805c000 tx=0x7fcf58052040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b4730 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcf5805c000 tx=0x7fcf58052040 comp rx=0 tx=0).stop 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b4730 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 msgr2=0x7fcf540b4730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b3d40 0x7fcf540b4730 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.695+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 0x7fcf540c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:56:14.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.696+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 0x7fcf540c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:56:14.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:14.698+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 0x7fcf540c18d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf5805b000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:56:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:14 smithi171 ceph-mon[37789]: pgmap v2920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:14 smithi044 ceph-mon[32404]: pgmap v2920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:14 smithi116 ceph-mon[37097]: pgmap v2920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:15.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:15.471+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:56:15.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:15.471+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:56:15.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:15.472+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T09:56:15.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:15.472+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:56:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4520.679131 secs 2024-07-26T09:56:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:16 smithi171 ceph-mon[37789]: pgmap v2921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4520.679131 secs 2024-07-26T09:56:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:16 smithi044 ceph-mon[32404]: pgmap v2921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4520.679131 secs 2024-07-26T09:56:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:16 smithi116 ceph-mon[37097]: pgmap v2921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:18 smithi171 ceph-mon[37789]: pgmap v2922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:18.979 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:18 smithi044 ceph-mon[32404]: pgmap v2922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:18 smithi116 ceph-mon[37097]: pgmap v2922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:20 smithi171 ceph-mon[37789]: pgmap v2923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:20 smithi044 ceph-mon[32404]: pgmap v2923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:20 smithi116 ceph-mon[37097]: pgmap v2923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4525.679284 secs 2024-07-26T09:56:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4525.679284 secs 2024-07-26T09:56:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4525.679284 secs 2024-07-26T09:56:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:22 smithi171 ceph-mon[37789]: pgmap v2924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:22 smithi044 ceph-mon[32404]: pgmap v2924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:22 smithi116 ceph-mon[37097]: pgmap v2924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:24 smithi171 ceph-mon[37789]: pgmap v2925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:24 smithi044 ceph-mon[32404]: pgmap v2925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:24 smithi116 ceph-mon[37097]: pgmap v2925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:26.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:26.680+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:56:26.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:26.680+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:56:26.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:26.680+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4530.679455 secs 2024-07-26T09:56:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:26 smithi044 ceph-mon[32404]: pgmap v2926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4530.679455 secs 2024-07-26T09:56:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:26 smithi116 ceph-mon[37097]: pgmap v2926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4530.679455 secs 2024-07-26T09:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:26 smithi171 ceph-mon[37789]: pgmap v2926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:28 smithi044 ceph-mon[32404]: pgmap v2927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:28 smithi116 ceph-mon[37097]: pgmap v2927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:28 smithi171 ceph-mon[37789]: pgmap v2927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:56:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:30 smithi044 ceph-mon[32404]: pgmap v2928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:30 smithi116 ceph-mon[37097]: pgmap v2928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:30 smithi171 ceph-mon[37789]: pgmap v2928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:31.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4535.679557 secs 2024-07-26T09:56:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4535.679557 secs 2024-07-26T09:56:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4535.679557 secs 2024-07-26T09:56:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:56:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:32 smithi044 ceph-mon[32404]: pgmap v2929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:56:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:32 smithi044 ceph-mon[32404]: pgmap v2930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:32 smithi116 ceph-mon[37097]: pgmap v2929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:56:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:32 smithi116 ceph-mon[37097]: pgmap v2930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:32 smithi171 ceph-mon[37789]: pgmap v2929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T09:56:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:32 smithi171 ceph-mon[37789]: pgmap v2930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:33 smithi044 ceph-mon[32404]: pgmap v2931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:33 smithi116 ceph-mon[37097]: pgmap v2931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:33 smithi171 ceph-mon[37789]: pgmap v2931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:36 smithi044 ceph-mon[32404]: pgmap v2932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:56:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4540.679715 secs 2024-07-26T09:56:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:36 smithi116 ceph-mon[37097]: pgmap v2932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:56:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4540.679715 secs 2024-07-26T09:56:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:36 smithi171 ceph-mon[37789]: pgmap v2932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:56:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4540.679715 secs 2024-07-26T09:56:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:38 smithi044 ceph-mon[32404]: pgmap v2933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:38 smithi116 ceph-mon[37097]: pgmap v2933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:38 smithi171 ceph-mon[37789]: pgmap v2933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:40 smithi044 ceph-mon[32404]: pgmap v2934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:40 smithi116 ceph-mon[37097]: pgmap v2934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:40 smithi171 ceph-mon[37789]: pgmap v2934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:56:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4545.679882 secs 2024-07-26T09:56:41.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:41.684+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:56:41.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:41.684+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:56:41.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:41.684+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:41.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4545.679882 secs 2024-07-26T09:56:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4545.679882 secs 2024-07-26T09:56:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:42 smithi171 ceph-mon[37789]: pgmap v2935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:56:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:42 smithi044 ceph-mon[32404]: pgmap v2935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:56:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:42 smithi116 ceph-mon[37097]: pgmap v2935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:56:44.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:44 smithi171 ceph-mon[37789]: pgmap v2936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:44.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:44 smithi044 ceph-mon[32404]: pgmap v2936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:44 smithi116 ceph-mon[37097]: pgmap v2936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:46 smithi171 ceph-mon[37789]: pgmap v2937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:46.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4550.680053 secs 2024-07-26T09:56:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:46 smithi044 ceph-mon[32404]: pgmap v2937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4550.680053 secs 2024-07-26T09:56:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:46 smithi116 ceph-mon[37097]: pgmap v2937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4550.680053 secs 2024-07-26T09:56:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:48 smithi171 ceph-mon[37789]: pgmap v2938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:48 smithi044 ceph-mon[32404]: pgmap v2938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:48 smithi116 ceph-mon[37097]: pgmap v2938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:50 smithi171 ceph-mon[37789]: pgmap v2939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:50.722 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:50 smithi044 ceph-mon[32404]: pgmap v2939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:50 smithi116 ceph-mon[37097]: pgmap v2939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4555.681096 secs 2024-07-26T09:56:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4555.681096 secs 2024-07-26T09:56:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4555.681096 secs 2024-07-26T09:56:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:52 smithi171 ceph-mon[37789]: pgmap v2940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:52 smithi044 ceph-mon[32404]: pgmap v2940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:52 smithi116 ceph-mon[37097]: pgmap v2940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:54 smithi171 ceph-mon[37789]: pgmap v2941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:54 smithi044 ceph-mon[32404]: pgmap v2941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:54 smithi116 ceph-mon[37097]: pgmap v2941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:56:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:56 smithi171 ceph-mon[37789]: pgmap v2942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:56.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4560.681215 secs 2024-07-26T09:56:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:56.693+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:56:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:56.693+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:56:56.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:56:56.693+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:56.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:56 smithi044 ceph-mon[32404]: pgmap v2942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:56.696 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4560.681215 secs 2024-07-26T09:56:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:56 smithi116 ceph-mon[37097]: pgmap v2942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:56:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4560.681215 secs 2024-07-26T09:56:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:56:58 smithi171 ceph-mon[37789]: pgmap v2943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:56:58 smithi044 ceph-mon[32404]: pgmap v2943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:56:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:56:58 smithi116 ceph-mon[37097]: pgmap v2943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:00 smithi044 ceph-mon[32404]: pgmap v2944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:00 smithi116 ceph-mon[37097]: pgmap v2944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:00 smithi171 ceph-mon[37789]: pgmap v2944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4565.681362 secs 2024-07-26T09:57:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4565.681362 secs 2024-07-26T09:57:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4565.681362 secs 2024-07-26T09:57:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:02 smithi044 ceph-mon[32404]: pgmap v2945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:02 smithi116 ceph-mon[37097]: pgmap v2945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:02 smithi171 ceph-mon[37789]: pgmap v2945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:04 smithi044 ceph-mon[32404]: pgmap v2946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:04 smithi116 ceph-mon[37097]: pgmap v2946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:04 smithi171 ceph-mon[37789]: pgmap v2946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:06.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:06 smithi171 ceph-mon[37789]: pgmap v2947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:06.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4570.681509 secs 2024-07-26T09:57:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:06 smithi044 ceph-mon[32404]: pgmap v2947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4570.681509 secs 2024-07-26T09:57:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:06 smithi116 ceph-mon[37097]: pgmap v2947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4570.681509 secs 2024-07-26T09:57:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:08 smithi044 ceph-mon[32404]: pgmap v2948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:08 smithi116 ceph-mon[37097]: pgmap v2948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:08 smithi171 ceph-mon[37789]: pgmap v2948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:10 smithi044 ceph-mon[32404]: pgmap v2949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:10 smithi116 ceph-mon[37097]: pgmap v2949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:10 smithi171 ceph-mon[37789]: pgmap v2949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:11.697+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:57:11.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:11.697+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:57:11.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:11.697+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:11.700 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4575.681673 secs 2024-07-26T09:57:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4575.681673 secs 2024-07-26T09:57:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4575.681673 secs 2024-07-26T09:57:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:12 smithi044 ceph-mon[32404]: pgmap v2950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:12 smithi116 ceph-mon[37097]: pgmap v2950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:12 smithi171 ceph-mon[37789]: pgmap v2950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:14 smithi044 ceph-mon[32404]: pgmap v2951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:14 smithi116 ceph-mon[37097]: pgmap v2951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:14 smithi171 ceph-mon[37789]: pgmap v2951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:16 smithi044 ceph-mon[32404]: pgmap v2952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4580.681833 secs 2024-07-26T09:57:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:16 smithi116 ceph-mon[37097]: pgmap v2952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4580.681833 secs 2024-07-26T09:57:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:16 smithi171 ceph-mon[37789]: pgmap v2952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4580.681833 secs 2024-07-26T09:57:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:18 smithi044 ceph-mon[32404]: pgmap v2953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:18 smithi116 ceph-mon[37097]: pgmap v2953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:18 smithi171 ceph-mon[37789]: pgmap v2953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:20 smithi044 ceph-mon[32404]: pgmap v2954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:20 smithi116 ceph-mon[37097]: pgmap v2954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:20 smithi171 ceph-mon[37789]: pgmap v2954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4585.681970 secs 2024-07-26T09:57:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4585.681970 secs 2024-07-26T09:57:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4585.681970 secs 2024-07-26T09:57:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:22 smithi044 ceph-mon[32404]: pgmap v2955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:22 smithi116 ceph-mon[37097]: pgmap v2955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:22 smithi171 ceph-mon[37789]: pgmap v2955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:24 smithi044 ceph-mon[32404]: pgmap v2956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:24 smithi116 ceph-mon[37097]: pgmap v2956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:24 smithi171 ceph-mon[37789]: pgmap v2956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:26.705+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:57:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:26.705+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:57:26.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:26.705+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:26.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:26 smithi044 ceph-mon[32404]: pgmap v2957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:26.709 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4590.682133 secs 2024-07-26T09:57:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:26 smithi116 ceph-mon[37097]: pgmap v2957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4590.682133 secs 2024-07-26T09:57:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:26 smithi171 ceph-mon[37789]: pgmap v2957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4590.682133 secs 2024-07-26T09:57:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:28 smithi044 ceph-mon[32404]: pgmap v2958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:28 smithi116 ceph-mon[37097]: pgmap v2958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:28 smithi171 ceph-mon[37789]: pgmap v2958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:30 smithi044 ceph-mon[32404]: pgmap v2959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:30 smithi116 ceph-mon[37097]: pgmap v2959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:30 smithi171 ceph-mon[37789]: pgmap v2959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4595.682272 secs 2024-07-26T09:57:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4595.682272 secs 2024-07-26T09:57:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4595.682272 secs 2024-07-26T09:57:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:32 smithi044 ceph-mon[32404]: pgmap v2960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:57:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:32 smithi116 ceph-mon[37097]: pgmap v2960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:57:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:32 smithi171 ceph-mon[37789]: pgmap v2960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:57:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: pgmap v2961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: pgmap v2961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: pgmap v2961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:57:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:36 smithi044 ceph-mon[32404]: pgmap v2962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:57:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:36 smithi116 ceph-mon[37097]: pgmap v2962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:57:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:36 smithi171 ceph-mon[37789]: pgmap v2962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:57:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4600.682373 secs 2024-07-26T09:57:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4600.682373 secs 2024-07-26T09:57:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4600.682373 secs 2024-07-26T09:57:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:57:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:38 smithi044 ceph-mon[32404]: pgmap v2963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:57:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:38 smithi044 ceph-mon[32404]: pgmap v2964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:38 smithi116 ceph-mon[37097]: pgmap v2963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:57:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:38 smithi116 ceph-mon[37097]: pgmap v2964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:38 smithi171 ceph-mon[37789]: pgmap v2963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T09:57:38.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:38 smithi171 ceph-mon[37789]: pgmap v2964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:40.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:40 smithi044 ceph-mon[32404]: pgmap v2965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:40 smithi116 ceph-mon[37097]: pgmap v2965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:40 smithi171 ceph-mon[37789]: pgmap v2965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:41.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:41.709+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:57:41.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:41.709+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:57:41.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:41.709+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4605.682523 secs 2024-07-26T09:57:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:42 smithi044 ceph-mon[32404]: pgmap v2966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4605.682523 secs 2024-07-26T09:57:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:42 smithi116 ceph-mon[37097]: pgmap v2966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4605.682523 secs 2024-07-26T09:57:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:42 smithi171 ceph-mon[37789]: pgmap v2966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:44 smithi044 ceph-mon[32404]: pgmap v2967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:57:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:44 smithi116 ceph-mon[37097]: pgmap v2967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:57:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:44 smithi171 ceph-mon[37789]: pgmap v2967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T09:57:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:46 smithi044 ceph-mon[32404]: pgmap v2968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:46 smithi116 ceph-mon[37097]: pgmap v2968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:46 smithi171 ceph-mon[37789]: pgmap v2968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4610.682684 secs 2024-07-26T09:57:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4610.682684 secs 2024-07-26T09:57:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4610.682684 secs 2024-07-26T09:57:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:48 smithi044 ceph-mon[32404]: pgmap v2969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:57:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:48 smithi116 ceph-mon[37097]: pgmap v2969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:57:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:48 smithi171 ceph-mon[37789]: pgmap v2969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:57:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:50 smithi044 ceph-mon[32404]: pgmap v2970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:50 smithi116 ceph-mon[37097]: pgmap v2970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:50 smithi171 ceph-mon[37789]: pgmap v2970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:57:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4615.682851 secs 2024-07-26T09:57:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:52 smithi044 ceph-mon[32404]: pgmap v2971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4615.682851 secs 2024-07-26T09:57:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:52 smithi116 ceph-mon[37097]: pgmap v2971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4615.682851 secs 2024-07-26T09:57:52.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:52 smithi171 ceph-mon[37789]: pgmap v2971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:57:54.421 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:54 smithi044 ceph-mon[32404]: pgmap v2972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:54.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:54 smithi116 ceph-mon[37097]: pgmap v2972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:54 smithi171 ceph-mon[37789]: pgmap v2972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:57:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:56 smithi044 ceph-mon[32404]: pgmap v2973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4620.682981 secs 2024-07-26T09:57:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:56 smithi116 ceph-mon[37097]: pgmap v2973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4620.682981 secs 2024-07-26T09:57:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:56 smithi171 ceph-mon[37789]: pgmap v2973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:57:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4620.682981 secs 2024-07-26T09:57:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:56.718+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:57:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:56.718+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:57:56.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:57:56.718+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:57:58 smithi044 ceph-mon[32404]: pgmap v2974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:57:58 smithi116 ceph-mon[37097]: pgmap v2974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:57:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:57:58 smithi171 ceph-mon[37789]: pgmap v2974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:00 smithi044 ceph-mon[32404]: pgmap v2975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:00 smithi116 ceph-mon[37097]: pgmap v2975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:00 smithi171 ceph-mon[37789]: pgmap v2975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4625.683118 secs 2024-07-26T09:58:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4625.683118 secs 2024-07-26T09:58:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4625.683118 secs 2024-07-26T09:58:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:02 smithi044 ceph-mon[32404]: pgmap v2976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:02 smithi116 ceph-mon[37097]: pgmap v2976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:02 smithi171 ceph-mon[37789]: pgmap v2976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:04 smithi044 ceph-mon[32404]: pgmap v2977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:04 smithi116 ceph-mon[37097]: pgmap v2977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:04 smithi171 ceph-mon[37789]: pgmap v2977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:06 smithi044 ceph-mon[32404]: pgmap v2978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4630.683217 secs 2024-07-26T09:58:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:06 smithi116 ceph-mon[37097]: pgmap v2978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4630.683217 secs 2024-07-26T09:58:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:06 smithi171 ceph-mon[37789]: pgmap v2978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4630.683217 secs 2024-07-26T09:58:08.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:08 smithi044 ceph-mon[32404]: pgmap v2979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:08 smithi116 ceph-mon[37097]: pgmap v2979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:08 smithi171 ceph-mon[37789]: pgmap v2979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:10 smithi044 ceph-mon[32404]: pgmap v2980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:10 smithi116 ceph-mon[37097]: pgmap v2980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:10 smithi171 ceph-mon[37789]: pgmap v2980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4635.683351 secs 2024-07-26T09:58:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4635.683351 secs 2024-07-26T09:58:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4635.683351 secs 2024-07-26T09:58:11.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:11.721+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:58:11.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:11.721+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:58:11.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:11.721+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:12 smithi044 ceph-mon[32404]: pgmap v2981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:12 smithi116 ceph-mon[37097]: pgmap v2981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:12 smithi171 ceph-mon[37789]: pgmap v2981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:14 smithi044 ceph-mon[32404]: pgmap v2982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:14 smithi116 ceph-mon[37097]: pgmap v2982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:14 smithi171 ceph-mon[37789]: pgmap v2982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:16 smithi116 ceph-mon[37097]: pgmap v2983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4640.683468 secs 2024-07-26T09:58:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:16 smithi171 ceph-mon[37789]: pgmap v2983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4640.683468 secs 2024-07-26T09:58:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:16 smithi044 ceph-mon[32404]: pgmap v2983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4640.683468 secs 2024-07-26T09:58:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:18 smithi171 ceph-mon[37789]: pgmap v2984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:18 smithi044 ceph-mon[32404]: pgmap v2984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:18 smithi116 ceph-mon[37097]: pgmap v2984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:20 smithi171 ceph-mon[37789]: pgmap v2985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:20 smithi044 ceph-mon[32404]: pgmap v2985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:20.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:20 smithi116 ceph-mon[37097]: pgmap v2985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4645.683659 secs 2024-07-26T09:58:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4645.683659 secs 2024-07-26T09:58:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4645.683659 secs 2024-07-26T09:58:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:22 smithi171 ceph-mon[37789]: pgmap v2986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:22 smithi044 ceph-mon[32404]: pgmap v2986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:22 smithi116 ceph-mon[37097]: pgmap v2986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:24 smithi171 ceph-mon[37789]: pgmap v2987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:24 smithi044 ceph-mon[32404]: pgmap v2987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:24 smithi116 ceph-mon[37097]: pgmap v2987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:26.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:26 smithi044 ceph-mon[32404]: pgmap v2988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:26.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4650.683769 secs 2024-07-26T09:58:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:26 smithi171 ceph-mon[37789]: pgmap v2988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4650.683769 secs 2024-07-26T09:58:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:26.728+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:58:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:26.729+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:58:26.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:26.729+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:26 smithi116 ceph-mon[37097]: pgmap v2988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4650.683769 secs 2024-07-26T09:58:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:28 smithi171 ceph-mon[37789]: pgmap v2989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:28 smithi044 ceph-mon[32404]: pgmap v2989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:28 smithi116 ceph-mon[37097]: pgmap v2989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:30 smithi171 ceph-mon[37789]: pgmap v2990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:30 smithi044 ceph-mon[32404]: pgmap v2990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:30 smithi116 ceph-mon[37097]: pgmap v2990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T09:58:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4655.683884 secs 2024-07-26T09:58:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4655.683884 secs 2024-07-26T09:58:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4655.683884 secs 2024-07-26T09:58:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:32 smithi171 ceph-mon[37789]: pgmap v2991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:32 smithi044 ceph-mon[32404]: pgmap v2991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:32 smithi116 ceph-mon[37097]: pgmap v2991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:34 smithi044 ceph-mon[32404]: pgmap v2992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:34 smithi116 ceph-mon[37097]: pgmap v2992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:34 smithi171 ceph-mon[37789]: pgmap v2992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:36.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:36 smithi171 ceph-mon[37789]: pgmap v2993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:36.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4660.684034 secs 2024-07-26T09:58:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:36 smithi044 ceph-mon[32404]: pgmap v2993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4660.684034 secs 2024-07-26T09:58:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:36 smithi116 ceph-mon[37097]: pgmap v2993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4660.684034 secs 2024-07-26T09:58:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:37 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:37 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:37 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:38.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:38 smithi044 ceph-mon[32404]: pgmap v2994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:38.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:38.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:38.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:58:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:38 smithi116 ceph-mon[37097]: pgmap v2994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:58:38.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:38 smithi171 ceph-mon[37789]: pgmap v2994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T09:58:38.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:38.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:38.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:58:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:39 smithi044 ceph-mon[32404]: pgmap v2995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:58:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:39 smithi044 ceph-mon[32404]: pgmap v2996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:39 smithi116 ceph-mon[37097]: pgmap v2995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:58:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:39 smithi116 ceph-mon[37097]: pgmap v2996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:39 smithi171 ceph-mon[37789]: pgmap v2995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T09:58:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:39 smithi171 ceph-mon[37789]: pgmap v2996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:41.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:41.733+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:58:41.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:41.733+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:58:41.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:41.733+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:41.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:41 smithi044 ceph-mon[32404]: pgmap v2997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:41.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4665.684183 secs 2024-07-26T09:58:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:41 smithi116 ceph-mon[37097]: pgmap v2997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4665.684183 secs 2024-07-26T09:58:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:41 smithi171 ceph-mon[37789]: pgmap v2997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4665.684183 secs 2024-07-26T09:58:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:43 smithi044 ceph-mon[32404]: pgmap v2998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:43 smithi116 ceph-mon[37097]: pgmap v2998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:43 smithi171 ceph-mon[37789]: pgmap v2998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T09:58:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:45 smithi044 ceph-mon[32404]: pgmap v2999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:45 smithi116 ceph-mon[37097]: pgmap v2999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:45 smithi171 ceph-mon[37789]: pgmap v2999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:58:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4670.684332 secs 2024-07-26T09:58:46.859 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4670.684332 secs 2024-07-26T09:58:46.859 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4670.684332 secs 2024-07-26T09:58:47.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:47 smithi044 ceph-mon[32404]: pgmap v3000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:58:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:47 smithi116 ceph-mon[37097]: pgmap v3000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:58:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:47 smithi171 ceph-mon[37789]: pgmap v3000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T09:58:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:49 smithi044 ceph-mon[32404]: pgmap v3001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:58:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:49 smithi116 ceph-mon[37097]: pgmap v3001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:58:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:49 smithi171 ceph-mon[37789]: pgmap v3001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:58:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:51 smithi044 ceph-mon[32404]: pgmap v3002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4675.684492 secs 2024-07-26T09:58:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:51 smithi116 ceph-mon[37097]: pgmap v3002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4675.684492 secs 2024-07-26T09:58:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:51 smithi171 ceph-mon[37789]: pgmap v3002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4675.684492 secs 2024-07-26T09:58:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:53 smithi044 ceph-mon[32404]: pgmap v3003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:53 smithi116 ceph-mon[37097]: pgmap v3003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:53 smithi171 ceph-mon[37789]: pgmap v3003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:58:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:55 smithi044 ceph-mon[32404]: pgmap v3004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:55 smithi116 ceph-mon[37097]: pgmap v3004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:55 smithi171 ceph-mon[37789]: pgmap v3004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:58:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:56.737+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:58:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:56.737+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:58:56.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:58:56.737+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:56.740 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4680.684666 secs 2024-07-26T09:58:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4680.684666 secs 2024-07-26T09:58:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4680.684666 secs 2024-07-26T09:58:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:57 smithi044 ceph-mon[32404]: pgmap v3005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:57 smithi116 ceph-mon[37097]: pgmap v3005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:57 smithi171 ceph-mon[37789]: pgmap v3005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:58:59.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:58:59 smithi044 ceph-mon[32404]: pgmap v3006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:58:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:58:59 smithi116 ceph-mon[37097]: pgmap v3006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:58:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:58:59 smithi171 ceph-mon[37789]: pgmap v3006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:01 smithi044 ceph-mon[32404]: pgmap v3007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4685.684822 secs 2024-07-26T09:59:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:01 smithi116 ceph-mon[37097]: pgmap v3007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4685.684822 secs 2024-07-26T09:59:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:01 smithi171 ceph-mon[37789]: pgmap v3007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4685.684822 secs 2024-07-26T09:59:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:03 smithi044 ceph-mon[32404]: pgmap v3008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:03 smithi116 ceph-mon[37097]: pgmap v3008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:03 smithi171 ceph-mon[37789]: pgmap v3008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:05 smithi044 ceph-mon[32404]: pgmap v3009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:05 smithi116 ceph-mon[37097]: pgmap v3009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:05 smithi171 ceph-mon[37789]: pgmap v3009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:06.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4690.684911 secs 2024-07-26T09:59:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4690.684911 secs 2024-07-26T09:59:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4690.684911 secs 2024-07-26T09:59:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:07 smithi044 ceph-mon[32404]: pgmap v3010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:59:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:07 smithi116 ceph-mon[37097]: pgmap v3010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:59:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:07 smithi171 ceph-mon[37789]: pgmap v3010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T09:59:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:09 smithi044 ceph-mon[32404]: pgmap v3011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:09 smithi116 ceph-mon[37097]: pgmap v3011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:09 smithi171 ceph-mon[37789]: pgmap v3011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:11.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:11.740+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:59:11.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:11.740+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:59:11.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:11.740+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:11 smithi044 ceph-mon[32404]: pgmap v3012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4695.685059 secs 2024-07-26T09:59:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:11 smithi116 ceph-mon[37097]: pgmap v3012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4695.685059 secs 2024-07-26T09:59:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:11 smithi171 ceph-mon[37789]: pgmap v3012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4695.685059 secs 2024-07-26T09:59:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:13 smithi044 ceph-mon[32404]: pgmap v3013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:13 smithi116 ceph-mon[37097]: pgmap v3013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:13 smithi171 ceph-mon[37789]: pgmap v3013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:15.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:15.504+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:59:15.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:15.504+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003900 con 0x7fcf6c1057b0 2024-07-26T09:59:15.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:15.505+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T09:59:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:15.515+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003900 con 0x7fcf6c1057b0 2024-07-26T09:59:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:15 smithi044 ceph-mon[32404]: pgmap v3014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:15 smithi116 ceph-mon[37097]: pgmap v3014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:15 smithi171 ceph-mon[37789]: pgmap v3014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4700.685157 secs 2024-07-26T09:59:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4700.685157 secs 2024-07-26T09:59:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4700.685157 secs 2024-07-26T09:59:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:17 smithi171 ceph-mon[37789]: pgmap v3015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:17 smithi044 ceph-mon[32404]: pgmap v3015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:17 smithi116 ceph-mon[37097]: pgmap v3015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:19 smithi171 ceph-mon[37789]: pgmap v3016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:19 smithi044 ceph-mon[32404]: pgmap v3016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:19 smithi116 ceph-mon[37097]: pgmap v3016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:21 smithi171 ceph-mon[37789]: pgmap v3017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4705.685259 secs 2024-07-26T09:59:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:21 smithi044 ceph-mon[32404]: pgmap v3017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4705.685259 secs 2024-07-26T09:59:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:21 smithi116 ceph-mon[37097]: pgmap v3017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4705.685259 secs 2024-07-26T09:59:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:23 smithi171 ceph-mon[37789]: pgmap v3018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:23 smithi044 ceph-mon[32404]: pgmap v3018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:23 smithi116 ceph-mon[37097]: pgmap v3018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:25 smithi171 ceph-mon[37789]: pgmap v3019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:25.858 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:25 smithi044 ceph-mon[32404]: pgmap v3019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:25 smithi116 ceph-mon[37097]: pgmap v3019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T09:59:26.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:26.749+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:59:26.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:26.749+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:59:26.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:26.749+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4710.685360 secs 2024-07-26T09:59:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4710.685360 secs 2024-07-26T09:59:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4710.685360 secs 2024-07-26T09:59:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:27 smithi171 ceph-mon[37789]: pgmap v3020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:59:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:27 smithi044 ceph-mon[32404]: pgmap v3020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:59:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:27 smithi116 ceph-mon[37097]: pgmap v3020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T09:59:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:29 smithi171 ceph-mon[37789]: pgmap v3021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:59:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:29 smithi044 ceph-mon[32404]: pgmap v3021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:29 smithi116 ceph-mon[37097]: pgmap v3021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T09:59:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:31 smithi171 ceph-mon[37789]: pgmap v3022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4715.685461 secs 2024-07-26T09:59:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:31 smithi044 ceph-mon[32404]: pgmap v3022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4715.685461 secs 2024-07-26T09:59:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:31 smithi116 ceph-mon[37097]: pgmap v3022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4715.685461 secs 2024-07-26T09:59:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:33 smithi171 ceph-mon[37789]: pgmap v3023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:33 smithi044 ceph-mon[32404]: pgmap v3023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:33 smithi116 ceph-mon[37097]: pgmap v3023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T09:59:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:35 smithi171 ceph-mon[37789]: pgmap v3024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:35 smithi044 ceph-mon[32404]: pgmap v3024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:35 smithi116 ceph-mon[37097]: pgmap v3024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T09:59:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4720.685569 secs 2024-07-26T09:59:36.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4720.685569 secs 2024-07-26T09:59:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4720.685569 secs 2024-07-26T09:59:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:37 smithi044 ceph-mon[32404]: pgmap v3025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:37 smithi116 ceph-mon[37097]: pgmap v3025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:37 smithi171 ceph-mon[37789]: pgmap v3025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:39 smithi044 ceph-mon[32404]: pgmap v3026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:59:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:39 smithi116 ceph-mon[37097]: pgmap v3026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:59:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:39 smithi171 ceph-mon[37789]: pgmap v3026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T09:59:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:40 smithi044 ceph-mon[32404]: pgmap v3027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:59:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:40 smithi044 ceph-mon[32404]: pgmap v3028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:40 smithi116 ceph-mon[37097]: pgmap v3027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:59:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:40 smithi116 ceph-mon[37097]: pgmap v3028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:40 smithi171 ceph-mon[37789]: pgmap v3027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T09:59:41.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:40 smithi171 ceph-mon[37789]: pgmap v3028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:41.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:41.752+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:59:41.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:41.752+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:59:41.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:41.752+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4725.685687 secs 2024-07-26T09:59:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4725.685687 secs 2024-07-26T09:59:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4725.685687 secs 2024-07-26T09:59:43.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:42 smithi044 ceph-mon[32404]: pgmap v3029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:42 smithi116 ceph-mon[37097]: pgmap v3029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:42 smithi171 ceph-mon[37789]: pgmap v3029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:45.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:44 smithi044 ceph-mon[32404]: pgmap v3030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:44 smithi116 ceph-mon[37097]: pgmap v3030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:44 smithi171 ceph-mon[37789]: pgmap v3030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4730.685858 secs 2024-07-26T09:59:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:46 smithi044 ceph-mon[32404]: pgmap v3031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4730.685858 secs 2024-07-26T09:59:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:46 smithi116 ceph-mon[37097]: pgmap v3031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4730.685858 secs 2024-07-26T09:59:47.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:46 smithi171 ceph-mon[37789]: pgmap v3031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T09:59:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:47 smithi044 ceph-mon[32404]: pgmap v3032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:47 smithi116 ceph-mon[37097]: pgmap v3032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:47 smithi171 ceph-mon[37789]: pgmap v3032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T09:59:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:50 smithi044 ceph-mon[32404]: pgmap v3033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:59:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:50 smithi116 ceph-mon[37097]: pgmap v3033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:59:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:50 smithi171 ceph-mon[37789]: pgmap v3033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T09:59:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4735.686043 secs 2024-07-26T09:59:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4735.686043 secs 2024-07-26T09:59:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4735.686043 secs 2024-07-26T09:59:52.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:52 smithi044 ceph-mon[32404]: pgmap v3034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:52 smithi116 ceph-mon[37097]: pgmap v3034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:52 smithi171 ceph-mon[37789]: pgmap v3034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:54 smithi044 ceph-mon[32404]: pgmap v3035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:54 smithi116 ceph-mon[37097]: pgmap v3035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:54 smithi171 ceph-mon[37789]: pgmap v3035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T09:59:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:56.761+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T09:59:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:56.761+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T09:59:56.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T09:59:56.761+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:56.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4740.686206 secs 2024-07-26T09:59:56.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:56 smithi044 ceph-mon[32404]: pgmap v3036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4740.686206 secs 2024-07-26T09:59:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:56 smithi116 ceph-mon[37097]: pgmap v3036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4740.686206 secs 2024-07-26T09:59:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:56 smithi171 ceph-mon[37789]: pgmap v3036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T09:59:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 09:59:58 smithi044 ceph-mon[32404]: pgmap v3037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 09:59:58 smithi116 ceph-mon[37097]: pgmap v3037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T09:59:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 09:59:58 smithi171 ceph-mon[37789]: pgmap v3037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:00 smithi116 ceph-mon[37097]: pgmap v3038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:00:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:00 smithi171 ceph-mon[37789]: pgmap v3038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:00:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:00 smithi044 ceph-mon[32404]: pgmap v3038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:00:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4745.686349 secs 2024-07-26T10:00:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4745.686349 secs 2024-07-26T10:00:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4745.686349 secs 2024-07-26T10:00:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:02 smithi171 ceph-mon[37789]: pgmap v3039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:02 smithi044 ceph-mon[32404]: pgmap v3039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:03.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:02 smithi116 ceph-mon[37097]: pgmap v3039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:04 smithi171 ceph-mon[37789]: pgmap v3040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:04 smithi044 ceph-mon[32404]: pgmap v3040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:04 smithi116 ceph-mon[37097]: pgmap v3040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:06.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4750.686493 secs 2024-07-26T10:00:06.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:06 smithi116 ceph-mon[37097]: pgmap v3041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4750.686493 secs 2024-07-26T10:00:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:06 smithi171 ceph-mon[37789]: pgmap v3041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4750.686493 secs 2024-07-26T10:00:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:06 smithi044 ceph-mon[32404]: pgmap v3041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:08 smithi171 ceph-mon[37789]: pgmap v3042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:08 smithi044 ceph-mon[32404]: pgmap v3042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:09.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:08 smithi116 ceph-mon[37097]: pgmap v3042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:10 smithi171 ceph-mon[37789]: pgmap v3043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:10 smithi044 ceph-mon[32404]: pgmap v3043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:10 smithi116 ceph-mon[37097]: pgmap v3043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:11.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:11.765+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:00:11.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:11.765+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:11.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:11.765+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:00:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4755.686690 secs 2024-07-26T10:00:11.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4755.686690 secs 2024-07-26T10:00:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4755.686690 secs 2024-07-26T10:00:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:12 smithi171 ceph-mon[37789]: pgmap v3044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:12 smithi044 ceph-mon[32404]: pgmap v3044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:12 smithi116 ceph-mon[37097]: pgmap v3044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:14 smithi171 ceph-mon[37789]: pgmap v3045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:14 smithi044 ceph-mon[32404]: pgmap v3045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:14 smithi116 ceph-mon[37097]: pgmap v3045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4760.686834 secs 2024-07-26T10:00:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:16 smithi171 ceph-mon[37789]: pgmap v3046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4760.686834 secs 2024-07-26T10:00:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:16 smithi044 ceph-mon[32404]: pgmap v3046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4760.686834 secs 2024-07-26T10:00:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:16 smithi116 ceph-mon[37097]: pgmap v3046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:18 smithi116 ceph-mon[37097]: pgmap v3047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:18 smithi044 ceph-mon[32404]: pgmap v3047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:18 smithi171 ceph-mon[37789]: pgmap v3047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:20 smithi044 ceph-mon[32404]: pgmap v3048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:20 smithi116 ceph-mon[37097]: pgmap v3048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:20 smithi171 ceph-mon[37789]: pgmap v3048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4765.686941 secs 2024-07-26T10:00:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4765.686941 secs 2024-07-26T10:00:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4765.686941 secs 2024-07-26T10:00:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:22 smithi044 ceph-mon[32404]: pgmap v3049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:22 smithi116 ceph-mon[37097]: pgmap v3049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:22 smithi171 ceph-mon[37789]: pgmap v3049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:25.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:24 smithi044 ceph-mon[32404]: pgmap v3050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:24 smithi116 ceph-mon[37097]: pgmap v3050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:24 smithi171 ceph-mon[37789]: pgmap v3050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:26.774+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:00:26.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:26.774+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:26.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:26.774+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:00:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4770.687087 secs 2024-07-26T10:00:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:26 smithi044 ceph-mon[32404]: pgmap v3051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4770.687087 secs 2024-07-26T10:00:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:26 smithi116 ceph-mon[37097]: pgmap v3051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4770.687087 secs 2024-07-26T10:00:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:26 smithi171 ceph-mon[37789]: pgmap v3051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:28 smithi044 ceph-mon[32404]: pgmap v3052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:28 smithi116 ceph-mon[37097]: pgmap v3052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:28 smithi171 ceph-mon[37789]: pgmap v3052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:30 smithi044 ceph-mon[32404]: pgmap v3053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:30 smithi116 ceph-mon[37097]: pgmap v3053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:30 smithi171 ceph-mon[37789]: pgmap v3053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:32.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4775.687232 secs 2024-07-26T10:00:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4775.687232 secs 2024-07-26T10:00:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4775.687232 secs 2024-07-26T10:00:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:32 smithi044 ceph-mon[32404]: pgmap v3054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:32 smithi116 ceph-mon[37097]: pgmap v3054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:32 smithi171 ceph-mon[37789]: pgmap v3054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:34 smithi044 ceph-mon[32404]: pgmap v3055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:34 smithi116 ceph-mon[37097]: pgmap v3055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:34 smithi171 ceph-mon[37789]: pgmap v3055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:00:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4780.687395 secs 2024-07-26T10:00:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:36 smithi044 ceph-mon[32404]: pgmap v3056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4780.687395 secs 2024-07-26T10:00:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:36 smithi116 ceph-mon[37097]: pgmap v3056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4780.687395 secs 2024-07-26T10:00:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:36 smithi171 ceph-mon[37789]: pgmap v3056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:00:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:38 smithi044 ceph-mon[32404]: pgmap v3057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:38 smithi116 ceph-mon[37097]: pgmap v3057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:39.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:38 smithi171 ceph-mon[37789]: pgmap v3057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:40 smithi044 ceph-mon[32404]: pgmap v3058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:40 smithi116 ceph-mon[37097]: pgmap v3058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:40 smithi171 ceph-mon[37789]: pgmap v3058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:41.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4785.687527 secs 2024-07-26T10:00:41.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:41.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:41.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:41 smithi044 ceph-mon[32404]: pgmap v3059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:00:41.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:41 smithi044 ceph-mon[32404]: pgmap v3060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:41.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:00:41.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:41.776+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:00:41.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:41.776+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:41.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:41.776+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:00:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4785.687527 secs 2024-07-26T10:00:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:41 smithi116 ceph-mon[37097]: pgmap v3059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:00:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:41 smithi116 ceph-mon[37097]: pgmap v3060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:00:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4785.687527 secs 2024-07-26T10:00:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:41 smithi171 ceph-mon[37789]: pgmap v3059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:00:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:41 smithi171 ceph-mon[37789]: pgmap v3060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:00:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:43 smithi116 ceph-mon[37097]: pgmap v3061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:00:44.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:43 smithi171 ceph-mon[37789]: pgmap v3061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:00:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:43 smithi044 ceph-mon[32404]: pgmap v3061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:00:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:45 smithi171 ceph-mon[37789]: pgmap v3062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:45 smithi044 ceph-mon[32404]: pgmap v3062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:45 smithi116 ceph-mon[37097]: pgmap v3062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4790.687694 secs 2024-07-26T10:00:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4790.687694 secs 2024-07-26T10:00:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4790.687694 secs 2024-07-26T10:00:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:47 smithi171 ceph-mon[37789]: pgmap v3063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:47 smithi044 ceph-mon[32404]: pgmap v3063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:47 smithi116 ceph-mon[37097]: pgmap v3063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:50.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:49 smithi171 ceph-mon[37789]: pgmap v3064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:50.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:49 smithi044 ceph-mon[32404]: pgmap v3064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:49 smithi116 ceph-mon[37097]: pgmap v3064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:00:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4795.687792 secs 2024-07-26T10:00:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:51 smithi171 ceph-mon[37789]: pgmap v3065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:00:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4795.687792 secs 2024-07-26T10:00:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:51 smithi044 ceph-mon[32404]: pgmap v3065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:00:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4795.687792 secs 2024-07-26T10:00:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:51 smithi116 ceph-mon[37097]: pgmap v3065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:00:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:53 smithi171 ceph-mon[37789]: pgmap v3066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:53 smithi044 ceph-mon[32404]: pgmap v3066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:53 smithi116 ceph-mon[37097]: pgmap v3066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:56.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:55 smithi044 ceph-mon[32404]: pgmap v3067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:55 smithi171 ceph-mon[37789]: pgmap v3067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:55 smithi116 ceph-mon[37097]: pgmap v3067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:00:56.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:56.785+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:00:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:56.785+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:00:56.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:00:56.785+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4800.687878 secs 2024-07-26T10:00:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4800.687878 secs 2024-07-26T10:00:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4800.687878 secs 2024-07-26T10:00:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:57 smithi171 ceph-mon[37789]: pgmap v3068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:58.170 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:57 smithi044 ceph-mon[32404]: pgmap v3068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:00:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:57 smithi116 ceph-mon[37097]: pgmap v3068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:00.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:00:59 smithi044 ceph-mon[32404]: pgmap v3069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:00:59 smithi116 ceph-mon[37097]: pgmap v3069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:00:59 smithi171 ceph-mon[37789]: pgmap v3069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4805.688040 secs 2024-07-26T10:01:02.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:01 smithi171 ceph-mon[37789]: pgmap v3070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4805.688040 secs 2024-07-26T10:01:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:01 smithi044 ceph-mon[32404]: pgmap v3070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4805.688040 secs 2024-07-26T10:01:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:01 smithi116 ceph-mon[37097]: pgmap v3070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:04.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:03 smithi044 ceph-mon[32404]: pgmap v3071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:03 smithi116 ceph-mon[37097]: pgmap v3071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:03 smithi171 ceph-mon[37789]: pgmap v3071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:06.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:05 smithi171 ceph-mon[37789]: pgmap v3072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:01:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:05 smithi044 ceph-mon[32404]: pgmap v3072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:01:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:05 smithi116 ceph-mon[37097]: pgmap v3072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:01:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4810.688200 secs 2024-07-26T10:01:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4810.688200 secs 2024-07-26T10:01:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4810.688200 secs 2024-07-26T10:01:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:07 smithi044 ceph-mon[32404]: pgmap v3073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:07 smithi116 ceph-mon[37097]: pgmap v3073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:07 smithi171 ceph-mon[37789]: pgmap v3073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:09 smithi044 ceph-mon[32404]: pgmap v3074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:09 smithi116 ceph-mon[37097]: pgmap v3074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:09 smithi171 ceph-mon[37789]: pgmap v3074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:11.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:11.789+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:01:11.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:11.789+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:11.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:11.789+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:01:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4815.688357 secs 2024-07-26T10:01:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:11 smithi044 ceph-mon[32404]: pgmap v3075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:01:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4815.688357 secs 2024-07-26T10:01:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:11 smithi116 ceph-mon[37097]: pgmap v3075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:01:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4815.688357 secs 2024-07-26T10:01:12.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:11 smithi171 ceph-mon[37789]: pgmap v3075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:01:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:13 smithi044 ceph-mon[32404]: pgmap v3076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:13 smithi116 ceph-mon[37097]: pgmap v3076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:13 smithi171 ceph-mon[37789]: pgmap v3076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:01:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:15 smithi044 ceph-mon[32404]: pgmap v3077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:15 smithi116 ceph-mon[37097]: pgmap v3077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:15 smithi171 ceph-mon[37789]: pgmap v3077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4820.688517 secs 2024-07-26T10:01:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4820.688517 secs 2024-07-26T10:01:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4820.688517 secs 2024-07-26T10:01:18.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:17 smithi044 ceph-mon[32404]: pgmap v3078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:17 smithi116 ceph-mon[37097]: pgmap v3078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:17 smithi171 ceph-mon[37789]: pgmap v3078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:20.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:19 smithi044 ceph-mon[32404]: pgmap v3079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:19 smithi116 ceph-mon[37097]: pgmap v3079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:19 smithi171 ceph-mon[37789]: pgmap v3079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4825.688680 secs 2024-07-26T10:01:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:21 smithi044 ceph-mon[32404]: pgmap v3080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:01:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4825.688680 secs 2024-07-26T10:01:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:21 smithi116 ceph-mon[37097]: pgmap v3080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:01:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4825.688680 secs 2024-07-26T10:01:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:21 smithi171 ceph-mon[37789]: pgmap v3080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:01:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:23 smithi044 ceph-mon[32404]: pgmap v3081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:23 smithi116 ceph-mon[37097]: pgmap v3081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:23 smithi171 ceph-mon[37789]: pgmap v3081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:25 smithi044 ceph-mon[32404]: pgmap v3082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:25 smithi116 ceph-mon[37097]: pgmap v3082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:26.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:25 smithi171 ceph-mon[37789]: pgmap v3082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:26.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:26.798+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:01:26.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:26.798+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:26.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:26.798+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:01:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4830.688814 secs 2024-07-26T10:01:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4830.688814 secs 2024-07-26T10:01:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4830.688814 secs 2024-07-26T10:01:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:27 smithi044 ceph-mon[32404]: pgmap v3083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:27 smithi116 ceph-mon[37097]: pgmap v3083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:27 smithi171 ceph-mon[37789]: pgmap v3083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:29 smithi044 ceph-mon[32404]: pgmap v3084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:29 smithi116 ceph-mon[37097]: pgmap v3084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:29 smithi171 ceph-mon[37789]: pgmap v3084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4835.688959 secs 2024-07-26T10:01:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:31 smithi044 ceph-mon[32404]: pgmap v3085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4835.688959 secs 2024-07-26T10:01:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:31 smithi116 ceph-mon[37097]: pgmap v3085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4835.688959 secs 2024-07-26T10:01:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:31 smithi171 ceph-mon[37789]: pgmap v3085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:33 smithi044 ceph-mon[32404]: pgmap v3086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:33 smithi116 ceph-mon[37097]: pgmap v3086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:33 smithi171 ceph-mon[37789]: pgmap v3086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:35 smithi044 ceph-mon[32404]: pgmap v3087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:35 smithi116 ceph-mon[37097]: pgmap v3087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:35 smithi171 ceph-mon[37789]: pgmap v3087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:01:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4840.689120 secs 2024-07-26T10:01:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4840.689120 secs 2024-07-26T10:01:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4840.689120 secs 2024-07-26T10:01:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:37 smithi044 ceph-mon[32404]: pgmap v3088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:37 smithi116 ceph-mon[37097]: pgmap v3088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:37 smithi171 ceph-mon[37789]: pgmap v3088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:39 smithi044 ceph-mon[32404]: pgmap v3089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:39 smithi116 ceph-mon[37097]: pgmap v3089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:39 smithi171 ceph-mon[37789]: pgmap v3089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:41.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:41.800+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:01:41.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:41.800+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:01:41.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:41.800+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4845.689253 secs 2024-07-26T10:01:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:42 smithi044 ceph-mon[32404]: pgmap v3090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4845.689253 secs 2024-07-26T10:01:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:42 smithi116 ceph-mon[37097]: pgmap v3090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4845.689253 secs 2024-07-26T10:01:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:42 smithi171 ceph-mon[37789]: pgmap v3090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:43 smithi171 ceph-mon[37789]: pgmap v3091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:01:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:43 smithi044 ceph-mon[32404]: pgmap v3091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:01:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:43 smithi116 ceph-mon[37097]: pgmap v3091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:01:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:45 smithi171 ceph-mon[37789]: pgmap v3092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:45 smithi044 ceph-mon[32404]: pgmap v3092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:45 smithi116 ceph-mon[37097]: pgmap v3092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4850.689402 secs 2024-07-26T10:01:47.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:47 smithi171 ceph-mon[37789]: pgmap v3093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:01:47.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4850.689402 secs 2024-07-26T10:01:47.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:47 smithi044 ceph-mon[32404]: pgmap v3093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:01:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4850.689402 secs 2024-07-26T10:01:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:47 smithi116 ceph-mon[37097]: pgmap v3093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:01:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:49 smithi044 ceph-mon[32404]: pgmap v3094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:49 smithi116 ceph-mon[37097]: pgmap v3094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:49.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:49 smithi171 ceph-mon[37789]: pgmap v3094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:51 smithi044 ceph-mon[32404]: pgmap v3095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:01:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4855.689568 secs 2024-07-26T10:01:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:51 smithi116 ceph-mon[37097]: pgmap v3095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:01:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4855.689568 secs 2024-07-26T10:01:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:51 smithi171 ceph-mon[37789]: pgmap v3095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:01:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4855.689568 secs 2024-07-26T10:01:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:53 smithi044 ceph-mon[32404]: pgmap v3096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:53 smithi116 ceph-mon[37097]: pgmap v3096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:53 smithi171 ceph-mon[37789]: pgmap v3096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:01:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:55 smithi044 ceph-mon[32404]: pgmap v3097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:55 smithi116 ceph-mon[37097]: pgmap v3097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:55 smithi171 ceph-mon[37789]: pgmap v3097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:01:56.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4860.689730 secs 2024-07-26T10:01:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4860.689730 secs 2024-07-26T10:01:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:56.810+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:01:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:56.810+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:01:56.810+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:01:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4860.689730 secs 2024-07-26T10:01:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:57 smithi044 ceph-mon[32404]: pgmap v3098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:57 smithi116 ceph-mon[37097]: pgmap v3098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:57 smithi171 ceph-mon[37789]: pgmap v3098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:01:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:01:59 smithi044 ceph-mon[32404]: pgmap v3099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:01:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:01:59 smithi116 ceph-mon[37097]: pgmap v3099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:01:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:01:59 smithi171 ceph-mon[37789]: pgmap v3099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:01 smithi044 ceph-mon[32404]: pgmap v3100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4865.689838 secs 2024-07-26T10:02:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:01 smithi116 ceph-mon[37097]: pgmap v3100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4865.689838 secs 2024-07-26T10:02:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:01 smithi171 ceph-mon[37789]: pgmap v3100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4865.689838 secs 2024-07-26T10:02:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:03 smithi044 ceph-mon[32404]: pgmap v3101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:03.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:03 smithi116 ceph-mon[37097]: pgmap v3101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:03.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:03 smithi171 ceph-mon[37789]: pgmap v3101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:05 smithi044 ceph-mon[32404]: pgmap v3102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:05 smithi116 ceph-mon[37097]: pgmap v3102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:05 smithi171 ceph-mon[37789]: pgmap v3102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:06.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4870.690002 secs 2024-07-26T10:02:06.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4870.690002 secs 2024-07-26T10:02:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4870.690002 secs 2024-07-26T10:02:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:07 smithi044 ceph-mon[32404]: pgmap v3103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:07 smithi116 ceph-mon[37097]: pgmap v3103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:07 smithi171 ceph-mon[37789]: pgmap v3103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:09 smithi044 ceph-mon[32404]: pgmap v3104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:09 smithi116 ceph-mon[37097]: pgmap v3104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:09 smithi171 ceph-mon[37789]: pgmap v3104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:11 smithi044 ceph-mon[32404]: pgmap v3105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4875.690150 secs 2024-07-26T10:02:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:11 smithi116 ceph-mon[37097]: pgmap v3105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4875.690150 secs 2024-07-26T10:02:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:11.814+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:02:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:11.814+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:11.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:11.814+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:02:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:11 smithi171 ceph-mon[37789]: pgmap v3105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4875.690150 secs 2024-07-26T10:02:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:13 smithi044 ceph-mon[32404]: pgmap v3106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:13 smithi116 ceph-mon[37097]: pgmap v3106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:13 smithi171 ceph-mon[37789]: pgmap v3106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:15.536+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003900 con 0x7fcf6c1057b0 2024-07-26T10:02:15.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:15.536+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:02:15.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:15.536+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003900 con 0x7fcf6c1057b0 2024-07-26T10:02:15.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:15.537+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:02:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:15 smithi044 ceph-mon[32404]: pgmap v3107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:15 smithi116 ceph-mon[37097]: pgmap v3107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:15 smithi171 ceph-mon[37789]: pgmap v3107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4880.690293 secs 2024-07-26T10:02:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4880.690293 secs 2024-07-26T10:02:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4880.690293 secs 2024-07-26T10:02:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:17 smithi044 ceph-mon[32404]: pgmap v3108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:17 smithi116 ceph-mon[37097]: pgmap v3108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:17 smithi171 ceph-mon[37789]: pgmap v3108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:19 smithi044 ceph-mon[32404]: pgmap v3109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:19 smithi116 ceph-mon[37097]: pgmap v3109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:19 smithi171 ceph-mon[37789]: pgmap v3109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:21 smithi044 ceph-mon[32404]: pgmap v3110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4885.690437 secs 2024-07-26T10:02:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:21 smithi116 ceph-mon[37097]: pgmap v3110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4885.690437 secs 2024-07-26T10:02:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:21 smithi171 ceph-mon[37789]: pgmap v3110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4885.690437 secs 2024-07-26T10:02:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:23 smithi044 ceph-mon[32404]: pgmap v3111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:23 smithi116 ceph-mon[37097]: pgmap v3111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:23 smithi171 ceph-mon[37789]: pgmap v3111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:25 smithi044 ceph-mon[32404]: pgmap v3112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:25 smithi116 ceph-mon[37097]: pgmap v3112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:25 smithi171 ceph-mon[37789]: pgmap v3112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4890.690598 secs 2024-07-26T10:02:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4890.690598 secs 2024-07-26T10:02:26.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:26.823+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:02:26.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:26.823+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:26.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:26.823+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:02:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4890.690598 secs 2024-07-26T10:02:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:27 smithi044 ceph-mon[32404]: pgmap v3113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:27 smithi116 ceph-mon[37097]: pgmap v3113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:27 smithi171 ceph-mon[37789]: pgmap v3113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:29 smithi044 ceph-mon[32404]: pgmap v3114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:29 smithi116 ceph-mon[37097]: pgmap v3114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:29 smithi171 ceph-mon[37789]: pgmap v3114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:02:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:31 smithi044 ceph-mon[32404]: pgmap v3115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4895.690759 secs 2024-07-26T10:02:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:31 smithi116 ceph-mon[37097]: pgmap v3115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4895.690759 secs 2024-07-26T10:02:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:31 smithi171 ceph-mon[37789]: pgmap v3115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4895.690759 secs 2024-07-26T10:02:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:33 smithi044 ceph-mon[32404]: pgmap v3116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:33 smithi116 ceph-mon[37097]: pgmap v3116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:33 smithi171 ceph-mon[37789]: pgmap v3116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:35 smithi116 ceph-mon[37097]: pgmap v3117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:35 smithi171 ceph-mon[37789]: pgmap v3117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:35 smithi044 ceph-mon[32404]: pgmap v3117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:02:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4900.690919 secs 2024-07-26T10:02:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4900.690919 secs 2024-07-26T10:02:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4900.690919 secs 2024-07-26T10:02:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:37 smithi171 ceph-mon[37789]: pgmap v3118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:37 smithi044 ceph-mon[32404]: pgmap v3118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:37 smithi116 ceph-mon[37097]: pgmap v3118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:39 smithi171 ceph-mon[37789]: pgmap v3119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:02:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:39 smithi044 ceph-mon[32404]: pgmap v3119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:02:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:39 smithi116 ceph-mon[37097]: pgmap v3119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:02:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:41.826+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:02:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:41.826+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:41.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:41.826+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:02:41.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:41 smithi044 ceph-mon[32404]: pgmap v3120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:02:41.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4905.691069 secs 2024-07-26T10:02:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:41 smithi171 ceph-mon[37789]: pgmap v3120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:02:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4905.691069 secs 2024-07-26T10:02:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:41 smithi116 ceph-mon[37097]: pgmap v3120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:02:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4905.691069 secs 2024-07-26T10:02:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:43 smithi171 ceph-mon[37789]: pgmap v3121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:02:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:43.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:43.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:02:44.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:43 smithi044 ceph-mon[32404]: pgmap v3121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:02:44.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:44.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:44.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:02:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:43 smithi116 ceph-mon[37097]: pgmap v3121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:02:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:02:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:44 smithi171 ceph-mon[37789]: pgmap v3122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:02:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:44 smithi171 ceph-mon[37789]: pgmap v3123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:44 smithi044 ceph-mon[32404]: pgmap v3122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:02:45.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:44 smithi044 ceph-mon[32404]: pgmap v3123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:44 smithi116 ceph-mon[37097]: pgmap v3122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:02:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:44 smithi116 ceph-mon[37097]: pgmap v3123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:02:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:46 smithi171 ceph-mon[37789]: pgmap v3124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:02:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4910.691231 secs 2024-07-26T10:02:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:46 smithi044 ceph-mon[32404]: pgmap v3124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:02:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4910.691231 secs 2024-07-26T10:02:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:46 smithi116 ceph-mon[37097]: pgmap v3124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:02:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4910.691231 secs 2024-07-26T10:02:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:48 smithi171 ceph-mon[37789]: pgmap v3125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:48 smithi044 ceph-mon[32404]: pgmap v3125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:48 smithi116 ceph-mon[37097]: pgmap v3125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:50 smithi044 ceph-mon[32404]: pgmap v3126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:02:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:50 smithi116 ceph-mon[37097]: pgmap v3126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:02:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:50 smithi171 ceph-mon[37789]: pgmap v3126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:02:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4915.691385 secs 2024-07-26T10:02:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4915.691385 secs 2024-07-26T10:02:52.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4915.691385 secs 2024-07-26T10:02:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:52 smithi044 ceph-mon[32404]: pgmap v3127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:52 smithi116 ceph-mon[37097]: pgmap v3127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:52 smithi171 ceph-mon[37789]: pgmap v3127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:55.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:54 smithi044 ceph-mon[32404]: pgmap v3128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:02:55.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:54 smithi116 ceph-mon[37097]: pgmap v3128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:02:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:54 smithi171 ceph-mon[37789]: pgmap v3128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:02:56.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:56.836+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:02:56.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:56.836+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:56.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:02:56.836+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:02:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:56 smithi044 ceph-mon[32404]: pgmap v3129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4920.691558 secs 2024-07-26T10:02:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:56 smithi116 ceph-mon[37097]: pgmap v3129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4920.691558 secs 2024-07-26T10:02:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:56 smithi171 ceph-mon[37789]: pgmap v3129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:02:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4920.691558 secs 2024-07-26T10:02:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:02:58 smithi044 ceph-mon[32404]: pgmap v3130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:02:58 smithi116 ceph-mon[37097]: pgmap v3130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:02:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:02:58 smithi171 ceph-mon[37789]: pgmap v3130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:00 smithi044 ceph-mon[32404]: pgmap v3131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:00 smithi116 ceph-mon[37097]: pgmap v3131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:00 smithi171 ceph-mon[37789]: pgmap v3131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4925.691705 secs 2024-07-26T10:03:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4925.691705 secs 2024-07-26T10:03:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4925.691705 secs 2024-07-26T10:03:02.946 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:02 smithi044 ceph-mon[32404]: pgmap v3132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:03.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:02 smithi116 ceph-mon[37097]: pgmap v3132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:02 smithi171 ceph-mon[37789]: pgmap v3132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:04 smithi044 ceph-mon[32404]: pgmap v3133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:03:05.027 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:04 smithi116 ceph-mon[37097]: pgmap v3133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:03:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:04 smithi171 ceph-mon[37789]: pgmap v3133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:03:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:06 smithi044 ceph-mon[32404]: pgmap v3134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:03:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4930.691798 secs 2024-07-26T10:03:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:06 smithi116 ceph-mon[37097]: pgmap v3134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:03:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4930.691798 secs 2024-07-26T10:03:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:06 smithi171 ceph-mon[37789]: pgmap v3134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:03:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4930.691798 secs 2024-07-26T10:03:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:08 smithi044 ceph-mon[32404]: pgmap v3135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:03:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:08 smithi116 ceph-mon[37097]: pgmap v3135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:03:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:08 smithi171 ceph-mon[37789]: pgmap v3135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:03:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:10 smithi044 ceph-mon[32404]: pgmap v3136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:10 smithi116 ceph-mon[37097]: pgmap v3136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:10 smithi171 ceph-mon[37789]: pgmap v3136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:03:11.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:11.840+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:03:11.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:11.840+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:11.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:11.840+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:03:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4935.691943 secs 2024-07-26T10:03:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4935.691943 secs 2024-07-26T10:03:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4935.691943 secs 2024-07-26T10:03:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:12 smithi044 ceph-mon[32404]: pgmap v3137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:03:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:12 smithi116 ceph-mon[37097]: pgmap v3137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:03:13.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:12 smithi171 ceph-mon[37789]: pgmap v3137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:03:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:14 smithi044 ceph-mon[32404]: pgmap v3138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:03:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:14 smithi116 ceph-mon[37097]: pgmap v3138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:03:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:14 smithi171 ceph-mon[37789]: pgmap v3138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:03:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:16 smithi044 ceph-mon[32404]: pgmap v3139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4940.692095 secs 2024-07-26T10:03:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:16 smithi116 ceph-mon[37097]: pgmap v3139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4940.692095 secs 2024-07-26T10:03:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:16 smithi171 ceph-mon[37789]: pgmap v3139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4940.692095 secs 2024-07-26T10:03:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:17 smithi044 ceph-mon[32404]: pgmap v3140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:17 smithi116 ceph-mon[37097]: pgmap v3140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:17 smithi171 ceph-mon[37789]: pgmap v3140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:20 smithi171 ceph-mon[37789]: pgmap v3141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:20 smithi044 ceph-mon[32404]: pgmap v3141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:20 smithi116 ceph-mon[37097]: pgmap v3141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4945.692250 secs 2024-07-26T10:03:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4945.692250 secs 2024-07-26T10:03:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4945.692250 secs 2024-07-26T10:03:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:22 smithi044 ceph-mon[32404]: pgmap v3142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:22 smithi116 ceph-mon[37097]: pgmap v3142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:22 smithi171 ceph-mon[37789]: pgmap v3142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:24 smithi044 ceph-mon[32404]: pgmap v3143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:24 smithi116 ceph-mon[37097]: pgmap v3143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:24 smithi171 ceph-mon[37789]: pgmap v3143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:26 smithi044 ceph-mon[32404]: pgmap v3144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4950.692415 secs 2024-07-26T10:03:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:26 smithi116 ceph-mon[37097]: pgmap v3144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4950.692415 secs 2024-07-26T10:03:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:26 smithi171 ceph-mon[37789]: pgmap v3144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4950.692415 secs 2024-07-26T10:03:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:26.841+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:03:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:26.841+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:26.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:26.841+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:03:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:28 smithi044 ceph-mon[32404]: pgmap v3145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:28 smithi116 ceph-mon[37097]: pgmap v3145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:28 smithi171 ceph-mon[37789]: pgmap v3145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:30 smithi044 ceph-mon[32404]: pgmap v3146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:30 smithi116 ceph-mon[37097]: pgmap v3146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:30 smithi171 ceph-mon[37789]: pgmap v3146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4955.692531 secs 2024-07-26T10:03:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4955.692531 secs 2024-07-26T10:03:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4955.692531 secs 2024-07-26T10:03:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:32 smithi044 ceph-mon[32404]: pgmap v3147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:32 smithi116 ceph-mon[37097]: pgmap v3147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:32 smithi171 ceph-mon[37789]: pgmap v3147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:34.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:34 smithi044 ceph-mon[32404]: pgmap v3148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:34 smithi116 ceph-mon[37097]: pgmap v3148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:34 smithi171 ceph-mon[37789]: pgmap v3148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:36 smithi044 ceph-mon[32404]: pgmap v3149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4960.692687 secs 2024-07-26T10:03:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:36 smithi116 ceph-mon[37097]: pgmap v3149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4960.692687 secs 2024-07-26T10:03:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:36 smithi171 ceph-mon[37789]: pgmap v3149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:36.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4960.692687 secs 2024-07-26T10:03:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:38 smithi044 ceph-mon[32404]: pgmap v3150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:38 smithi116 ceph-mon[37097]: pgmap v3150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:38 smithi171 ceph-mon[37789]: pgmap v3150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:40 smithi044 ceph-mon[32404]: pgmap v3151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:40 smithi116 ceph-mon[37097]: pgmap v3151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:40.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:40 smithi171 ceph-mon[37789]: pgmap v3151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4965.692847 secs 2024-07-26T10:03:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4965.692847 secs 2024-07-26T10:03:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4965.692847 secs 2024-07-26T10:03:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:41.843+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:03:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:41.843+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:03:41.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:41.843+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:42 smithi044 ceph-mon[32404]: pgmap v3152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:42 smithi116 ceph-mon[37097]: pgmap v3152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:42 smithi171 ceph-mon[37789]: pgmap v3152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:44.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:44 smithi116 ceph-mon[37097]: pgmap v3153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:44.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:44 smithi044 ceph-mon[32404]: pgmap v3153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:44 smithi171 ceph-mon[37789]: pgmap v3153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:03:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:45.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:45.551+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:03:45.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:45.551+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:03:45.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:45.552+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:03:45.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:45.552+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:03:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:46 smithi044 ceph-mon[32404]: pgmap v3154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4970.693006 secs 2024-07-26T10:03:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:46 smithi116 ceph-mon[37097]: pgmap v3154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4970.693006 secs 2024-07-26T10:03:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:46 smithi171 ceph-mon[37789]: pgmap v3154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4970.693006 secs 2024-07-26T10:03:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:48 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:48.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:48 smithi044 ceph-mon[32404]: pgmap v3155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:48 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:48 smithi116 ceph-mon[37097]: pgmap v3155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:48 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:48 smithi171 ceph-mon[37789]: pgmap v3155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:03:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:50 smithi044 ceph-mon[32404]: pgmap v3156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:50 smithi116 ceph-mon[37097]: pgmap v3156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:50 smithi171 ceph-mon[37789]: pgmap v3156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:03:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4975.693119 secs 2024-07-26T10:03:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: pgmap v3157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: pgmap v3158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:51.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4975.693119 secs 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: pgmap v3157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: pgmap v3158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4975.693119 secs 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: pgmap v3157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: pgmap v3158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:03:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:53 smithi044 ceph-mon[32404]: pgmap v3159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:53 smithi116 ceph-mon[37097]: pgmap v3159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:53 smithi171 ceph-mon[37789]: pgmap v3159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:03:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:55 smithi044 ceph-mon[32404]: pgmap v3160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:55.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:55 smithi116 ceph-mon[37097]: pgmap v3160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:55 smithi171 ceph-mon[37789]: pgmap v3160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:56.755 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4980.693275 secs 2024-07-26T10:03:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4980.693275 secs 2024-07-26T10:03:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4980.693275 secs 2024-07-26T10:03:56.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:56.849+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:03:56.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:56.849+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:56.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:03:56.849+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:03:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:57 smithi044 ceph-mon[32404]: pgmap v3161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:03:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:57 smithi116 ceph-mon[37097]: pgmap v3161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:03:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:57 smithi171 ceph-mon[37789]: pgmap v3161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:03:59.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:03:59 smithi044 ceph-mon[32404]: pgmap v3162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:03:59 smithi116 ceph-mon[37097]: pgmap v3162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:03:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:03:59 smithi171 ceph-mon[37789]: pgmap v3162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:04:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4985.693444 secs 2024-07-26T10:04:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:01 smithi044 ceph-mon[32404]: pgmap v3163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4985.693444 secs 2024-07-26T10:04:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:01 smithi116 ceph-mon[37097]: pgmap v3163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4985.693444 secs 2024-07-26T10:04:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:01 smithi171 ceph-mon[37789]: pgmap v3163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:04:03.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:03 smithi044 ceph-mon[32404]: pgmap v3164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:03 smithi116 ceph-mon[37097]: pgmap v3164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:03 smithi171 ceph-mon[37789]: pgmap v3164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:05 smithi044 ceph-mon[32404]: pgmap v3165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:05 smithi116 ceph-mon[37097]: pgmap v3165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:05 smithi171 ceph-mon[37789]: pgmap v3165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4990.693602 secs 2024-07-26T10:04:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4990.693602 secs 2024-07-26T10:04:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4990.693602 secs 2024-07-26T10:04:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:07 smithi044 ceph-mon[32404]: pgmap v3166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:07 smithi116 ceph-mon[37097]: pgmap v3166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:07 smithi171 ceph-mon[37789]: pgmap v3166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:09 smithi044 ceph-mon[32404]: pgmap v3167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:09 smithi116 ceph-mon[37097]: pgmap v3167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:09 smithi171 ceph-mon[37789]: pgmap v3167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 4995.693766 secs 2024-07-26T10:04:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:11 smithi116 ceph-mon[37097]: pgmap v3168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 4995.693766 secs 2024-07-26T10:04:11.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:11 smithi171 ceph-mon[37789]: pgmap v3168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:11.856 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 4995.693766 secs 2024-07-26T10:04:11.856 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:11 smithi044 ceph-mon[32404]: pgmap v3168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:11.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:11.853+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:04:11.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:11.853+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:04:11.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:11.853+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:13 smithi171 ceph-mon[37789]: pgmap v3169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:13 smithi044 ceph-mon[32404]: pgmap v3169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:13 smithi116 ceph-mon[37097]: pgmap v3169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:15 smithi171 ceph-mon[37789]: pgmap v3170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:15 smithi044 ceph-mon[32404]: pgmap v3170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:15 smithi116 ceph-mon[37097]: pgmap v3170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5000.693930 secs 2024-07-26T10:04:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5000.693930 secs 2024-07-26T10:04:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5000.693930 secs 2024-07-26T10:04:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:17 smithi044 ceph-mon[32404]: pgmap v3171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:17 smithi116 ceph-mon[37097]: pgmap v3171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:17 smithi171 ceph-mon[37789]: pgmap v3171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:19.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:19 smithi171 ceph-mon[37789]: pgmap v3172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:19 smithi044 ceph-mon[32404]: pgmap v3172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:19 smithi116 ceph-mon[37097]: pgmap v3172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5005.694074 secs 2024-07-26T10:04:21.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:21 smithi171 ceph-mon[37789]: pgmap v3173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5005.694074 secs 2024-07-26T10:04:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:21 smithi044 ceph-mon[32404]: pgmap v3173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5005.694074 secs 2024-07-26T10:04:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:21 smithi116 ceph-mon[37097]: pgmap v3173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:23 smithi171 ceph-mon[37789]: pgmap v3174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:23 smithi044 ceph-mon[32404]: pgmap v3174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:23 smithi116 ceph-mon[37097]: pgmap v3174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:25 smithi171 ceph-mon[37789]: pgmap v3175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:25 smithi044 ceph-mon[32404]: pgmap v3175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:25 smithi116 ceph-mon[37097]: pgmap v3175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5010.694240 secs 2024-07-26T10:04:26.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:26.861+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:04:26.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:26.862+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:04:26.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:26.862+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:26.865 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5010.694240 secs 2024-07-26T10:04:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5010.694240 secs 2024-07-26T10:04:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:27 smithi171 ceph-mon[37789]: pgmap v3176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:27 smithi044 ceph-mon[32404]: pgmap v3176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:27 smithi116 ceph-mon[37097]: pgmap v3176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:29 smithi044 ceph-mon[32404]: pgmap v3177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:29 smithi116 ceph-mon[37097]: pgmap v3177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:29 smithi171 ceph-mon[37789]: pgmap v3177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5015.694391 secs 2024-07-26T10:04:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:31 smithi044 ceph-mon[32404]: pgmap v3178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5015.694391 secs 2024-07-26T10:04:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:31 smithi116 ceph-mon[37097]: pgmap v3178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5015.694391 secs 2024-07-26T10:04:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:31 smithi171 ceph-mon[37789]: pgmap v3178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:33 smithi044 ceph-mon[32404]: pgmap v3179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:33 smithi116 ceph-mon[37097]: pgmap v3179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:33 smithi171 ceph-mon[37789]: pgmap v3179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:35 smithi044 ceph-mon[32404]: pgmap v3180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:35 smithi116 ceph-mon[37097]: pgmap v3180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:35 smithi171 ceph-mon[37789]: pgmap v3180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:36.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5020.694559 secs 2024-07-26T10:04:36.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5020.694559 secs 2024-07-26T10:04:36.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5020.694559 secs 2024-07-26T10:04:38.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:37 smithi044 ceph-mon[32404]: pgmap v3181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:37 smithi116 ceph-mon[37097]: pgmap v3181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:37 smithi171 ceph-mon[37789]: pgmap v3181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:39 smithi044 ceph-mon[32404]: pgmap v3182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:39 smithi116 ceph-mon[37097]: pgmap v3182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:39 smithi171 ceph-mon[37789]: pgmap v3182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:41.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:41.866+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:04:41.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:41.866+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:41.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:41.866+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:04:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5025.694730 secs 2024-07-26T10:04:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:41 smithi044 ceph-mon[32404]: pgmap v3183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5025.694730 secs 2024-07-26T10:04:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:41 smithi116 ceph-mon[37097]: pgmap v3183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5025.694730 secs 2024-07-26T10:04:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:41 smithi171 ceph-mon[37789]: pgmap v3183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:44.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:43 smithi044 ceph-mon[32404]: pgmap v3184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:43 smithi116 ceph-mon[37097]: pgmap v3184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:43 smithi171 ceph-mon[37789]: pgmap v3184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:45 smithi044 ceph-mon[32404]: pgmap v3185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:45 smithi116 ceph-mon[37097]: pgmap v3185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:45 smithi171 ceph-mon[37789]: pgmap v3185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5030.694904 secs 2024-07-26T10:04:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5030.694904 secs 2024-07-26T10:04:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5030.694904 secs 2024-07-26T10:04:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:47 smithi044 ceph-mon[32404]: pgmap v3186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:47 smithi116 ceph-mon[37097]: pgmap v3186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:47 smithi171 ceph-mon[37789]: pgmap v3186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:49 smithi044 ceph-mon[32404]: pgmap v3187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:49 smithi116 ceph-mon[37097]: pgmap v3187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:49 smithi171 ceph-mon[37789]: pgmap v3187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:04:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:50 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:50 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:50 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5035.695064 secs 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: pgmap v3188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5035.695064 secs 2024-07-26T10:04:52.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: pgmap v3188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:52.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5035.695064 secs 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: pgmap v3188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:52 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:04:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:52 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:04:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:52 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:04:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:52 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:04:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:52 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:04:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:52 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:04:53.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:53 smithi171 ceph-mon[37789]: pgmap v3189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:53 smithi044 ceph-mon[32404]: pgmap v3189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:53 smithi116 ceph-mon[37097]: pgmap v3189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: pgmap v3190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: pgmap v3191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: pgmap v3192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:04:55.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: pgmap v3190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:56.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:56.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: pgmap v3191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:04:56.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: pgmap v3192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:04:56.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: pgmap v3190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: pgmap v3191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: pgmap v3192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:04:56.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:04:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5040.695208 secs 2024-07-26T10:04:56.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:56.874+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:04:56.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:56.874+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:56.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:04:56.874+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:04:56.878 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5040.695208 secs 2024-07-26T10:04:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5040.695208 secs 2024-07-26T10:04:58.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:57 smithi044 ceph-mon[32404]: pgmap v3193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:57 smithi116 ceph-mon[37097]: pgmap v3193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:04:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:57 smithi171 ceph-mon[37789]: pgmap v3193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:04:59 smithi044 ceph-mon[32404]: pgmap v3194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:05:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:04:59 smithi116 ceph-mon[37097]: pgmap v3194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:05:00.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:04:59 smithi171 ceph-mon[37789]: pgmap v3194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:05:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5045.695373 secs 2024-07-26T10:05:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:01 smithi044 ceph-mon[32404]: pgmap v3195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5045.695373 secs 2024-07-26T10:05:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:01 smithi116 ceph-mon[37097]: pgmap v3195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5045.695373 secs 2024-07-26T10:05:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:01 smithi171 ceph-mon[37789]: pgmap v3195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:03 smithi044 ceph-mon[32404]: pgmap v3196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:03 smithi116 ceph-mon[37097]: pgmap v3196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:03 smithi171 ceph-mon[37789]: pgmap v3196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:05:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:05 smithi044 ceph-mon[32404]: pgmap v3197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:05:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:05 smithi116 ceph-mon[37097]: pgmap v3197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:05:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:05 smithi171 ceph-mon[37789]: pgmap v3197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:05:06.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5050.695539 secs 2024-07-26T10:05:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5050.695539 secs 2024-07-26T10:05:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5050.695539 secs 2024-07-26T10:05:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:07 smithi044 ceph-mon[32404]: pgmap v3198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:07 smithi116 ceph-mon[37097]: pgmap v3198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:07 smithi171 ceph-mon[37789]: pgmap v3198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:09 smithi044 ceph-mon[32404]: pgmap v3199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:09 smithi116 ceph-mon[37097]: pgmap v3199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:09 smithi171 ceph-mon[37789]: pgmap v3199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:10 smithi171 ceph-mon[37789]: pgmap v3200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:10 smithi044 ceph-mon[32404]: pgmap v3200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:10 smithi116 ceph-mon[37097]: pgmap v3200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:11.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:11.878+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:05:11.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:11.878+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:11.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:11.878+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:05:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5055.695705 secs 2024-07-26T10:05:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5055.695705 secs 2024-07-26T10:05:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5055.695705 secs 2024-07-26T10:05:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:12 smithi171 ceph-mon[37789]: pgmap v3201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:12 smithi044 ceph-mon[32404]: pgmap v3201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:12 smithi116 ceph-mon[37097]: pgmap v3201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:15.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:15.569+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:05:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:15.569+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:05:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:15.569+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:05:15.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:15.570+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:05:16.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:15 smithi044 ceph-mon[32404]: pgmap v3202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:15 smithi116 ceph-mon[37097]: pgmap v3202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:15 smithi171 ceph-mon[37789]: pgmap v3202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5060.695868 secs 2024-07-26T10:05:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5060.695868 secs 2024-07-26T10:05:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5060.695868 secs 2024-07-26T10:05:18.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:17 smithi044 ceph-mon[32404]: pgmap v3203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:17 smithi116 ceph-mon[37097]: pgmap v3203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:17 smithi171 ceph-mon[37789]: pgmap v3203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:19 smithi044 ceph-mon[32404]: pgmap v3204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:19 smithi116 ceph-mon[37097]: pgmap v3204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:19 smithi171 ceph-mon[37789]: pgmap v3204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5065.696018 secs 2024-07-26T10:05:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:21 smithi044 ceph-mon[32404]: pgmap v3205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5065.696018 secs 2024-07-26T10:05:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:21 smithi116 ceph-mon[37097]: pgmap v3205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5065.696018 secs 2024-07-26T10:05:22.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:21 smithi171 ceph-mon[37789]: pgmap v3205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:23 smithi044 ceph-mon[32404]: pgmap v3206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:23 smithi116 ceph-mon[37097]: pgmap v3206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:23 smithi171 ceph-mon[37789]: pgmap v3206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:24 smithi044 ceph-mon[32404]: pgmap v3207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:24 smithi116 ceph-mon[37097]: pgmap v3207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:25.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:24 smithi171 ceph-mon[37789]: pgmap v3207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5070.696168 secs 2024-07-26T10:05:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5070.696168 secs 2024-07-26T10:05:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5070.696168 secs 2024-07-26T10:05:26.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:26.887+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:05:26.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:26.887+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:26.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:26.887+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:05:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:27 smithi044 ceph-mon[32404]: pgmap v3208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:27 smithi116 ceph-mon[37097]: pgmap v3208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:27 smithi171 ceph-mon[37789]: pgmap v3208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:30.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:29 smithi044 ceph-mon[32404]: pgmap v3209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:29 smithi116 ceph-mon[37097]: pgmap v3209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:29 smithi171 ceph-mon[37789]: pgmap v3209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5075.696310 secs 2024-07-26T10:05:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:31 smithi044 ceph-mon[32404]: pgmap v3210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5075.696310 secs 2024-07-26T10:05:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:31 smithi116 ceph-mon[37097]: pgmap v3210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5075.696310 secs 2024-07-26T10:05:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:31 smithi171 ceph-mon[37789]: pgmap v3210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:33 smithi044 ceph-mon[32404]: pgmap v3211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:34.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:33 smithi116 ceph-mon[37097]: pgmap v3211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:33 smithi171 ceph-mon[37789]: pgmap v3211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:35 smithi044 ceph-mon[32404]: pgmap v3212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:35 smithi116 ceph-mon[37097]: pgmap v3212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:35 smithi171 ceph-mon[37789]: pgmap v3212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:36.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5080.696472 secs 2024-07-26T10:05:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5080.696472 secs 2024-07-26T10:05:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5080.696472 secs 2024-07-26T10:05:38.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:37 smithi044 ceph-mon[32404]: pgmap v3213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:37 smithi116 ceph-mon[37097]: pgmap v3213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:37 smithi171 ceph-mon[37789]: pgmap v3213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:38.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:38 smithi044 ceph-mon[32404]: pgmap v3214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:38 smithi116 ceph-mon[37097]: pgmap v3214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:38 smithi171 ceph-mon[37789]: pgmap v3214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:41.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:41.891+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:05:41.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:41.891+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:41.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:41.891+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5085.696674 secs 2024-07-26T10:05:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:41 smithi044 ceph-mon[32404]: pgmap v3215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5085.696674 secs 2024-07-26T10:05:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:41 smithi116 ceph-mon[37097]: pgmap v3215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5085.696674 secs 2024-07-26T10:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:41 smithi171 ceph-mon[37789]: pgmap v3215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:43 smithi044 ceph-mon[32404]: pgmap v3216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:43 smithi116 ceph-mon[37097]: pgmap v3216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:43 smithi171 ceph-mon[37789]: pgmap v3216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:45 smithi044 ceph-mon[32404]: pgmap v3217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:45 smithi116 ceph-mon[37097]: pgmap v3217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:45 smithi171 ceph-mon[37789]: pgmap v3217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5090.696818 secs 2024-07-26T10:05:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5090.696818 secs 2024-07-26T10:05:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5090.696818 secs 2024-07-26T10:05:48.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:47 smithi044 ceph-mon[32404]: pgmap v3218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:47 smithi116 ceph-mon[37097]: pgmap v3218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:47 smithi171 ceph-mon[37789]: pgmap v3218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:48 smithi044 ceph-mon[32404]: pgmap v3219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:48 smithi116 ceph-mon[37097]: pgmap v3219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:48 smithi171 ceph-mon[37789]: pgmap v3219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5095.696991 secs 2024-07-26T10:05:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:51 smithi044 ceph-mon[32404]: pgmap v3220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5095.696991 secs 2024-07-26T10:05:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:51 smithi116 ceph-mon[37097]: pgmap v3220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5095.696991 secs 2024-07-26T10:05:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:51 smithi171 ceph-mon[37789]: pgmap v3220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:05:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:53 smithi044 ceph-mon[32404]: pgmap v3221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:53 smithi116 ceph-mon[37097]: pgmap v3221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:53 smithi171 ceph-mon[37789]: pgmap v3221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:05:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:55 smithi044 ceph-mon[32404]: pgmap v3222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:55 smithi116 ceph-mon[37097]: pgmap v3222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:55 smithi171 ceph-mon[37789]: pgmap v3222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:05:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:56.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:56.899+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:05:56.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:56.899+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:56.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:05:56.899+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:05:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5100.697103 secs 2024-07-26T10:05:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:05:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5100.697103 secs 2024-07-26T10:05:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:05:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5100.697103 secs 2024-07-26T10:05:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:05:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:57 smithi044 ceph-mon[32404]: pgmap v3223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:05:58.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:57 smithi044 ceph-mon[32404]: pgmap v3224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:05:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:57 smithi116 ceph-mon[37097]: pgmap v3223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:05:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:57 smithi116 ceph-mon[37097]: pgmap v3224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:05:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:57 smithi171 ceph-mon[37789]: pgmap v3223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:05:58.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:57 smithi171 ceph-mon[37789]: pgmap v3224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:05:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:05:58 smithi044 ceph-mon[32404]: pgmap v3225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:05:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:05:58 smithi116 ceph-mon[37097]: pgmap v3225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:05:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:05:58 smithi171 ceph-mon[37789]: pgmap v3225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:00 smithi044 ceph-mon[32404]: pgmap v3226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:00 smithi116 ceph-mon[37097]: pgmap v3226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:00 smithi171 ceph-mon[37789]: pgmap v3226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5105.697266 secs 2024-07-26T10:06:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5105.697266 secs 2024-07-26T10:06:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5105.697266 secs 2024-07-26T10:06:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:03 smithi044 ceph-mon[32404]: pgmap v3227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:03 smithi116 ceph-mon[37097]: pgmap v3227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:03 smithi171 ceph-mon[37789]: pgmap v3227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T10:06:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:05 smithi044 ceph-mon[32404]: pgmap v3228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:06:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:05 smithi116 ceph-mon[37097]: pgmap v3228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:06:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:05 smithi171 ceph-mon[37789]: pgmap v3228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:06:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5110.697427 secs 2024-07-26T10:06:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:07 smithi044 ceph-mon[32404]: pgmap v3229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5110.697427 secs 2024-07-26T10:06:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:07 smithi116 ceph-mon[37097]: pgmap v3229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5110.697427 secs 2024-07-26T10:06:07.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:07 smithi171 ceph-mon[37789]: pgmap v3229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:06:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:09 smithi116 ceph-mon[37097]: pgmap v3230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:09 smithi171 ceph-mon[37789]: pgmap v3230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:09 smithi044 ceph-mon[32404]: pgmap v3230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:11 smithi171 ceph-mon[37789]: pgmap v3231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:11 smithi044 ceph-mon[32404]: pgmap v3231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:11 smithi116 ceph-mon[37097]: pgmap v3231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:11.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:11.903+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:06:11.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:11.903+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:11.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:11.903+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:06:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5115.697586 secs 2024-07-26T10:06:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5115.697586 secs 2024-07-26T10:06:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5115.697586 secs 2024-07-26T10:06:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:13 smithi171 ceph-mon[37789]: pgmap v3232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:13 smithi044 ceph-mon[32404]: pgmap v3232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:13 smithi116 ceph-mon[37097]: pgmap v3232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:15 smithi171 ceph-mon[37789]: pgmap v3233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:15 smithi044 ceph-mon[32404]: pgmap v3233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:15 smithi116 ceph-mon[37097]: pgmap v3233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5120.697757 secs 2024-07-26T10:06:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:17 smithi171 ceph-mon[37789]: pgmap v3234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5120.697757 secs 2024-07-26T10:06:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:17 smithi044 ceph-mon[32404]: pgmap v3234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5120.697757 secs 2024-07-26T10:06:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:17 smithi116 ceph-mon[37097]: pgmap v3234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:19 smithi171 ceph-mon[37789]: pgmap v3235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:19 smithi044 ceph-mon[32404]: pgmap v3235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:19 smithi116 ceph-mon[37097]: pgmap v3235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:21 smithi171 ceph-mon[37789]: pgmap v3236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:21 smithi044 ceph-mon[32404]: pgmap v3236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:21 smithi116 ceph-mon[37097]: pgmap v3236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5125.697941 secs 2024-07-26T10:06:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5125.697941 secs 2024-07-26T10:06:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5125.697941 secs 2024-07-26T10:06:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:23 smithi171 ceph-mon[37789]: pgmap v3237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:23 smithi044 ceph-mon[32404]: pgmap v3237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:23 smithi116 ceph-mon[37097]: pgmap v3237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:25 smithi171 ceph-mon[37789]: pgmap v3238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:25 smithi044 ceph-mon[32404]: pgmap v3238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:25 smithi116 ceph-mon[37097]: pgmap v3238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:26.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:26.905+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:06:26.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:26.905+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:06:26.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:26.905+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5130.698098 secs 2024-07-26T10:06:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:27 smithi044 ceph-mon[32404]: pgmap v3239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5130.698098 secs 2024-07-26T10:06:27.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:27 smithi171 ceph-mon[37789]: pgmap v3239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5130.698098 secs 2024-07-26T10:06:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:27 smithi116 ceph-mon[37097]: pgmap v3239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:29 smithi044 ceph-mon[32404]: pgmap v3240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:29 smithi116 ceph-mon[37097]: pgmap v3240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:29 smithi171 ceph-mon[37789]: pgmap v3240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:31 smithi044 ceph-mon[32404]: pgmap v3241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:31 smithi116 ceph-mon[37097]: pgmap v3241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:31 smithi171 ceph-mon[37789]: pgmap v3241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5135.698279 secs 2024-07-26T10:06:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5135.698279 secs 2024-07-26T10:06:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5135.698279 secs 2024-07-26T10:06:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:33 smithi044 ceph-mon[32404]: pgmap v3242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:33 smithi116 ceph-mon[37097]: pgmap v3242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:33 smithi171 ceph-mon[37789]: pgmap v3242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:35 smithi044 ceph-mon[32404]: pgmap v3243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:35 smithi116 ceph-mon[37097]: pgmap v3243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:35 smithi171 ceph-mon[37789]: pgmap v3243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:37.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5140.698438 secs 2024-07-26T10:06:37.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:37 smithi171 ceph-mon[37789]: pgmap v3244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5140.698438 secs 2024-07-26T10:06:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:37 smithi044 ceph-mon[32404]: pgmap v3244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5140.698438 secs 2024-07-26T10:06:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:37 smithi116 ceph-mon[37097]: pgmap v3244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:39 smithi044 ceph-mon[32404]: pgmap v3245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:39 smithi116 ceph-mon[37097]: pgmap v3245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:39 smithi171 ceph-mon[37789]: pgmap v3245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:41.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:41 smithi044 ceph-mon[32404]: pgmap v3246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:41 smithi116 ceph-mon[37097]: pgmap v3246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:41 smithi171 ceph-mon[37789]: pgmap v3246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:41.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:41.909+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:06:41.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:41.909+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:06:41.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:41.909+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5145.698598 secs 2024-07-26T10:06:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5145.698598 secs 2024-07-26T10:06:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5145.698598 secs 2024-07-26T10:06:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:43 smithi044 ceph-mon[32404]: pgmap v3247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:43 smithi116 ceph-mon[37097]: pgmap v3247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:43 smithi171 ceph-mon[37789]: pgmap v3247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:45 smithi044 ceph-mon[32404]: pgmap v3248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:45 smithi116 ceph-mon[37097]: pgmap v3248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:45 smithi171 ceph-mon[37789]: pgmap v3248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5150.698775 secs 2024-07-26T10:06:47.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:47 smithi044 ceph-mon[32404]: pgmap v3249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5150.698775 secs 2024-07-26T10:06:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:47 smithi116 ceph-mon[37097]: pgmap v3249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5150.698775 secs 2024-07-26T10:06:47.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:47 smithi171 ceph-mon[37789]: pgmap v3249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:49 smithi044 ceph-mon[32404]: pgmap v3250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:49.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:49 smithi116 ceph-mon[37097]: pgmap v3250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:49 smithi171 ceph-mon[37789]: pgmap v3250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:06:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:51 smithi044 ceph-mon[32404]: pgmap v3251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5155.698886 secs 2024-07-26T10:06:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:51 smithi116 ceph-mon[37097]: pgmap v3251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5155.698886 secs 2024-07-26T10:06:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:51 smithi171 ceph-mon[37789]: pgmap v3251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5155.698886 secs 2024-07-26T10:06:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:53 smithi044 ceph-mon[32404]: pgmap v3252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:53 smithi116 ceph-mon[37097]: pgmap v3252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:53 smithi171 ceph-mon[37789]: pgmap v3252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:55 smithi116 ceph-mon[37097]: pgmap v3253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:55 smithi171 ceph-mon[37789]: pgmap v3253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:55 smithi044 ceph-mon[32404]: pgmap v3253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:06:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5160.699061 secs 2024-07-26T10:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:56.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5160.699061 secs 2024-07-26T10:06:56.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:56.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:56.917+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:06:56.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:56.917+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:56.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:06:56.917+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:06:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5160.699061 secs 2024-07-26T10:06:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:57 smithi171 ceph-mon[37789]: pgmap v3254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:57.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:57 smithi044 ceph-mon[32404]: pgmap v3254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:58.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:58.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:57 smithi116 ceph-mon[37097]: pgmap v3254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:06:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:58.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:06:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:59 smithi171 ceph-mon[37789]: pgmap v3255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5 KiB/s rd, 9 op/s 2024-07-26T10:06:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:06:59 smithi171 ceph-mon[37789]: pgmap v3256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:59 smithi044 ceph-mon[32404]: pgmap v3255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5 KiB/s rd, 9 op/s 2024-07-26T10:07:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:06:59 smithi044 ceph-mon[32404]: pgmap v3256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:59 smithi116 ceph-mon[37097]: pgmap v3255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5 KiB/s rd, 9 op/s 2024-07-26T10:07:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:06:59 smithi116 ceph-mon[37097]: pgmap v3256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:01 smithi171 ceph-mon[37789]: pgmap v3257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5165.699222 secs 2024-07-26T10:07:01.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:01 smithi044 ceph-mon[32404]: pgmap v3257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:01.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5165.699222 secs 2024-07-26T10:07:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:01 smithi116 ceph-mon[37097]: pgmap v3257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5165.699222 secs 2024-07-26T10:07:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:03 smithi171 ceph-mon[37789]: pgmap v3258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:03 smithi044 ceph-mon[32404]: pgmap v3258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:03 smithi116 ceph-mon[37097]: pgmap v3258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:05 smithi171 ceph-mon[37789]: pgmap v3259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:05 smithi044 ceph-mon[32404]: pgmap v3259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:05 smithi116 ceph-mon[37097]: pgmap v3259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5170.699364 secs 2024-07-26T10:07:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5170.699364 secs 2024-07-26T10:07:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5170.699364 secs 2024-07-26T10:07:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:07 smithi171 ceph-mon[37789]: pgmap v3260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:07.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:07 smithi116 ceph-mon[37097]: pgmap v3260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:07 smithi044 ceph-mon[32404]: pgmap v3260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T10:07:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:09 smithi171 ceph-mon[37789]: pgmap v3261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:07:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:09 smithi044 ceph-mon[32404]: pgmap v3261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:07:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:09 smithi116 ceph-mon[37097]: pgmap v3261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:07:11.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:11.921+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:07:11.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:11.921+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:11.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:11.921+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:07:11.924 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:11 smithi044 ceph-mon[32404]: pgmap v3262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:11.924 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5175.699531 secs 2024-07-26T10:07:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:11 smithi116 ceph-mon[37097]: pgmap v3262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5175.699531 secs 2024-07-26T10:07:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:11 smithi171 ceph-mon[37789]: pgmap v3262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5175.699531 secs 2024-07-26T10:07:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:13 smithi044 ceph-mon[32404]: pgmap v3263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:13 smithi116 ceph-mon[37097]: pgmap v3263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:13 smithi171 ceph-mon[37789]: pgmap v3263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:16.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:15 smithi044 ceph-mon[32404]: pgmap v3264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:15 smithi116 ceph-mon[37097]: pgmap v3264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:15 smithi171 ceph-mon[37789]: pgmap v3264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5180.699667 secs 2024-07-26T10:07:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5180.699667 secs 2024-07-26T10:07:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5180.699667 secs 2024-07-26T10:07:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:17 smithi044 ceph-mon[32404]: pgmap v3265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:17 smithi116 ceph-mon[37097]: pgmap v3265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:17 smithi171 ceph-mon[37789]: pgmap v3265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:19 smithi044 ceph-mon[32404]: pgmap v3266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:07:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:19 smithi116 ceph-mon[37097]: pgmap v3266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:07:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:19 smithi171 ceph-mon[37789]: pgmap v3266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:07:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:21 smithi044 ceph-mon[32404]: pgmap v3267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:07:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5185.699806 secs 2024-07-26T10:07:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:21 smithi116 ceph-mon[37097]: pgmap v3267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:07:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5185.699806 secs 2024-07-26T10:07:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:21 smithi171 ceph-mon[37789]: pgmap v3267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:07:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5185.699806 secs 2024-07-26T10:07:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:23 smithi044 ceph-mon[32404]: pgmap v3268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:07:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:23 smithi116 ceph-mon[37097]: pgmap v3268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:07:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:23 smithi171 ceph-mon[37789]: pgmap v3268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:07:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:25 smithi044 ceph-mon[32404]: pgmap v3269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:07:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:25 smithi116 ceph-mon[37097]: pgmap v3269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:07:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:25 smithi171 ceph-mon[37789]: pgmap v3269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:07:26.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5190.699945 secs 2024-07-26T10:07:26.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:26.930+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:07:26.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:26.930+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:07:26.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:26.930+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5190.699945 secs 2024-07-26T10:07:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5190.699945 secs 2024-07-26T10:07:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:27 smithi044 ceph-mon[32404]: pgmap v3270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:07:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:27 smithi116 ceph-mon[37097]: pgmap v3270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:07:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:27 smithi171 ceph-mon[37789]: pgmap v3270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:07:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:29 smithi044 ceph-mon[32404]: pgmap v3271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:07:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:29 smithi116 ceph-mon[37097]: pgmap v3271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:07:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:29 smithi171 ceph-mon[37789]: pgmap v3271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:07:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:31 smithi044 ceph-mon[32404]: pgmap v3272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5195.700062 secs 2024-07-26T10:07:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:31 smithi116 ceph-mon[37097]: pgmap v3272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5195.700062 secs 2024-07-26T10:07:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:31 smithi171 ceph-mon[37789]: pgmap v3272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5195.700062 secs 2024-07-26T10:07:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:33 smithi044 ceph-mon[32404]: pgmap v3273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:33 smithi116 ceph-mon[37097]: pgmap v3273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:33 smithi171 ceph-mon[37789]: pgmap v3273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:35 smithi044 ceph-mon[32404]: pgmap v3274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:35 smithi116 ceph-mon[37097]: pgmap v3274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:35 smithi171 ceph-mon[37789]: pgmap v3274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5200.700181 secs 2024-07-26T10:07:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5200.700181 secs 2024-07-26T10:07:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5200.700181 secs 2024-07-26T10:07:37.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:37 smithi171 ceph-mon[37789]: pgmap v3275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:37 smithi044 ceph-mon[32404]: pgmap v3275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:37 smithi116 ceph-mon[37097]: pgmap v3275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:39 smithi044 ceph-mon[32404]: pgmap v3276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:39 smithi116 ceph-mon[37097]: pgmap v3276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:39 smithi171 ceph-mon[37789]: pgmap v3276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:41.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:41.934+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:07:41.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:41.934+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:07:41.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:41.934+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:41 smithi044 ceph-mon[32404]: pgmap v3277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5205.700294 secs 2024-07-26T10:07:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:41 smithi116 ceph-mon[37097]: pgmap v3277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5205.700294 secs 2024-07-26T10:07:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:41 smithi171 ceph-mon[37789]: pgmap v3277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5205.700294 secs 2024-07-26T10:07:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:43 smithi044 ceph-mon[32404]: pgmap v3278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:43 smithi116 ceph-mon[37097]: pgmap v3278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:43 smithi171 ceph-mon[37789]: pgmap v3278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:45 smithi044 ceph-mon[32404]: pgmap v3279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:45 smithi116 ceph-mon[37097]: pgmap v3279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:45 smithi171 ceph-mon[37789]: pgmap v3279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5210.700470 secs 2024-07-26T10:07:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5210.700470 secs 2024-07-26T10:07:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5210.700470 secs 2024-07-26T10:07:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:47 smithi044 ceph-mon[32404]: pgmap v3280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:47 smithi116 ceph-mon[37097]: pgmap v3280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:47 smithi171 ceph-mon[37789]: pgmap v3280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:49 smithi044 ceph-mon[32404]: pgmap v3281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:49 smithi116 ceph-mon[37097]: pgmap v3281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:49 smithi171 ceph-mon[37789]: pgmap v3281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:51 smithi044 ceph-mon[32404]: pgmap v3282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5215.700695 secs 2024-07-26T10:07:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:51 smithi116 ceph-mon[37097]: pgmap v3282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5215.700695 secs 2024-07-26T10:07:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:51 smithi171 ceph-mon[37789]: pgmap v3282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5215.700695 secs 2024-07-26T10:07:54.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:53 smithi044 ceph-mon[32404]: pgmap v3283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:53 smithi116 ceph-mon[37097]: pgmap v3283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:53 smithi171 ceph-mon[37789]: pgmap v3283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:55 smithi044 ceph-mon[32404]: pgmap v3284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:55 smithi116 ceph-mon[37097]: pgmap v3284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:55 smithi171 ceph-mon[37789]: pgmap v3284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:07:56.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:56.943+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:07:56.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:56.943+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:56.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:07:56.943+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:07:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5220.700870 secs 2024-07-26T10:07:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5220.700870 secs 2024-07-26T10:07:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5220.700870 secs 2024-07-26T10:07:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:57 smithi116 ceph-mon[37097]: pgmap v3285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:57 smithi171 ceph-mon[37789]: pgmap v3285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:57 smithi044 ceph-mon[32404]: pgmap v3285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:07:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:58 smithi171 ceph-mon[37789]: pgmap v3286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:07:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.175 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:58 smithi116 ceph-mon[37097]: pgmap v3286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:59.176 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:59.176 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.176 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.176 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.176 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:07:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:58 smithi044 ceph-mon[32404]: pgmap v3286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:07:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:07:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:07:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:01 smithi044 ceph-mon[32404]: pgmap v3287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:01 smithi116 ceph-mon[37097]: pgmap v3287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:01 smithi171 ceph-mon[37789]: pgmap v3287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:01.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5225.700976 secs 2024-07-26T10:08:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5225.700976 secs 2024-07-26T10:08:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5225.700976 secs 2024-07-26T10:08:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:02.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:03 smithi044 ceph-mon[32404]: pgmap v3288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:03 smithi116 ceph-mon[37097]: pgmap v3288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:03 smithi171 ceph-mon[37789]: pgmap v3288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:03.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:03.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:03.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:08:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:04 smithi044 ceph-mon[32404]: pgmap v3289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:08:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:04 smithi044 ceph-mon[32404]: pgmap v3290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:04 smithi116 ceph-mon[37097]: pgmap v3289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:08:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:04 smithi116 ceph-mon[37097]: pgmap v3290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:04 smithi171 ceph-mon[37789]: pgmap v3289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:08:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:04 smithi171 ceph-mon[37789]: pgmap v3290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:06 smithi044 ceph-mon[32404]: pgmap v3291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:06 smithi116 ceph-mon[37097]: pgmap v3291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:06 smithi171 ceph-mon[37789]: pgmap v3291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:07.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5230.701157 secs 2024-07-26T10:08:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5230.701157 secs 2024-07-26T10:08:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5230.701157 secs 2024-07-26T10:08:08.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:08 smithi044 ceph-mon[32404]: pgmap v3292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:08 smithi116 ceph-mon[37097]: pgmap v3292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:08 smithi171 ceph-mon[37789]: pgmap v3292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:10 smithi044 ceph-mon[32404]: pgmap v3293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:10 smithi116 ceph-mon[37097]: pgmap v3293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:10 smithi171 ceph-mon[37789]: pgmap v3293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:08:11.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:11.947+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:08:11.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:11.947+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:11.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:11.947+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:08:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5235.701303 secs 2024-07-26T10:08:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:12 smithi044 ceph-mon[32404]: pgmap v3294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5235.701303 secs 2024-07-26T10:08:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:12 smithi116 ceph-mon[37097]: pgmap v3294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5235.701303 secs 2024-07-26T10:08:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:12 smithi171 ceph-mon[37789]: pgmap v3294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:14 smithi044 ceph-mon[32404]: pgmap v3295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:08:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:14 smithi116 ceph-mon[37097]: pgmap v3295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:08:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:14 smithi171 ceph-mon[37789]: pgmap v3295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:08:15.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:15.601+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:08:15.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:15.601+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:08:15.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:15.602+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:08:15.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:15.602+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:08:16.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:16 smithi044 ceph-mon[32404]: pgmap v3296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:16 smithi116 ceph-mon[37097]: pgmap v3296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:16 smithi171 ceph-mon[37789]: pgmap v3296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5240.701482 secs 2024-07-26T10:08:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5240.701482 secs 2024-07-26T10:08:17.677 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5240.701482 secs 2024-07-26T10:08:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:18 smithi171 ceph-mon[37789]: pgmap v3297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:18 smithi044 ceph-mon[32404]: pgmap v3297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:18 smithi116 ceph-mon[37097]: pgmap v3297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:20 smithi171 ceph-mon[37789]: pgmap v3298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:20 smithi044 ceph-mon[32404]: pgmap v3298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:20 smithi116 ceph-mon[37097]: pgmap v3298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5245.701669 secs 2024-07-26T10:08:22.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:22 smithi171 ceph-mon[37789]: pgmap v3299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5245.701669 secs 2024-07-26T10:08:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:22 smithi044 ceph-mon[32404]: pgmap v3299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5245.701669 secs 2024-07-26T10:08:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:22 smithi116 ceph-mon[37097]: pgmap v3299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:24 smithi171 ceph-mon[37789]: pgmap v3300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:24 smithi044 ceph-mon[32404]: pgmap v3300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:24 smithi116 ceph-mon[37097]: pgmap v3300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:26.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:26 smithi044 ceph-mon[32404]: pgmap v3301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:26 smithi171 ceph-mon[37789]: pgmap v3301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:26 smithi116 ceph-mon[37097]: pgmap v3301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:26.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:26.954+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:08:26.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:26.954+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:26.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:26.954+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:08:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5250.701832 secs 2024-07-26T10:08:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5250.701832 secs 2024-07-26T10:08:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5250.701832 secs 2024-07-26T10:08:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:28 smithi171 ceph-mon[37789]: pgmap v3302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:28 smithi044 ceph-mon[32404]: pgmap v3302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:28 smithi116 ceph-mon[37097]: pgmap v3302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:30 smithi171 ceph-mon[37789]: pgmap v3303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:30 smithi044 ceph-mon[32404]: pgmap v3303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:30 smithi116 ceph-mon[37097]: pgmap v3303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5255.702020 secs 2024-07-26T10:08:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5255.702020 secs 2024-07-26T10:08:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5255.702020 secs 2024-07-26T10:08:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:32 smithi171 ceph-mon[37789]: pgmap v3304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:32 smithi044 ceph-mon[32404]: pgmap v3304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:32 smithi116 ceph-mon[37097]: pgmap v3304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:34 smithi044 ceph-mon[32404]: pgmap v3305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:34 smithi116 ceph-mon[37097]: pgmap v3305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:34 smithi171 ceph-mon[37789]: pgmap v3305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:36 smithi044 ceph-mon[32404]: pgmap v3306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5260.702175 secs 2024-07-26T10:08:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:36 smithi116 ceph-mon[37097]: pgmap v3306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5260.702175 secs 2024-07-26T10:08:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:36 smithi171 ceph-mon[37789]: pgmap v3306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:08:36.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5260.702175 secs 2024-07-26T10:08:38.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:38 smithi171 ceph-mon[37789]: pgmap v3307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:38 smithi044 ceph-mon[32404]: pgmap v3307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:38 smithi116 ceph-mon[37097]: pgmap v3307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:40 smithi044 ceph-mon[32404]: pgmap v3308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:40 smithi116 ceph-mon[37097]: pgmap v3308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:40 smithi171 ceph-mon[37789]: pgmap v3308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5265.702317 secs 2024-07-26T10:08:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5265.702317 secs 2024-07-26T10:08:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5265.702317 secs 2024-07-26T10:08:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:41.958+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:08:41.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:41.958+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:41.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:41.958+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:08:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:42 smithi044 ceph-mon[32404]: pgmap v3309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:42 smithi116 ceph-mon[37097]: pgmap v3309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:42 smithi171 ceph-mon[37789]: pgmap v3309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:44.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:44 smithi044 ceph-mon[32404]: pgmap v3310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:44 smithi116 ceph-mon[37097]: pgmap v3310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:44 smithi171 ceph-mon[37789]: pgmap v3310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:08:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:46 smithi044 ceph-mon[32404]: pgmap v3311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5270.702492 secs 2024-07-26T10:08:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:46 smithi116 ceph-mon[37097]: pgmap v3311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5270.702492 secs 2024-07-26T10:08:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:46 smithi171 ceph-mon[37789]: pgmap v3311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5270.702492 secs 2024-07-26T10:08:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:48 smithi044 ceph-mon[32404]: pgmap v3312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:48 smithi116 ceph-mon[37097]: pgmap v3312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:48 smithi171 ceph-mon[37789]: pgmap v3312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:50 smithi044 ceph-mon[32404]: pgmap v3313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:50 smithi116 ceph-mon[37097]: pgmap v3313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:50 smithi171 ceph-mon[37789]: pgmap v3313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:08:51.770 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5275.702668 secs 2024-07-26T10:08:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5275.702668 secs 2024-07-26T10:08:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5275.702668 secs 2024-07-26T10:08:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:52 smithi044 ceph-mon[32404]: pgmap v3314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:52 smithi116 ceph-mon[37097]: pgmap v3314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:52 smithi171 ceph-mon[37789]: pgmap v3314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:08:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:54 smithi044 ceph-mon[32404]: pgmap v3315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:08:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:54 smithi116 ceph-mon[37097]: pgmap v3315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:08:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:54 smithi171 ceph-mon[37789]: pgmap v3315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:08:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:56 smithi044 ceph-mon[32404]: pgmap v3316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:08:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5280.702829 secs 2024-07-26T10:08:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:56 smithi116 ceph-mon[37097]: pgmap v3316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:08:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5280.702829 secs 2024-07-26T10:08:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:56 smithi171 ceph-mon[37789]: pgmap v3316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:08:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5280.702829 secs 2024-07-26T10:08:56.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:56.963+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:08:56.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:56.963+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:56.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:08:56.963+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:08:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:08:58 smithi044 ceph-mon[32404]: pgmap v3317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:08:58 smithi116 ceph-mon[37097]: pgmap v3317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:08:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:08:58 smithi171 ceph-mon[37789]: pgmap v3317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:00 smithi044 ceph-mon[32404]: pgmap v3318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:00 smithi116 ceph-mon[37097]: pgmap v3318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:00 smithi171 ceph-mon[37789]: pgmap v3318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:09:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5285.702941 secs 2024-07-26T10:09:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5285.702941 secs 2024-07-26T10:09:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5285.702941 secs 2024-07-26T10:09:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:02 smithi044 ceph-mon[32404]: pgmap v3319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:09:02.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:02 smithi116 ceph-mon[37097]: pgmap v3319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:09:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:02 smithi171 ceph-mon[37789]: pgmap v3319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:09:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:04 smithi044 ceph-mon[32404]: pgmap v3320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:09:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:04.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:09:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:04 smithi116 ceph-mon[37097]: pgmap v3320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:09:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:09:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:04 smithi171 ceph-mon[37789]: pgmap v3320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:09:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:09:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:05 smithi044 ceph-mon[32404]: pgmap v3321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:09:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:05 smithi044 ceph-mon[32404]: pgmap v3322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:05 smithi116 ceph-mon[37097]: pgmap v3321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:09:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:05 smithi116 ceph-mon[37097]: pgmap v3322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:05.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:05 smithi171 ceph-mon[37789]: pgmap v3321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:09:05.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:05 smithi171 ceph-mon[37789]: pgmap v3322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5290.703095 secs 2024-07-26T10:09:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5290.703095 secs 2024-07-26T10:09:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5290.703095 secs 2024-07-26T10:09:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:07 smithi044 ceph-mon[32404]: pgmap v3323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:07 smithi116 ceph-mon[37097]: pgmap v3323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:07 smithi171 ceph-mon[37789]: pgmap v3323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:09 smithi171 ceph-mon[37789]: pgmap v3324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:09 smithi044 ceph-mon[32404]: pgmap v3324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:09 smithi116 ceph-mon[37097]: pgmap v3324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:11 smithi171 ceph-mon[37789]: pgmap v3325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:11.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5295.703236 secs 2024-07-26T10:09:11.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:11.967+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:09:11.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:11.967+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:11.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:11.967+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:09:11.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:11 smithi044 ceph-mon[32404]: pgmap v3325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:11.971 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5295.703236 secs 2024-07-26T10:09:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:11 smithi116 ceph-mon[37097]: pgmap v3325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:09:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5295.703236 secs 2024-07-26T10:09:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:13 smithi171 ceph-mon[37789]: pgmap v3326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:13 smithi044 ceph-mon[32404]: pgmap v3326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:13 smithi116 ceph-mon[37097]: pgmap v3326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:09:15.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:15 smithi171 ceph-mon[37789]: pgmap v3327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:09:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:15 smithi044 ceph-mon[32404]: pgmap v3327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:09:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:15 smithi116 ceph-mon[37097]: pgmap v3327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:09:16.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5300.703415 secs 2024-07-26T10:09:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5300.703415 secs 2024-07-26T10:09:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5300.703415 secs 2024-07-26T10:09:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:17 smithi044 ceph-mon[32404]: pgmap v3328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:17 smithi116 ceph-mon[37097]: pgmap v3328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:17 smithi171 ceph-mon[37789]: pgmap v3328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:19 smithi044 ceph-mon[32404]: pgmap v3329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:19 smithi116 ceph-mon[37097]: pgmap v3329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:19 smithi171 ceph-mon[37789]: pgmap v3329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:22.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:21 smithi044 ceph-mon[32404]: pgmap v3330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:22.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5305.703591 secs 2024-07-26T10:09:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:21 smithi116 ceph-mon[37097]: pgmap v3330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5305.703591 secs 2024-07-26T10:09:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:21 smithi171 ceph-mon[37789]: pgmap v3330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5305.703591 secs 2024-07-26T10:09:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:23 smithi044 ceph-mon[32404]: pgmap v3331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:23 smithi116 ceph-mon[37097]: pgmap v3331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:23 smithi171 ceph-mon[37789]: pgmap v3331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:25 smithi044 ceph-mon[32404]: pgmap v3332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:25 smithi116 ceph-mon[37097]: pgmap v3332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:25 smithi171 ceph-mon[37789]: pgmap v3332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:26.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5310.703743 secs 2024-07-26T10:09:26.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:26.972+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:09:26.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:26.973+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:09:26.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:26.973+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5310.703743 secs 2024-07-26T10:09:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5310.703743 secs 2024-07-26T10:09:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:27 smithi044 ceph-mon[32404]: pgmap v3333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:27 smithi116 ceph-mon[37097]: pgmap v3333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:27 smithi171 ceph-mon[37789]: pgmap v3333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:29 smithi044 ceph-mon[32404]: pgmap v3334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:29 smithi116 ceph-mon[37097]: pgmap v3334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:29 smithi171 ceph-mon[37789]: pgmap v3334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:31 smithi044 ceph-mon[32404]: pgmap v3335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5315.703845 secs 2024-07-26T10:09:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:31 smithi116 ceph-mon[37097]: pgmap v3335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5315.703845 secs 2024-07-26T10:09:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:31 smithi171 ceph-mon[37789]: pgmap v3335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5315.703845 secs 2024-07-26T10:09:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:33 smithi044 ceph-mon[32404]: pgmap v3336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:33 smithi116 ceph-mon[37097]: pgmap v3336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:33 smithi171 ceph-mon[37789]: pgmap v3336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:35 smithi044 ceph-mon[32404]: pgmap v3337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:35 smithi116 ceph-mon[37097]: pgmap v3337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:35 smithi171 ceph-mon[37789]: pgmap v3337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5320.704017 secs 2024-07-26T10:09:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5320.704017 secs 2024-07-26T10:09:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5320.704017 secs 2024-07-26T10:09:38.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:37 smithi044 ceph-mon[32404]: pgmap v3338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:37 smithi116 ceph-mon[37097]: pgmap v3338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:37 smithi171 ceph-mon[37789]: pgmap v3338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:39 smithi044 ceph-mon[32404]: pgmap v3339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:39 smithi116 ceph-mon[37097]: pgmap v3339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:39 smithi171 ceph-mon[37789]: pgmap v3339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:41.977+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:09:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:41.977+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:41.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:41.977+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:09:41.981 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:41 smithi044 ceph-mon[32404]: pgmap v3340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:41.981 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5325.704156 secs 2024-07-26T10:09:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:41 smithi116 ceph-mon[37097]: pgmap v3340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5325.704156 secs 2024-07-26T10:09:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:41 smithi171 ceph-mon[37789]: pgmap v3340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5325.704156 secs 2024-07-26T10:09:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:43 smithi044 ceph-mon[32404]: pgmap v3341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:43 smithi116 ceph-mon[37097]: pgmap v3341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:43 smithi171 ceph-mon[37789]: pgmap v3341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:45 smithi044 ceph-mon[32404]: pgmap v3342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:45 smithi116 ceph-mon[37097]: pgmap v3342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:45 smithi171 ceph-mon[37789]: pgmap v3342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5330.704257 secs 2024-07-26T10:09:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5330.704257 secs 2024-07-26T10:09:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5330.704257 secs 2024-07-26T10:09:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:47 smithi044 ceph-mon[32404]: pgmap v3343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:47 smithi116 ceph-mon[37097]: pgmap v3343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:47 smithi171 ceph-mon[37789]: pgmap v3343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:49 smithi044 ceph-mon[32404]: pgmap v3344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:49 smithi116 ceph-mon[37097]: pgmap v3344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:49 smithi171 ceph-mon[37789]: pgmap v3344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:09:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:51 smithi044 ceph-mon[32404]: pgmap v3345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5335.704423 secs 2024-07-26T10:09:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:51 smithi116 ceph-mon[37097]: pgmap v3345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5335.704423 secs 2024-07-26T10:09:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:51 smithi171 ceph-mon[37789]: pgmap v3345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5335.704423 secs 2024-07-26T10:09:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:53 smithi044 ceph-mon[32404]: pgmap v3346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:53 smithi116 ceph-mon[37097]: pgmap v3346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:53 smithi171 ceph-mon[37789]: pgmap v3346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:09:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:55 smithi044 ceph-mon[32404]: pgmap v3347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:55 smithi116 ceph-mon[37097]: pgmap v3347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:55 smithi171 ceph-mon[37789]: pgmap v3347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:09:56.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:56.986+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:09:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:56.986+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:09:56.986+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:09:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5340.704577 secs 2024-07-26T10:09:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5340.704577 secs 2024-07-26T10:09:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5340.704577 secs 2024-07-26T10:09:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:57 smithi044 ceph-mon[32404]: pgmap v3348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:57 smithi116 ceph-mon[37097]: pgmap v3348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:09:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:57 smithi171 ceph-mon[37789]: pgmap v3348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:09:59 smithi044 ceph-mon[32404]: pgmap v3349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:10:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:09:59 smithi116 ceph-mon[37097]: pgmap v3349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:10:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:09:59 smithi171 ceph-mon[37789]: pgmap v3349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:10:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:10:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:10:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:10:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:01 smithi116 ceph-mon[37097]: pgmap v3350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5345.704742 secs 2024-07-26T10:10:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:01 smithi171 ceph-mon[37789]: pgmap v3350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5345.704742 secs 2024-07-26T10:10:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:01 smithi044 ceph-mon[32404]: pgmap v3350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5345.704742 secs 2024-07-26T10:10:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:03 smithi171 ceph-mon[37789]: pgmap v3351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:03 smithi044 ceph-mon[32404]: pgmap v3351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:03 smithi116 ceph-mon[37097]: pgmap v3351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:05 smithi171 ceph-mon[37789]: pgmap v3352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:10:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:05 smithi044 ceph-mon[32404]: pgmap v3352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:10:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:05 smithi116 ceph-mon[37097]: pgmap v3352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:10:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5350.704835 secs 2024-07-26T10:10:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:06 smithi171 ceph-mon[37789]: pgmap v3353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:10:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:06 smithi171 ceph-mon[37789]: pgmap v3354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5350.704835 secs 2024-07-26T10:10:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:06 smithi044 ceph-mon[32404]: pgmap v3353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:10:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:06 smithi044 ceph-mon[32404]: pgmap v3354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5350.704835 secs 2024-07-26T10:10:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:06 smithi116 ceph-mon[37097]: pgmap v3353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:10:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:06 smithi116 ceph-mon[37097]: pgmap v3354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:07 smithi171 ceph-mon[37789]: pgmap v3355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:10:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:07 smithi044 ceph-mon[32404]: pgmap v3355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:10:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:07 smithi116 ceph-mon[37097]: pgmap v3355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:10:10.976 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:10 smithi116 ceph-mon[37097]: pgmap v3356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:10 smithi044 ceph-mon[32404]: pgmap v3356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:10 smithi171 ceph-mon[37789]: pgmap v3356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:11.988+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:10:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:11.988+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:10:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:11.988+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:11.994 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5355.705005 secs 2024-07-26T10:10:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5355.705005 secs 2024-07-26T10:10:12.073 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5355.705005 secs 2024-07-26T10:10:13.003 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:12 smithi171 ceph-mon[37789]: pgmap v3357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:10:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:12 smithi044 ceph-mon[32404]: pgmap v3357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:10:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:12 smithi116 ceph-mon[37097]: pgmap v3357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:10:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:14 smithi044 ceph-mon[32404]: pgmap v3358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:14 smithi116 ceph-mon[37097]: pgmap v3358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:14 smithi171 ceph-mon[37789]: pgmap v3358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:10:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5360.705064 secs 2024-07-26T10:10:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:16 smithi044 ceph-mon[32404]: pgmap v3359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:10:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5360.705064 secs 2024-07-26T10:10:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:16 smithi116 ceph-mon[37097]: pgmap v3359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:10:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5360.705064 secs 2024-07-26T10:10:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:16 smithi171 ceph-mon[37789]: pgmap v3359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:10:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:18 smithi044 ceph-mon[32404]: pgmap v3360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:18 smithi116 ceph-mon[37097]: pgmap v3360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:18 smithi171 ceph-mon[37789]: pgmap v3360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:20 smithi044 ceph-mon[32404]: pgmap v3361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:20 smithi116 ceph-mon[37097]: pgmap v3361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:20 smithi171 ceph-mon[37789]: pgmap v3361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5365.705207 secs 2024-07-26T10:10:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5365.705207 secs 2024-07-26T10:10:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5365.705207 secs 2024-07-26T10:10:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:22 smithi171 ceph-mon[37789]: pgmap v3362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:22 smithi044 ceph-mon[32404]: pgmap v3362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:22 smithi116 ceph-mon[37097]: pgmap v3362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:24 smithi044 ceph-mon[32404]: pgmap v3363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:24 smithi116 ceph-mon[37097]: pgmap v3363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:24 smithi171 ceph-mon[37789]: pgmap v3363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:26.997+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:10:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:26.998+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:27.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:26.998+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:10:27.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5370.705349 secs 2024-07-26T10:10:27.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:26 smithi044 ceph-mon[32404]: pgmap v3364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5370.705349 secs 2024-07-26T10:10:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:26 smithi116 ceph-mon[37097]: pgmap v3364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5370.705349 secs 2024-07-26T10:10:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:26 smithi171 ceph-mon[37789]: pgmap v3364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:28 smithi044 ceph-mon[32404]: pgmap v3365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:28 smithi116 ceph-mon[37097]: pgmap v3365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:28 smithi171 ceph-mon[37789]: pgmap v3365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:30 smithi044 ceph-mon[32404]: pgmap v3366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:30 smithi116 ceph-mon[37097]: pgmap v3366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:30 smithi171 ceph-mon[37789]: pgmap v3366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5375.705509 secs 2024-07-26T10:10:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5375.705509 secs 2024-07-26T10:10:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5375.705509 secs 2024-07-26T10:10:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:32 smithi044 ceph-mon[32404]: pgmap v3367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:32 smithi116 ceph-mon[37097]: pgmap v3367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:32 smithi171 ceph-mon[37789]: pgmap v3367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:34 smithi044 ceph-mon[32404]: pgmap v3368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:34 smithi116 ceph-mon[37097]: pgmap v3368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:35.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:34 smithi171 ceph-mon[37789]: pgmap v3368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5380.705608 secs 2024-07-26T10:10:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:36 smithi044 ceph-mon[32404]: pgmap v3369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5380.705608 secs 2024-07-26T10:10:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:36 smithi116 ceph-mon[37097]: pgmap v3369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5380.705608 secs 2024-07-26T10:10:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:36 smithi171 ceph-mon[37789]: pgmap v3369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:38 smithi044 ceph-mon[32404]: pgmap v3370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:38 smithi116 ceph-mon[37097]: pgmap v3370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:38 smithi171 ceph-mon[37789]: pgmap v3370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:40 smithi044 ceph-mon[32404]: pgmap v3371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:40 smithi116 ceph-mon[37097]: pgmap v3371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:40 smithi171 ceph-mon[37789]: pgmap v3371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:42.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:42.002+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:10:42.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:42.002+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:42.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:42.002+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:10:42.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5385.705768 secs 2024-07-26T10:10:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5385.705768 secs 2024-07-26T10:10:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5385.705768 secs 2024-07-26T10:10:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:42 smithi044 ceph-mon[32404]: pgmap v3372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:42 smithi116 ceph-mon[37097]: pgmap v3372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:43.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:42 smithi171 ceph-mon[37789]: pgmap v3372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:44 smithi044 ceph-mon[32404]: pgmap v3373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:44 smithi116 ceph-mon[37097]: pgmap v3373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:44 smithi171 ceph-mon[37789]: pgmap v3373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5390.705927 secs 2024-07-26T10:10:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:46 smithi044 ceph-mon[32404]: pgmap v3374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5390.705927 secs 2024-07-26T10:10:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:46 smithi116 ceph-mon[37097]: pgmap v3374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5390.705927 secs 2024-07-26T10:10:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:46 smithi171 ceph-mon[37789]: pgmap v3374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:48 smithi044 ceph-mon[32404]: pgmap v3375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:48 smithi116 ceph-mon[37097]: pgmap v3375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:48 smithi171 ceph-mon[37789]: pgmap v3375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:50 smithi044 ceph-mon[32404]: pgmap v3376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:50 smithi116 ceph-mon[37097]: pgmap v3376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:50 smithi171 ceph-mon[37789]: pgmap v3376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:10:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5395.706064 secs 2024-07-26T10:10:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5395.706064 secs 2024-07-26T10:10:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5395.706064 secs 2024-07-26T10:10:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:52 smithi116 ceph-mon[37097]: pgmap v3377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:52 smithi171 ceph-mon[37789]: pgmap v3377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:52 smithi044 ceph-mon[32404]: pgmap v3377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:54 smithi171 ceph-mon[37789]: pgmap v3378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:54 smithi044 ceph-mon[32404]: pgmap v3378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:54 smithi116 ceph-mon[37097]: pgmap v3378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:10:57.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:57.011+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:10:57.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:57.012+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:10:57.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:10:57.012+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:57.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5400.706227 secs 2024-07-26T10:10:57.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:56 smithi044 ceph-mon[32404]: pgmap v3379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5400.706227 secs 2024-07-26T10:10:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:56 smithi171 ceph-mon[37789]: pgmap v3379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5400.706227 secs 2024-07-26T10:10:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:56 smithi116 ceph-mon[37097]: pgmap v3379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:10:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:10:58 smithi171 ceph-mon[37789]: pgmap v3380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:10:58 smithi044 ceph-mon[32404]: pgmap v3380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:10:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:10:58 smithi116 ceph-mon[37097]: pgmap v3380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:00 smithi171 ceph-mon[37789]: pgmap v3381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:00 smithi044 ceph-mon[32404]: pgmap v3381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:00 smithi116 ceph-mon[37097]: pgmap v3381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5405.706381 secs 2024-07-26T10:11:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5405.706381 secs 2024-07-26T10:11:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5405.706381 secs 2024-07-26T10:11:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:02 smithi171 ceph-mon[37789]: pgmap v3382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:11:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:02 smithi044 ceph-mon[32404]: pgmap v3382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:11:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:02 smithi116 ceph-mon[37097]: pgmap v3382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:11:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:03 smithi171 ceph-mon[37789]: pgmap v3383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:11:04.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:03 smithi044 ceph-mon[32404]: pgmap v3383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:11:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:03 smithi116 ceph-mon[37097]: pgmap v3383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:11:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5410.706526 secs 2024-07-26T10:11:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:06 smithi044 ceph-mon[32404]: pgmap v3384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5410.706526 secs 2024-07-26T10:11:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:06 smithi116 ceph-mon[37097]: pgmap v3384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5410.706526 secs 2024-07-26T10:11:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:06 smithi171 ceph-mon[37789]: pgmap v3384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:08.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:11:08.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:11:08.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:11:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:11:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:11:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:11:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:11:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:11:08.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:11:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:08 smithi044 ceph-mon[32404]: pgmap v3385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:11:09.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:08 smithi044 ceph-mon[32404]: pgmap v3386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:08 smithi116 ceph-mon[37097]: pgmap v3385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:11:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:08 smithi116 ceph-mon[37097]: pgmap v3386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:08 smithi171 ceph-mon[37789]: pgmap v3385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:11:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:08 smithi171 ceph-mon[37789]: pgmap v3386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:10 smithi044 ceph-mon[32404]: pgmap v3387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:10 smithi116 ceph-mon[37097]: pgmap v3387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:10 smithi171 ceph-mon[37789]: pgmap v3387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5415.706669 secs 2024-07-26T10:11:12.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:12.014+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:11:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:12.014+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:12.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:12.014+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:11:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5415.706669 secs 2024-07-26T10:11:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5415.706669 secs 2024-07-26T10:11:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:12 smithi044 ceph-mon[32404]: pgmap v3388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:11:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:12 smithi116 ceph-mon[37097]: pgmap v3388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:11:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:12 smithi171 ceph-mon[37789]: pgmap v3388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 0x7fcf540c18d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf5805b000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 0x7fcf540c18d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf5805b000 comp rx=0 tx=0).stop 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5fa0 0x7fcf540c18d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.698+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 0x7fcf540b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:11:14.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.699+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 0x7fcf540b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:11:14.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:14.702+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 0x7fcf540b5300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcf5805c000 tx=0x7fcf58064000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:11:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:14 smithi044 ceph-mon[32404]: pgmap v3389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:11:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:14 smithi116 ceph-mon[37097]: pgmap v3389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:11:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:14 smithi171 ceph-mon[37789]: pgmap v3389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:11:15.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:15.632+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:11:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:15.632+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:11:15.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:15.633+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:11:15.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:15.634+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:11:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:16 smithi044 ceph-mon[32404]: pgmap v3390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5420.706746 secs 2024-07-26T10:11:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:16 smithi116 ceph-mon[37097]: pgmap v3390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5420.706746 secs 2024-07-26T10:11:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:16 smithi171 ceph-mon[37789]: pgmap v3390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:11:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5420.706746 secs 2024-07-26T10:11:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:18 smithi044 ceph-mon[32404]: pgmap v3391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:11:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:18 smithi116 ceph-mon[37097]: pgmap v3391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:11:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:18 smithi171 ceph-mon[37789]: pgmap v3391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:11:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:20 smithi044 ceph-mon[32404]: pgmap v3392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:20 smithi116 ceph-mon[37097]: pgmap v3392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:20 smithi171 ceph-mon[37789]: pgmap v3392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5425.706905 secs 2024-07-26T10:11:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5425.706905 secs 2024-07-26T10:11:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5425.706905 secs 2024-07-26T10:11:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:22 smithi044 ceph-mon[32404]: pgmap v3393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:22 smithi116 ceph-mon[37097]: pgmap v3393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:22 smithi171 ceph-mon[37789]: pgmap v3393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:24 smithi044 ceph-mon[32404]: pgmap v3394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:24 smithi116 ceph-mon[37097]: pgmap v3394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:24 smithi171 ceph-mon[37789]: pgmap v3394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:26 smithi044 ceph-mon[32404]: pgmap v3395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5430.707037 secs 2024-07-26T10:11:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:26 smithi116 ceph-mon[37097]: pgmap v3395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5430.707037 secs 2024-07-26T10:11:27.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:27.021+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:11:27.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:27.021+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:27.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:27.021+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:11:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:26 smithi171 ceph-mon[37789]: pgmap v3395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5430.707037 secs 2024-07-26T10:11:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:28 smithi044 ceph-mon[32404]: pgmap v3396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:28 smithi116 ceph-mon[37097]: pgmap v3396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:28 smithi171 ceph-mon[37789]: pgmap v3396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:30 smithi044 ceph-mon[32404]: pgmap v3397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:30 smithi116 ceph-mon[37097]: pgmap v3397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:30 smithi171 ceph-mon[37789]: pgmap v3397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5435.707198 secs 2024-07-26T10:11:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5435.707198 secs 2024-07-26T10:11:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5435.707198 secs 2024-07-26T10:11:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:32 smithi044 ceph-mon[32404]: pgmap v3398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:32 smithi116 ceph-mon[37097]: pgmap v3398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:32 smithi171 ceph-mon[37789]: pgmap v3398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:34 smithi116 ceph-mon[37097]: pgmap v3399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:34 smithi171 ceph-mon[37789]: pgmap v3399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:34 smithi044 ceph-mon[32404]: pgmap v3399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:37.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:36 smithi171 ceph-mon[37789]: pgmap v3400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:37.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5440.707364 secs 2024-07-26T10:11:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:36 smithi044 ceph-mon[32404]: pgmap v3400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5440.707364 secs 2024-07-26T10:11:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:36 smithi116 ceph-mon[37097]: pgmap v3400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5440.707364 secs 2024-07-26T10:11:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:38 smithi171 ceph-mon[37789]: pgmap v3401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:38 smithi044 ceph-mon[32404]: pgmap v3401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:38 smithi116 ceph-mon[37097]: pgmap v3401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:41.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:40 smithi116 ceph-mon[37097]: pgmap v3402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:40 smithi171 ceph-mon[37789]: pgmap v3402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:40 smithi044 ceph-mon[32404]: pgmap v3402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:42.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:42.025+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:11:42.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:42.025+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:42.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:42.025+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:11:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5445.707526 secs 2024-07-26T10:11:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5445.707526 secs 2024-07-26T10:11:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5445.707526 secs 2024-07-26T10:11:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:42 smithi171 ceph-mon[37789]: pgmap v3403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:42 smithi044 ceph-mon[32404]: pgmap v3403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:42 smithi116 ceph-mon[37097]: pgmap v3403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:43 smithi171 ceph-mon[37789]: pgmap v3404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:43 smithi044 ceph-mon[32404]: pgmap v3404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:43 smithi116 ceph-mon[37097]: pgmap v3404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:45 smithi044 ceph-mon[32404]: pgmap v3405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:45 smithi116 ceph-mon[37097]: pgmap v3405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:45 smithi171 ceph-mon[37789]: pgmap v3405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5450.707689 secs 2024-07-26T10:11:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5450.707689 secs 2024-07-26T10:11:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5450.707689 secs 2024-07-26T10:11:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:47 smithi044 ceph-mon[32404]: pgmap v3406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:48.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:47 smithi116 ceph-mon[37097]: pgmap v3406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:47 smithi171 ceph-mon[37789]: pgmap v3406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:49 smithi044 ceph-mon[32404]: pgmap v3407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:49 smithi116 ceph-mon[37097]: pgmap v3407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:49 smithi171 ceph-mon[37789]: pgmap v3407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5455.707825 secs 2024-07-26T10:11:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:51 smithi044 ceph-mon[32404]: pgmap v3408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5455.707825 secs 2024-07-26T10:11:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:51 smithi116 ceph-mon[37097]: pgmap v3408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5455.707825 secs 2024-07-26T10:11:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:51 smithi171 ceph-mon[37789]: pgmap v3408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:11:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:53 smithi044 ceph-mon[32404]: pgmap v3409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:53 smithi116 ceph-mon[37097]: pgmap v3409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:53 smithi171 ceph-mon[37789]: pgmap v3409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:55 smithi044 ceph-mon[32404]: pgmap v3410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:55 smithi116 ceph-mon[37097]: pgmap v3410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:55 smithi171 ceph-mon[37789]: pgmap v3410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:11:57.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:57.028+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:11:57.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:57.028+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:57.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:11:57.028+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:11:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5460.707961 secs 2024-07-26T10:11:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5460.707961 secs 2024-07-26T10:11:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5460.707961 secs 2024-07-26T10:11:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:57 smithi044 ceph-mon[32404]: pgmap v3411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:57 smithi116 ceph-mon[37097]: pgmap v3411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:11:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:57 smithi171 ceph-mon[37789]: pgmap v3411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:11:59 smithi044 ceph-mon[32404]: pgmap v3412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:11:59 smithi116 ceph-mon[37097]: pgmap v3412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:11:59 smithi171 ceph-mon[37789]: pgmap v3412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:00.641+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:12:00.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:00.641+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:12:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:00.642+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003740 con 0x7fcf6c1057b0 2024-07-26T10:12:00.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:00.642+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:12:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5465.708083 secs 2024-07-26T10:12:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:01 smithi044 ceph-mon[32404]: pgmap v3413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5465.708083 secs 2024-07-26T10:12:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:01 smithi116 ceph-mon[37097]: pgmap v3413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5465.708083 secs 2024-07-26T10:12:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:01 smithi171 ceph-mon[37789]: pgmap v3413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:03 smithi044 ceph-mon[32404]: pgmap v3414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:03 smithi116 ceph-mon[37097]: pgmap v3414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:03 smithi171 ceph-mon[37789]: pgmap v3414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:05 smithi044 ceph-mon[32404]: pgmap v3415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:05 smithi116 ceph-mon[37097]: pgmap v3415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:05 smithi171 ceph-mon[37789]: pgmap v3415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5470.708226 secs 2024-07-26T10:12:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5470.708226 secs 2024-07-26T10:12:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5470.708226 secs 2024-07-26T10:12:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:08 smithi044 ceph-mon[32404]: pgmap v3416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:08 smithi116 ceph-mon[37097]: pgmap v3416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:08 smithi171 ceph-mon[37789]: pgmap v3416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:09 smithi116 ceph-mon[37097]: pgmap v3417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:12:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:09 smithi116 ceph-mon[37097]: pgmap v3418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:12:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:09 smithi171 ceph-mon[37789]: pgmap v3417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:12:09.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:09 smithi171 ceph-mon[37789]: pgmap v3418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:09.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:12:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:09 smithi044 ceph-mon[32404]: pgmap v3417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:12:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:09 smithi044 ceph-mon[32404]: pgmap v3418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:09.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:12:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:11 smithi044 ceph-mon[32404]: pgmap v3419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:11 smithi116 ceph-mon[37097]: pgmap v3419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:11 smithi171 ceph-mon[37789]: pgmap v3419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:12.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:12.029+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:12:12.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:12.029+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:12.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:12.029+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:12:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5475.708384 secs 2024-07-26T10:12:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5475.708384 secs 2024-07-26T10:12:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5475.708384 secs 2024-07-26T10:12:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:13 smithi044 ceph-mon[32404]: pgmap v3420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:13 smithi116 ceph-mon[37097]: pgmap v3420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:13 smithi171 ceph-mon[37789]: pgmap v3420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:15 smithi044 ceph-mon[32404]: pgmap v3421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:15 smithi116 ceph-mon[37097]: pgmap v3421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:15 smithi171 ceph-mon[37789]: pgmap v3421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:12:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5480.708477 secs 2024-07-26T10:12:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:17 smithi044 ceph-mon[32404]: pgmap v3422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:12:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5480.708477 secs 2024-07-26T10:12:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:17 smithi116 ceph-mon[37097]: pgmap v3422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:12:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5480.708477 secs 2024-07-26T10:12:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:17 smithi171 ceph-mon[37789]: pgmap v3422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:12:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:19 smithi044 ceph-mon[32404]: pgmap v3423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:12:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:19 smithi116 ceph-mon[37097]: pgmap v3423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:12:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:19 smithi171 ceph-mon[37789]: pgmap v3423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:12:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:21 smithi044 ceph-mon[32404]: pgmap v3424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:21 smithi116 ceph-mon[37097]: pgmap v3424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:21 smithi171 ceph-mon[37789]: pgmap v3424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5485.708688 secs 2024-07-26T10:12:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5485.708688 secs 2024-07-26T10:12:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5485.708688 secs 2024-07-26T10:12:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:23 smithi044 ceph-mon[32404]: pgmap v3425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:23 smithi116 ceph-mon[37097]: pgmap v3425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:23 smithi171 ceph-mon[37789]: pgmap v3425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:25 smithi044 ceph-mon[32404]: pgmap v3426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:25 smithi116 ceph-mon[37097]: pgmap v3426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:25 smithi171 ceph-mon[37789]: pgmap v3426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5490.708847 secs 2024-07-26T10:12:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5490.708847 secs 2024-07-26T10:12:27.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:27.033+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:12:27.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:27.033+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:27.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:27.033+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:12:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5490.708847 secs 2024-07-26T10:12:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:27 smithi044 ceph-mon[32404]: pgmap v3427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:27 smithi116 ceph-mon[37097]: pgmap v3427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:27 smithi171 ceph-mon[37789]: pgmap v3427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:29 smithi044 ceph-mon[32404]: pgmap v3428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:29 smithi116 ceph-mon[37097]: pgmap v3428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:29 smithi171 ceph-mon[37789]: pgmap v3428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:31 smithi044 ceph-mon[32404]: pgmap v3429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5495.709016 secs 2024-07-26T10:12:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:31 smithi116 ceph-mon[37097]: pgmap v3429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5495.709016 secs 2024-07-26T10:12:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:31 smithi171 ceph-mon[37789]: pgmap v3429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5495.709016 secs 2024-07-26T10:12:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:33 smithi044 ceph-mon[32404]: pgmap v3430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:33 smithi116 ceph-mon[37097]: pgmap v3430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:33 smithi171 ceph-mon[37789]: pgmap v3430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:35 smithi044 ceph-mon[32404]: pgmap v3431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:35 smithi116 ceph-mon[37097]: pgmap v3431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:35 smithi171 ceph-mon[37789]: pgmap v3431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5500.709159 secs 2024-07-26T10:12:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5500.709159 secs 2024-07-26T10:12:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5500.709159 secs 2024-07-26T10:12:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:37 smithi044 ceph-mon[32404]: pgmap v3432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:37 smithi116 ceph-mon[37097]: pgmap v3432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:37 smithi171 ceph-mon[37789]: pgmap v3432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:39 smithi044 ceph-mon[32404]: pgmap v3433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:39 smithi116 ceph-mon[37097]: pgmap v3433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:39 smithi171 ceph-mon[37789]: pgmap v3433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:41 smithi044 ceph-mon[32404]: pgmap v3434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5505.709326 secs 2024-07-26T10:12:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:41 smithi116 ceph-mon[37097]: pgmap v3434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5505.709326 secs 2024-07-26T10:12:42.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:42.037+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:12:42.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:42.037+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:12:42.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:42.037+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:41 smithi171 ceph-mon[37789]: pgmap v3434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5505.709326 secs 2024-07-26T10:12:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:43 smithi044 ceph-mon[32404]: pgmap v3435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:43 smithi116 ceph-mon[37097]: pgmap v3435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:43 smithi171 ceph-mon[37789]: pgmap v3435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:45 smithi044 ceph-mon[32404]: pgmap v3436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:45 smithi116 ceph-mon[37097]: pgmap v3436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:45 smithi171 ceph-mon[37789]: pgmap v3436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5510.709496 secs 2024-07-26T10:12:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5510.709496 secs 2024-07-26T10:12:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5510.709496 secs 2024-07-26T10:12:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:47 smithi044 ceph-mon[32404]: pgmap v3437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:47 smithi116 ceph-mon[37097]: pgmap v3437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:47 smithi171 ceph-mon[37789]: pgmap v3437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:49 smithi044 ceph-mon[32404]: pgmap v3438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:49 smithi116 ceph-mon[37097]: pgmap v3438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:49 smithi171 ceph-mon[37789]: pgmap v3438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:12:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:51 smithi044 ceph-mon[32404]: pgmap v3439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5515.709602 secs 2024-07-26T10:12:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:51 smithi116 ceph-mon[37097]: pgmap v3439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5515.709602 secs 2024-07-26T10:12:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:51 smithi171 ceph-mon[37789]: pgmap v3439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5515.709602 secs 2024-07-26T10:12:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:53 smithi044 ceph-mon[32404]: pgmap v3440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:53 smithi116 ceph-mon[37097]: pgmap v3440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:53 smithi171 ceph-mon[37789]: pgmap v3440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:12:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:55 smithi116 ceph-mon[37097]: pgmap v3441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5520.709761 secs 2024-07-26T10:12:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:55 smithi171 ceph-mon[37789]: pgmap v3441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5520.709761 secs 2024-07-26T10:12:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:55 smithi044 ceph-mon[32404]: pgmap v3441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:12:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5520.709761 secs 2024-07-26T10:12:57.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:57.046+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:12:57.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:57.046+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:57.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:12:57.046+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:12:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:57 smithi171 ceph-mon[37789]: pgmap v3442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:57 smithi044 ceph-mon[32404]: pgmap v3442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:12:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:57 smithi116 ceph-mon[37097]: pgmap v3442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:12:59 smithi171 ceph-mon[37789]: pgmap v3443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:12:59 smithi044 ceph-mon[32404]: pgmap v3443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:12:59 smithi116 ceph-mon[37097]: pgmap v3443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5525.709858 secs 2024-07-26T10:13:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5525.709858 secs 2024-07-26T10:13:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5525.709858 secs 2024-07-26T10:13:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:01 smithi171 ceph-mon[37789]: pgmap v3444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:01 smithi044 ceph-mon[32404]: pgmap v3444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:01 smithi116 ceph-mon[37097]: pgmap v3444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:03 smithi171 ceph-mon[37789]: pgmap v3445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:03 smithi044 ceph-mon[32404]: pgmap v3445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:03 smithi116 ceph-mon[37097]: pgmap v3445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:05 smithi171 ceph-mon[37789]: pgmap v3446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5530.709958 secs 2024-07-26T10:13:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:05 smithi044 ceph-mon[32404]: pgmap v3446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5530.709958 secs 2024-07-26T10:13:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:05 smithi116 ceph-mon[37097]: pgmap v3446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5530.709958 secs 2024-07-26T10:13:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:07 smithi171 ceph-mon[37789]: pgmap v3447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:07 smithi044 ceph-mon[32404]: pgmap v3447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:07 smithi116 ceph-mon[37097]: pgmap v3447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:09 smithi171 ceph-mon[37789]: pgmap v3448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:10.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:10.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:13:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:09 smithi044 ceph-mon[32404]: pgmap v3448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:13:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:09 smithi116 ceph-mon[37097]: pgmap v3448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:13:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:10 smithi044 ceph-mon[32404]: pgmap v3449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:13:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:10 smithi044 ceph-mon[32404]: pgmap v3450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5535.710126 secs 2024-07-26T10:13:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:10 smithi171 ceph-mon[37789]: pgmap v3449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:10 smithi171 ceph-mon[37789]: pgmap v3450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5535.710126 secs 2024-07-26T10:13:11.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:10 smithi116 ceph-mon[37097]: pgmap v3449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:13:11.175 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:10 smithi116 ceph-mon[37097]: pgmap v3450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:11.175 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5535.710126 secs 2024-07-26T10:13:12.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:12.050+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:13:12.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:12.050+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:12.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:12.050+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:13:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:11 smithi171 ceph-mon[37789]: pgmap v3451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:11 smithi044 ceph-mon[32404]: pgmap v3451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:11 smithi116 ceph-mon[37097]: pgmap v3451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:14 smithi044 ceph-mon[32404]: pgmap v3452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:14 smithi116 ceph-mon[37097]: pgmap v3452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:14 smithi171 ceph-mon[37789]: pgmap v3452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:15.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5540.710284 secs 2024-07-26T10:13:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5540.710284 secs 2024-07-26T10:13:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5540.710284 secs 2024-07-26T10:13:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:16 smithi044 ceph-mon[32404]: pgmap v3453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:16 smithi116 ceph-mon[37097]: pgmap v3453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:16 smithi171 ceph-mon[37789]: pgmap v3453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:13:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:18 smithi044 ceph-mon[32404]: pgmap v3454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:18 smithi116 ceph-mon[37097]: pgmap v3454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:18 smithi171 ceph-mon[37789]: pgmap v3454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:20 smithi044 ceph-mon[32404]: pgmap v3455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5545.710450 secs 2024-07-26T10:13:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:20 smithi116 ceph-mon[37097]: pgmap v3455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5545.710450 secs 2024-07-26T10:13:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:20 smithi171 ceph-mon[37789]: pgmap v3455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5545.710450 secs 2024-07-26T10:13:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:22 smithi044 ceph-mon[32404]: pgmap v3456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:22 smithi116 ceph-mon[37097]: pgmap v3456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:22 smithi171 ceph-mon[37789]: pgmap v3456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:24 smithi044 ceph-mon[32404]: pgmap v3457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:13:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:24 smithi116 ceph-mon[37097]: pgmap v3457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:13:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:24 smithi171 ceph-mon[37789]: pgmap v3457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:13:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5550.710656 secs 2024-07-26T10:13:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5550.710656 secs 2024-07-26T10:13:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5550.710656 secs 2024-07-26T10:13:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:26 smithi116 ceph-mon[37097]: pgmap v3458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:26 smithi171 ceph-mon[37789]: pgmap v3458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:26 smithi044 ceph-mon[32404]: pgmap v3458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:13:27.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:27.058+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:13:27.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:27.058+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:27.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:27.058+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:13:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:28 smithi171 ceph-mon[37789]: pgmap v3459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:28 smithi044 ceph-mon[32404]: pgmap v3459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:28 smithi116 ceph-mon[37097]: pgmap v3459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:13:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:30 smithi171 ceph-mon[37789]: pgmap v3460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5555.710779 secs 2024-07-26T10:13:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:30 smithi044 ceph-mon[32404]: pgmap v3460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5555.710779 secs 2024-07-26T10:13:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:30 smithi116 ceph-mon[37097]: pgmap v3460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:13:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5555.710779 secs 2024-07-26T10:13:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:32 smithi171 ceph-mon[37789]: pgmap v3461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:32 smithi044 ceph-mon[32404]: pgmap v3461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:32 smithi116 ceph-mon[37097]: pgmap v3461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:13:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:34 smithi171 ceph-mon[37789]: pgmap v3462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:13:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:34 smithi044 ceph-mon[32404]: pgmap v3462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:13:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:34 smithi116 ceph-mon[37097]: pgmap v3462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:13:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5560.710934 secs 2024-07-26T10:13:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5560.710934 secs 2024-07-26T10:13:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5560.710934 secs 2024-07-26T10:13:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:36 smithi171 ceph-mon[37789]: pgmap v3463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:36 smithi044 ceph-mon[32404]: pgmap v3463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:36 smithi116 ceph-mon[37097]: pgmap v3463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:38 smithi171 ceph-mon[37789]: pgmap v3464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:38 smithi044 ceph-mon[32404]: pgmap v3464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:38 smithi116 ceph-mon[37097]: pgmap v3464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:40 smithi171 ceph-mon[37789]: pgmap v3465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5565.711101 secs 2024-07-26T10:13:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:40 smithi044 ceph-mon[32404]: pgmap v3465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5565.711101 secs 2024-07-26T10:13:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:40 smithi116 ceph-mon[37097]: pgmap v3465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5565.711101 secs 2024-07-26T10:13:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:42.062+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:13:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:42.062+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:13:42.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:42.062+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:42.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:42 smithi116 ceph-mon[37097]: pgmap v3466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:42 smithi171 ceph-mon[37789]: pgmap v3466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:42 smithi044 ceph-mon[32404]: pgmap v3466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:44 smithi171 ceph-mon[37789]: pgmap v3467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:45.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:44 smithi044 ceph-mon[32404]: pgmap v3467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:44 smithi116 ceph-mon[37097]: pgmap v3467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5570.711268 secs 2024-07-26T10:13:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5570.711268 secs 2024-07-26T10:13:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5570.711268 secs 2024-07-26T10:13:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:46 smithi044 ceph-mon[32404]: pgmap v3468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:46 smithi116 ceph-mon[37097]: pgmap v3468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:46 smithi171 ceph-mon[37789]: pgmap v3468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:48 smithi044 ceph-mon[32404]: pgmap v3469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:48 smithi116 ceph-mon[37097]: pgmap v3469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:48 smithi171 ceph-mon[37789]: pgmap v3469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:50 smithi044 ceph-mon[32404]: pgmap v3470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5575.711426 secs 2024-07-26T10:13:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:50 smithi116 ceph-mon[37097]: pgmap v3470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5575.711426 secs 2024-07-26T10:13:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:50 smithi171 ceph-mon[37789]: pgmap v3470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:13:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5575.711426 secs 2024-07-26T10:13:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:52 smithi044 ceph-mon[32404]: pgmap v3471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:53.046 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:52 smithi116 ceph-mon[37097]: pgmap v3471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:52 smithi171 ceph-mon[37789]: pgmap v3471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:54 smithi044 ceph-mon[32404]: pgmap v3472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:54 smithi116 ceph-mon[37097]: pgmap v3472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:54 smithi171 ceph-mon[37789]: pgmap v3472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:13:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5580.711574 secs 2024-07-26T10:13:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5580.711574 secs 2024-07-26T10:13:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5580.711574 secs 2024-07-26T10:13:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:56 smithi044 ceph-mon[32404]: pgmap v3473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:56 smithi116 ceph-mon[37097]: pgmap v3473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:57.071+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:13:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:57.071+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:13:57.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:13:57.071+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:56 smithi171 ceph-mon[37789]: pgmap v3473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:13:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:13:58 smithi044 ceph-mon[32404]: pgmap v3474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:13:58 smithi116 ceph-mon[37097]: pgmap v3474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:13:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:13:58 smithi171 ceph-mon[37789]: pgmap v3474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:00 smithi044 ceph-mon[32404]: pgmap v3475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5585.711757 secs 2024-07-26T10:14:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:00 smithi116 ceph-mon[37097]: pgmap v3475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5585.711757 secs 2024-07-26T10:14:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:00 smithi171 ceph-mon[37789]: pgmap v3475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:01.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5585.711757 secs 2024-07-26T10:14:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:02 smithi044 ceph-mon[32404]: pgmap v3476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:03.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:02 smithi116 ceph-mon[37097]: pgmap v3476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:02 smithi171 ceph-mon[37789]: pgmap v3476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:04 smithi044 ceph-mon[32404]: pgmap v3477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:14:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:04 smithi116 ceph-mon[37097]: pgmap v3477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:14:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:04 smithi171 ceph-mon[37789]: pgmap v3477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:14:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5590.711921 secs 2024-07-26T10:14:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5590.711921 secs 2024-07-26T10:14:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5590.711921 secs 2024-07-26T10:14:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:06 smithi044 ceph-mon[32404]: pgmap v3478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:06 smithi116 ceph-mon[37097]: pgmap v3478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:06 smithi171 ceph-mon[37789]: pgmap v3478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:08 smithi044 ceph-mon[32404]: pgmap v3479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:08 smithi116 ceph-mon[37097]: pgmap v3479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:08 smithi171 ceph-mon[37789]: pgmap v3479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: pgmap v3480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5595.712052 secs 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: pgmap v3480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5595.712052 secs 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: pgmap v3480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5595.712052 secs 2024-07-26T10:14:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: pgmap v3481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:14:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: pgmap v3482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:12.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: pgmap v3481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: pgmap v3482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:14:12.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:12.075+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:14:12.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:12.075+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:12.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:12.075+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:14:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: pgmap v3481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: pgmap v3482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:14:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:13 smithi116 ceph-mon[37097]: pgmap v3483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:14:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:13 smithi171 ceph-mon[37789]: pgmap v3483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:14:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:13 smithi044 ceph-mon[32404]: pgmap v3483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:14:15.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:15.665+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:14:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:15.665+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:14:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:15.666+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:14:15.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:15.666+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:14:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:15 smithi044 ceph-mon[32404]: pgmap v3484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:16.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5600.712235 secs 2024-07-26T10:14:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:15 smithi171 ceph-mon[37789]: pgmap v3484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5600.712235 secs 2024-07-26T10:14:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:15 smithi116 ceph-mon[37097]: pgmap v3484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5600.712235 secs 2024-07-26T10:14:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:17 smithi171 ceph-mon[37789]: pgmap v3485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:17 smithi044 ceph-mon[32404]: pgmap v3485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:17 smithi116 ceph-mon[37097]: pgmap v3485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:19 smithi171 ceph-mon[37789]: pgmap v3486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:19 smithi044 ceph-mon[32404]: pgmap v3486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:19 smithi116 ceph-mon[37097]: pgmap v3486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:14:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5605.712398 secs 2024-07-26T10:14:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5605.712398 secs 2024-07-26T10:14:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5605.712398 secs 2024-07-26T10:14:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:21 smithi171 ceph-mon[37789]: pgmap v3487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:21 smithi044 ceph-mon[32404]: pgmap v3487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:21 smithi116 ceph-mon[37097]: pgmap v3487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:14:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:23 smithi171 ceph-mon[37789]: pgmap v3488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:23 smithi044 ceph-mon[32404]: pgmap v3488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:23 smithi116 ceph-mon[37097]: pgmap v3488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:25 smithi171 ceph-mon[37789]: pgmap v3489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5610.712577 secs 2024-07-26T10:14:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:25 smithi044 ceph-mon[32404]: pgmap v3489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5610.712577 secs 2024-07-26T10:14:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:25 smithi116 ceph-mon[37097]: pgmap v3489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5610.712577 secs 2024-07-26T10:14:27.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:27.083+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:14:27.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:27.083+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:14:27.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:27.083+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:27 smithi171 ceph-mon[37789]: pgmap v3490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:27 smithi044 ceph-mon[32404]: pgmap v3490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:27 smithi116 ceph-mon[37097]: pgmap v3490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:29 smithi171 ceph-mon[37789]: pgmap v3491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:29 smithi044 ceph-mon[32404]: pgmap v3491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:29 smithi116 ceph-mon[37097]: pgmap v3491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5615.712738 secs 2024-07-26T10:14:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5615.712738 secs 2024-07-26T10:14:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5615.712738 secs 2024-07-26T10:14:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:31 smithi044 ceph-mon[32404]: pgmap v3492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:14:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:31 smithi116 ceph-mon[37097]: pgmap v3492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:14:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:31 smithi171 ceph-mon[37789]: pgmap v3492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:14:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:33 smithi044 ceph-mon[32404]: pgmap v3493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:33 smithi116 ceph-mon[37097]: pgmap v3493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:33 smithi171 ceph-mon[37789]: pgmap v3493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:35 smithi044 ceph-mon[32404]: pgmap v3494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5620.712923 secs 2024-07-26T10:14:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:35 smithi116 ceph-mon[37097]: pgmap v3494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5620.712923 secs 2024-07-26T10:14:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:35 smithi171 ceph-mon[37789]: pgmap v3494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:14:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5620.712923 secs 2024-07-26T10:14:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:36 smithi044 ceph-mon[32404]: pgmap v3495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:36 smithi116 ceph-mon[37097]: pgmap v3495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:36 smithi171 ceph-mon[37789]: pgmap v3495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:39 smithi171 ceph-mon[37789]: pgmap v3496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:39 smithi044 ceph-mon[32404]: pgmap v3496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:39 smithi116 ceph-mon[37097]: pgmap v3496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5625.713098 secs 2024-07-26T10:14:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5625.713098 secs 2024-07-26T10:14:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5625.713098 secs 2024-07-26T10:14:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:41 smithi171 ceph-mon[37789]: pgmap v3497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:42.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:42.087+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:14:42.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:42.087+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:42.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:42.087+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:14:42.090 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:41 smithi044 ceph-mon[32404]: pgmap v3497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:41 smithi116 ceph-mon[37097]: pgmap v3497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:14:44.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:43 smithi116 ceph-mon[37097]: pgmap v3498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:43 smithi171 ceph-mon[37789]: pgmap v3498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:43 smithi044 ceph-mon[32404]: pgmap v3498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:45 smithi171 ceph-mon[37789]: pgmap v3499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:46.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5630.713280 secs 2024-07-26T10:14:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:45 smithi044 ceph-mon[32404]: pgmap v3499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5630.713280 secs 2024-07-26T10:14:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:45 smithi116 ceph-mon[37097]: pgmap v3499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5630.713280 secs 2024-07-26T10:14:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:47 smithi171 ceph-mon[37789]: pgmap v3500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:47 smithi044 ceph-mon[32404]: pgmap v3500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:47 smithi116 ceph-mon[37097]: pgmap v3500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:49 smithi171 ceph-mon[37789]: pgmap v3501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:49 smithi044 ceph-mon[32404]: pgmap v3501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:49 smithi116 ceph-mon[37097]: pgmap v3501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5635.713450 secs 2024-07-26T10:14:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5635.713450 secs 2024-07-26T10:14:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5635.713450 secs 2024-07-26T10:14:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:51 smithi171 ceph-mon[37789]: pgmap v3502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:14:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:51 smithi044 ceph-mon[32404]: pgmap v3502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:14:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:51 smithi116 ceph-mon[37097]: pgmap v3502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:14:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:53 smithi044 ceph-mon[32404]: pgmap v3503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:53 smithi116 ceph-mon[37097]: pgmap v3503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:53 smithi171 ceph-mon[37789]: pgmap v3503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:14:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:55 smithi044 ceph-mon[32404]: pgmap v3504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5640.713601 secs 2024-07-26T10:14:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:55 smithi116 ceph-mon[37097]: pgmap v3504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5640.713601 secs 2024-07-26T10:14:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:55 smithi171 ceph-mon[37789]: pgmap v3504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:14:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5640.713601 secs 2024-07-26T10:14:57.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:57.096+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:14:57.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:57.096+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:57.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:14:57.096+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:14:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:57 smithi044 ceph-mon[32404]: pgmap v3505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:57 smithi116 ceph-mon[37097]: pgmap v3505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:57 smithi171 ceph-mon[37789]: pgmap v3505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:14:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:14:58 smithi044 ceph-mon[32404]: pgmap v3506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:14:58 smithi116 ceph-mon[37097]: pgmap v3506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:14:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:14:58 smithi171 ceph-mon[37789]: pgmap v3506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5645.713761 secs 2024-07-26T10:15:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5645.713761 secs 2024-07-26T10:15:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5645.713761 secs 2024-07-26T10:15:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:01 smithi044 ceph-mon[32404]: pgmap v3507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:01 smithi116 ceph-mon[37097]: pgmap v3507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:01 smithi171 ceph-mon[37789]: pgmap v3507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:03 smithi171 ceph-mon[37789]: pgmap v3508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:04.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:03 smithi044 ceph-mon[32404]: pgmap v3508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:03 smithi116 ceph-mon[37097]: pgmap v3508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:05 smithi171 ceph-mon[37789]: pgmap v3509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5650.713923 secs 2024-07-26T10:15:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:05 smithi044 ceph-mon[32404]: pgmap v3509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5650.713923 secs 2024-07-26T10:15:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:05 smithi116 ceph-mon[37097]: pgmap v3509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:15:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5650.713923 secs 2024-07-26T10:15:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:07 smithi171 ceph-mon[37789]: pgmap v3510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:07 smithi044 ceph-mon[32404]: pgmap v3510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:07 smithi116 ceph-mon[37097]: pgmap v3510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:09 smithi171 ceph-mon[37789]: pgmap v3511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:09 smithi044 ceph-mon[32404]: pgmap v3511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:09 smithi116 ceph-mon[37097]: pgmap v3511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5655.714084 secs 2024-07-26T10:15:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5655.714084 secs 2024-07-26T10:15:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5655.714084 secs 2024-07-26T10:15:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: pgmap v3512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:12.099+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:15:12.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:12.099+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:12.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:12.099+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: pgmap v3512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.103 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: pgmap v3512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:12.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:15:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:15:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:15:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:15:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:15:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:15:14.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:13 smithi116 ceph-mon[37097]: pgmap v3513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:15:14.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:13 smithi116 ceph-mon[37097]: pgmap v3514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:14.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:13 smithi171 ceph-mon[37789]: pgmap v3513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:15:14.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:13 smithi171 ceph-mon[37789]: pgmap v3514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:13 smithi044 ceph-mon[32404]: pgmap v3513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:15:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:13 smithi044 ceph-mon[32404]: pgmap v3514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:14 smithi044 ceph-mon[32404]: pgmap v3515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:15.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:14 smithi116 ceph-mon[37097]: pgmap v3515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:14 smithi171 ceph-mon[37789]: pgmap v3515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5660.714260 secs 2024-07-26T10:15:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5660.714260 secs 2024-07-26T10:15:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5660.714260 secs 2024-07-26T10:15:17.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:16 smithi044 ceph-mon[32404]: pgmap v3516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:15:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:16 smithi116 ceph-mon[37097]: pgmap v3516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:15:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:16 smithi171 ceph-mon[37789]: pgmap v3516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:15:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:19 smithi171 ceph-mon[37789]: pgmap v3517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:19 smithi044 ceph-mon[32404]: pgmap v3517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:19 smithi116 ceph-mon[37097]: pgmap v3517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:21 smithi171 ceph-mon[37789]: pgmap v3518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5665.714427 secs 2024-07-26T10:15:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:21 smithi044 ceph-mon[32404]: pgmap v3518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5665.714427 secs 2024-07-26T10:15:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:21 smithi116 ceph-mon[37097]: pgmap v3518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:15:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5665.714427 secs 2024-07-26T10:15:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:23 smithi171 ceph-mon[37789]: pgmap v3519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:23 smithi044 ceph-mon[32404]: pgmap v3519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:23 smithi116 ceph-mon[37097]: pgmap v3519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:15:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:25 smithi171 ceph-mon[37789]: pgmap v3520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:25 smithi044 ceph-mon[32404]: pgmap v3520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:25 smithi116 ceph-mon[37097]: pgmap v3520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5670.714550 secs 2024-07-26T10:15:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5670.714550 secs 2024-07-26T10:15:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5670.714550 secs 2024-07-26T10:15:27.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:27.108+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:15:27.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:27.108+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:27.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:27.108+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:15:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:27 smithi171 ceph-mon[37789]: pgmap v3521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:27.421 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:27 smithi044 ceph-mon[32404]: pgmap v3521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:27 smithi116 ceph-mon[37097]: pgmap v3521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:29 smithi171 ceph-mon[37789]: pgmap v3522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:29 smithi044 ceph-mon[32404]: pgmap v3522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:29 smithi116 ceph-mon[37097]: pgmap v3522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:31 smithi171 ceph-mon[37789]: pgmap v3523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5675.714723 secs 2024-07-26T10:15:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:31 smithi044 ceph-mon[32404]: pgmap v3523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5675.714723 secs 2024-07-26T10:15:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:31 smithi116 ceph-mon[37097]: pgmap v3523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5675.714723 secs 2024-07-26T10:15:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:33 smithi171 ceph-mon[37789]: pgmap v3524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:33 smithi044 ceph-mon[32404]: pgmap v3524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:33 smithi116 ceph-mon[37097]: pgmap v3524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:35 smithi044 ceph-mon[32404]: pgmap v3525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:35.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:35 smithi116 ceph-mon[37097]: pgmap v3525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:35 smithi171 ceph-mon[37789]: pgmap v3525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5680.714886 secs 2024-07-26T10:15:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5680.714886 secs 2024-07-26T10:15:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5680.714886 secs 2024-07-26T10:15:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:37 smithi044 ceph-mon[32404]: pgmap v3526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:37 smithi116 ceph-mon[37097]: pgmap v3526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:37 smithi171 ceph-mon[37789]: pgmap v3526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:39 smithi044 ceph-mon[32404]: pgmap v3527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:39 smithi116 ceph-mon[37097]: pgmap v3527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:39 smithi171 ceph-mon[37789]: pgmap v3527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:41 smithi044 ceph-mon[32404]: pgmap v3528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5685.715046 secs 2024-07-26T10:15:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:41 smithi116 ceph-mon[37097]: pgmap v3528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5685.715046 secs 2024-07-26T10:15:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:41 smithi171 ceph-mon[37789]: pgmap v3528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:15:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5685.715046 secs 2024-07-26T10:15:42.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:42.111+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:15:42.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:42.111+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:15:42.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:42.111+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:43 smithi044 ceph-mon[32404]: pgmap v3529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:43 smithi116 ceph-mon[37097]: pgmap v3529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:43 smithi171 ceph-mon[37789]: pgmap v3529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:45.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:45 smithi171 ceph-mon[37789]: pgmap v3530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:45 smithi044 ceph-mon[32404]: pgmap v3530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:45 smithi116 ceph-mon[37097]: pgmap v3530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5690.715217 secs 2024-07-26T10:15:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5690.715217 secs 2024-07-26T10:15:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5690.715217 secs 2024-07-26T10:15:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:47 smithi044 ceph-mon[32404]: pgmap v3531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:47 smithi116 ceph-mon[37097]: pgmap v3531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:47 smithi171 ceph-mon[37789]: pgmap v3531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:49 smithi044 ceph-mon[32404]: pgmap v3532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:49 smithi116 ceph-mon[37097]: pgmap v3532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:49 smithi171 ceph-mon[37789]: pgmap v3532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:15:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:51 smithi044 ceph-mon[32404]: pgmap v3533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5695.715403 secs 2024-07-26T10:15:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:51 smithi116 ceph-mon[37097]: pgmap v3533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5695.715403 secs 2024-07-26T10:15:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:51 smithi171 ceph-mon[37789]: pgmap v3533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5695.715403 secs 2024-07-26T10:15:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:53 smithi116 ceph-mon[37097]: pgmap v3534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:53 smithi171 ceph-mon[37789]: pgmap v3534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:53 smithi044 ceph-mon[32404]: pgmap v3534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:55 smithi116 ceph-mon[37097]: pgmap v3535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5700.715582 secs 2024-07-26T10:15:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:55 smithi171 ceph-mon[37789]: pgmap v3535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5700.715582 secs 2024-07-26T10:15:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:55 smithi044 ceph-mon[32404]: pgmap v3535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:15:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5700.715582 secs 2024-07-26T10:15:57.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:57.120+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:15:57.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:57.120+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:57.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:15:57.120+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:15:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:57 smithi171 ceph-mon[37789]: pgmap v3536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:57 smithi044 ceph-mon[32404]: pgmap v3536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:15:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:57 smithi116 ceph-mon[37097]: pgmap v3536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:15:59 smithi171 ceph-mon[37789]: pgmap v3537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:15:59 smithi044 ceph-mon[32404]: pgmap v3537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:15:59 smithi116 ceph-mon[37097]: pgmap v3537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5705.715713 secs 2024-07-26T10:16:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5705.715713 secs 2024-07-26T10:16:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5705.715713 secs 2024-07-26T10:16:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:01 smithi044 ceph-mon[32404]: pgmap v3538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:01 smithi171 ceph-mon[37789]: pgmap v3538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:01 smithi116 ceph-mon[37097]: pgmap v3538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:03 smithi171 ceph-mon[37789]: pgmap v3539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:03 smithi044 ceph-mon[32404]: pgmap v3539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:03 smithi116 ceph-mon[37097]: pgmap v3539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:05 smithi171 ceph-mon[37789]: pgmap v3540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:06.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5710.715905 secs 2024-07-26T10:16:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:05 smithi044 ceph-mon[32404]: pgmap v3540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5710.715905 secs 2024-07-26T10:16:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:05 smithi116 ceph-mon[37097]: pgmap v3540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5710.715905 secs 2024-07-26T10:16:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:07 smithi171 ceph-mon[37789]: pgmap v3541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:07 smithi044 ceph-mon[32404]: pgmap v3541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:07 smithi116 ceph-mon[37097]: pgmap v3541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:09 smithi171 ceph-mon[37789]: pgmap v3542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:09 smithi044 ceph-mon[32404]: pgmap v3542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:09 smithi116 ceph-mon[37097]: pgmap v3542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5715.716066 secs 2024-07-26T10:16:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5715.716066 secs 2024-07-26T10:16:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5715.716066 secs 2024-07-26T10:16:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:11 smithi171 ceph-mon[37789]: pgmap v3543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:12.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:12.124+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:16:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:12.124+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:16:12.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:12.124+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:12.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:11 smithi044 ceph-mon[32404]: pgmap v3543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:11 smithi116 ceph-mon[37097]: pgmap v3543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:13 smithi044 ceph-mon[32404]: pgmap v3544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:16:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:13 smithi116 ceph-mon[37097]: pgmap v3544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:16:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:13 smithi171 ceph-mon[37789]: pgmap v3544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:16:15.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:14 smithi116 ceph-mon[37097]: pgmap v3545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:16:15.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:14 smithi116 ceph-mon[37097]: pgmap v3546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:15.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:14 smithi171 ceph-mon[37789]: pgmap v3545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:16:15.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:14 smithi171 ceph-mon[37789]: pgmap v3546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:14 smithi044 ceph-mon[32404]: pgmap v3545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:16:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:14 smithi044 ceph-mon[32404]: pgmap v3546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5720.716248 secs 2024-07-26T10:16:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5720.716248 secs 2024-07-26T10:16:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5720.716248 secs 2024-07-26T10:16:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:16 smithi044 ceph-mon[32404]: pgmap v3547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:16 smithi116 ceph-mon[37097]: pgmap v3547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:16 smithi171 ceph-mon[37789]: pgmap v3547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:17 smithi044 ceph-mon[32404]: pgmap v3548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:16:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:17 smithi116 ceph-mon[37097]: pgmap v3548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:16:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:17 smithi171 ceph-mon[37789]: pgmap v3548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:16:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:20 smithi171 ceph-mon[37789]: pgmap v3549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:20 smithi044 ceph-mon[32404]: pgmap v3549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:20 smithi116 ceph-mon[37097]: pgmap v3549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:16:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5725.716412 secs 2024-07-26T10:16:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5725.716412 secs 2024-07-26T10:16:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5725.716412 secs 2024-07-26T10:16:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:22 smithi044 ceph-mon[32404]: pgmap v3550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:16:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:22 smithi116 ceph-mon[37097]: pgmap v3550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:16:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:22 smithi171 ceph-mon[37789]: pgmap v3550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:16:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:24 smithi044 ceph-mon[32404]: pgmap v3551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:16:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:24 smithi116 ceph-mon[37097]: pgmap v3551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:16:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:24 smithi171 ceph-mon[37789]: pgmap v3551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:16:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:26 smithi044 ceph-mon[32404]: pgmap v3552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5730.716520 secs 2024-07-26T10:16:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:26 smithi116 ceph-mon[37097]: pgmap v3552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5730.716520 secs 2024-07-26T10:16:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:26 smithi171 ceph-mon[37789]: pgmap v3552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5730.716520 secs 2024-07-26T10:16:27.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:27.130+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:16:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:27.130+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:16:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:27.130+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:28 smithi044 ceph-mon[32404]: pgmap v3553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:28 smithi116 ceph-mon[37097]: pgmap v3553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:28 smithi171 ceph-mon[37789]: pgmap v3553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:30 smithi044 ceph-mon[32404]: pgmap v3554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:30 smithi116 ceph-mon[37097]: pgmap v3554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:30 smithi171 ceph-mon[37789]: pgmap v3554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5735.716684 secs 2024-07-26T10:16:31.798 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5735.716684 secs 2024-07-26T10:16:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5735.716684 secs 2024-07-26T10:16:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:32 smithi044 ceph-mon[32404]: pgmap v3555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:32 smithi116 ceph-mon[37097]: pgmap v3555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:32 smithi171 ceph-mon[37789]: pgmap v3555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:34 smithi044 ceph-mon[32404]: pgmap v3556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:34 smithi116 ceph-mon[37097]: pgmap v3556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:34 smithi171 ceph-mon[37789]: pgmap v3556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:36 smithi044 ceph-mon[32404]: pgmap v3557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5740.716820 secs 2024-07-26T10:16:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:36 smithi116 ceph-mon[37097]: pgmap v3557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5740.716820 secs 2024-07-26T10:16:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:36 smithi171 ceph-mon[37789]: pgmap v3557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5740.716820 secs 2024-07-26T10:16:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:38 smithi044 ceph-mon[32404]: pgmap v3558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:38 smithi116 ceph-mon[37097]: pgmap v3558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:38 smithi171 ceph-mon[37789]: pgmap v3558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:40 smithi044 ceph-mon[32404]: pgmap v3559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:40 smithi116 ceph-mon[37097]: pgmap v3559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:40 smithi171 ceph-mon[37789]: pgmap v3559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5745.716987 secs 2024-07-26T10:16:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5745.716987 secs 2024-07-26T10:16:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5745.716987 secs 2024-07-26T10:16:42.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:42.133+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:16:42.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:42.133+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:42.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:42.133+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:16:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:42 smithi044 ceph-mon[32404]: pgmap v3560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:42 smithi116 ceph-mon[37097]: pgmap v3560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:42 smithi171 ceph-mon[37789]: pgmap v3560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:44 smithi044 ceph-mon[32404]: pgmap v3561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:44 smithi116 ceph-mon[37097]: pgmap v3561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:44 smithi171 ceph-mon[37789]: pgmap v3561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:45.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5750.717146 secs 2024-07-26T10:16:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5750.717146 secs 2024-07-26T10:16:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5750.717146 secs 2024-07-26T10:16:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:46 smithi044 ceph-mon[32404]: pgmap v3562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:46 smithi116 ceph-mon[37097]: pgmap v3562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:46 smithi171 ceph-mon[37789]: pgmap v3562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:48 smithi044 ceph-mon[32404]: pgmap v3563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:48 smithi116 ceph-mon[37097]: pgmap v3563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:48 smithi171 ceph-mon[37789]: pgmap v3563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:50 smithi044 ceph-mon[32404]: pgmap v3564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5755.717311 secs 2024-07-26T10:16:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:50 smithi116 ceph-mon[37097]: pgmap v3564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5755.717311 secs 2024-07-26T10:16:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:50 smithi171 ceph-mon[37789]: pgmap v3564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:16:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5755.717311 secs 2024-07-26T10:16:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:52 smithi044 ceph-mon[32404]: pgmap v3565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:52 smithi116 ceph-mon[37097]: pgmap v3565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:52 smithi171 ceph-mon[37789]: pgmap v3565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:54 smithi044 ceph-mon[32404]: pgmap v3566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:54 smithi116 ceph-mon[37097]: pgmap v3566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:54 smithi171 ceph-mon[37789]: pgmap v3566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:16:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:55 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 5760.717473 secs 2024-07-26T10:16:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:55 smithi044 ceph-mon[32404]: slow request 5760.717472 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T10:16:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:55 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 5760.717473 secs 2024-07-26T10:16:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:55 smithi116 ceph-mon[37097]: slow request 5760.717472 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T10:16:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:55 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 5760.717473 secs 2024-07-26T10:16:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:55 smithi171 ceph-mon[37789]: slow request 5760.717472 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T10:16:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:56 smithi044 ceph-mon[32404]: pgmap v3567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:56 smithi116 ceph-mon[37097]: pgmap v3567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:56 smithi171 ceph-mon[37789]: pgmap v3567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:16:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:57.136+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:16:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:57.137+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:16:57.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:16:57.137+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:16:58 smithi044 ceph-mon[32404]: pgmap v3568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:16:58 smithi116 ceph-mon[37097]: pgmap v3568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:16:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:16:58 smithi171 ceph-mon[37789]: pgmap v3568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:00 smithi044 ceph-mon[32404]: pgmap v3569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5765.717650 secs 2024-07-26T10:17:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:00 smithi116 ceph-mon[37097]: pgmap v3569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5765.717650 secs 2024-07-26T10:17:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:00 smithi171 ceph-mon[37789]: pgmap v3569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5765.717650 secs 2024-07-26T10:17:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:02 smithi044 ceph-mon[32404]: pgmap v3570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:02 smithi116 ceph-mon[37097]: pgmap v3570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:02 smithi171 ceph-mon[37789]: pgmap v3570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:04 smithi044 ceph-mon[32404]: pgmap v3571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:04.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:04 smithi116 ceph-mon[37097]: pgmap v3571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:04 smithi171 ceph-mon[37789]: pgmap v3571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5770.717780 secs 2024-07-26T10:17:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5770.717780 secs 2024-07-26T10:17:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5770.717780 secs 2024-07-26T10:17:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:06 smithi044 ceph-mon[32404]: pgmap v3572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:06 smithi116 ceph-mon[37097]: pgmap v3572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:06 smithi171 ceph-mon[37789]: pgmap v3572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:08 smithi044 ceph-mon[32404]: pgmap v3573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:08 smithi116 ceph-mon[37097]: pgmap v3573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:08 smithi171 ceph-mon[37789]: pgmap v3573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:10 smithi116 ceph-mon[37097]: pgmap v3574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5775.717894 secs 2024-07-26T10:17:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:10 smithi171 ceph-mon[37789]: pgmap v3574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5775.717894 secs 2024-07-26T10:17:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:10 smithi044 ceph-mon[32404]: pgmap v3574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5775.717894 secs 2024-07-26T10:17:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:12.141+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:17:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:12.141+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:17:12.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:12.142+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:12 smithi171 ceph-mon[37789]: pgmap v3575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:12 smithi044 ceph-mon[32404]: pgmap v3575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:12 smithi116 ceph-mon[37097]: pgmap v3575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:14 smithi171 ceph-mon[37789]: pgmap v3576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:14 smithi044 ceph-mon[32404]: pgmap v3576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:14 smithi116 ceph-mon[37097]: pgmap v3576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:15.697+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:17:15.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:15.697+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:17:15.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:15.698+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:17:15.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:15.698+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:17:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5780.717996 secs 2024-07-26T10:17:15.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:15.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:15.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:15.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5780.717996 secs 2024-07-26T10:17:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:16.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:17:16.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5780.717996 secs 2024-07-26T10:17:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:16 smithi171 ceph-mon[37789]: pgmap v3577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:16 smithi044 ceph-mon[32404]: pgmap v3577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:16 smithi116 ceph-mon[37097]: pgmap v3577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:18 smithi171 ceph-mon[37789]: pgmap v3578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:19.014 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:18 smithi044 ceph-mon[32404]: pgmap v3578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:18 smithi116 ceph-mon[37097]: pgmap v3578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:20 smithi171 ceph-mon[37789]: pgmap v3579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5785.718142 secs 2024-07-26T10:17:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:20 smithi044 ceph-mon[32404]: pgmap v3579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5785.718142 secs 2024-07-26T10:17:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:20 smithi116 ceph-mon[37097]: pgmap v3579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5785.718142 secs 2024-07-26T10:17:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:22 smithi171 ceph-mon[37789]: pgmap v3580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:22 smithi044 ceph-mon[32404]: pgmap v3580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:22 smithi116 ceph-mon[37097]: pgmap v3580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:24 smithi171 ceph-mon[37789]: pgmap v3581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:24 smithi044 ceph-mon[32404]: pgmap v3581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:24 smithi116 ceph-mon[37097]: pgmap v3581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5790.718232 secs 2024-07-26T10:17:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5790.718232 secs 2024-07-26T10:17:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5790.718232 secs 2024-07-26T10:17:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:26 smithi044 ceph-mon[32404]: pgmap v3582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:26 smithi116 ceph-mon[37097]: pgmap v3582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:26 smithi171 ceph-mon[37789]: pgmap v3582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:17:27.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:27.151+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:17:27.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:27.151+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:27.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:27.151+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:17:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:28 smithi044 ceph-mon[32404]: pgmap v3583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:28 smithi116 ceph-mon[37097]: pgmap v3583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:28 smithi171 ceph-mon[37789]: pgmap v3583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:30 smithi044 ceph-mon[32404]: pgmap v3584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5795.718379 secs 2024-07-26T10:17:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:30 smithi116 ceph-mon[37097]: pgmap v3584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5795.718379 secs 2024-07-26T10:17:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:30 smithi171 ceph-mon[37789]: pgmap v3584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5795.718379 secs 2024-07-26T10:17:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:32 smithi044 ceph-mon[32404]: pgmap v3585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:32 smithi116 ceph-mon[37097]: pgmap v3585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:32 smithi171 ceph-mon[37789]: pgmap v3585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:34.882 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:34 smithi044 ceph-mon[32404]: pgmap v3586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:34 smithi116 ceph-mon[37097]: pgmap v3586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:34 smithi171 ceph-mon[37789]: pgmap v3586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5800.718535 secs 2024-07-26T10:17:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5800.718535 secs 2024-07-26T10:17:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5800.718535 secs 2024-07-26T10:17:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:36 smithi044 ceph-mon[32404]: pgmap v3587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:36 smithi116 ceph-mon[37097]: pgmap v3587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:36 smithi171 ceph-mon[37789]: pgmap v3587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:38 smithi044 ceph-mon[32404]: pgmap v3588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:38 smithi116 ceph-mon[37097]: pgmap v3588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:38 smithi171 ceph-mon[37789]: pgmap v3588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:40 smithi044 ceph-mon[32404]: pgmap v3589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5805.718681 secs 2024-07-26T10:17:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:40 smithi116 ceph-mon[37097]: pgmap v3589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5805.718681 secs 2024-07-26T10:17:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:40 smithi171 ceph-mon[37789]: pgmap v3589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5805.718681 secs 2024-07-26T10:17:42.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:42.154+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:17:42.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:42.154+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:42.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:42.154+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:17:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:42 smithi044 ceph-mon[32404]: pgmap v3590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:42 smithi116 ceph-mon[37097]: pgmap v3590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:42 smithi171 ceph-mon[37789]: pgmap v3590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:44 smithi044 ceph-mon[32404]: pgmap v3591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:44 smithi116 ceph-mon[37097]: pgmap v3591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:44 smithi171 ceph-mon[37789]: pgmap v3591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:45.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5810.718819 secs 2024-07-26T10:17:45.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5810.718819 secs 2024-07-26T10:17:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5810.718819 secs 2024-07-26T10:17:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:46 smithi044 ceph-mon[32404]: pgmap v3592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:46 smithi116 ceph-mon[37097]: pgmap v3592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:46 smithi171 ceph-mon[37789]: pgmap v3592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:48 smithi044 ceph-mon[32404]: pgmap v3593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:48 smithi116 ceph-mon[37097]: pgmap v3593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:48 smithi171 ceph-mon[37789]: pgmap v3593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:50 smithi044 ceph-mon[32404]: pgmap v3594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5815.718928 secs 2024-07-26T10:17:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:50 smithi116 ceph-mon[37097]: pgmap v3594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5815.718928 secs 2024-07-26T10:17:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:50 smithi171 ceph-mon[37789]: pgmap v3594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:17:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5815.718928 secs 2024-07-26T10:17:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:52 smithi044 ceph-mon[32404]: pgmap v3595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:52 smithi116 ceph-mon[37097]: pgmap v3595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:52 smithi171 ceph-mon[37789]: pgmap v3595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:54 smithi044 ceph-mon[32404]: pgmap v3596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:54 smithi116 ceph-mon[37097]: pgmap v3596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:54 smithi171 ceph-mon[37789]: pgmap v3596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:17:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5820.719077 secs 2024-07-26T10:17:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5820.719077 secs 2024-07-26T10:17:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5820.719077 secs 2024-07-26T10:17:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:56 smithi044 ceph-mon[32404]: pgmap v3597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:56 smithi116 ceph-mon[37097]: pgmap v3597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:56 smithi171 ceph-mon[37789]: pgmap v3597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:17:57.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:57.159+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:17:57.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:57.159+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:57.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:17:57.159+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:17:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:17:58 smithi044 ceph-mon[32404]: pgmap v3598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:17:58 smithi116 ceph-mon[37097]: pgmap v3598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:17:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:17:58 smithi171 ceph-mon[37789]: pgmap v3598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:00 smithi044 ceph-mon[32404]: pgmap v3599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5825.719169 secs 2024-07-26T10:18:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:00 smithi116 ceph-mon[37097]: pgmap v3599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5825.719169 secs 2024-07-26T10:18:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:00 smithi171 ceph-mon[37789]: pgmap v3599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5825.719169 secs 2024-07-26T10:18:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:02 smithi044 ceph-mon[32404]: pgmap v3600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:03.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:02 smithi116 ceph-mon[37097]: pgmap v3600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:02 smithi171 ceph-mon[37789]: pgmap v3600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:04 smithi044 ceph-mon[32404]: pgmap v3601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:04 smithi116 ceph-mon[37097]: pgmap v3601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:04 smithi171 ceph-mon[37789]: pgmap v3601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:05.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5830.719330 secs 2024-07-26T10:18:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5830.719330 secs 2024-07-26T10:18:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5830.719330 secs 2024-07-26T10:18:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:06 smithi044 ceph-mon[32404]: pgmap v3602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:06 smithi116 ceph-mon[37097]: pgmap v3602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:06 smithi171 ceph-mon[37789]: pgmap v3602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:08 smithi044 ceph-mon[32404]: pgmap v3603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:09.047 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:08 smithi116 ceph-mon[37097]: pgmap v3603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:08 smithi171 ceph-mon[37789]: pgmap v3603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:10 smithi044 ceph-mon[32404]: pgmap v3604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5835.719481 secs 2024-07-26T10:18:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:10 smithi116 ceph-mon[37097]: pgmap v3604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5835.719481 secs 2024-07-26T10:18:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:10 smithi171 ceph-mon[37789]: pgmap v3604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5835.719481 secs 2024-07-26T10:18:12.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:12.163+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:18:12.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:12.163+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:18:12.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:12.163+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:12 smithi044 ceph-mon[32404]: pgmap v3605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:12 smithi116 ceph-mon[37097]: pgmap v3605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:12 smithi171 ceph-mon[37789]: pgmap v3605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:14 smithi044 ceph-mon[32404]: pgmap v3606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:14 smithi116 ceph-mon[37097]: pgmap v3606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:14 smithi171 ceph-mon[37789]: pgmap v3606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5840.719576 secs 2024-07-26T10:18:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5840.719576 secs 2024-07-26T10:18:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5840.719576 secs 2024-07-26T10:18:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:16 smithi044 ceph-mon[32404]: pgmap v3607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:16 smithi116 ceph-mon[37097]: pgmap v3607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:16 smithi171 ceph-mon[37789]: pgmap v3607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:17.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:18 smithi044 ceph-mon[32404]: pgmap v3608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:18 smithi116 ceph-mon[37097]: pgmap v3608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:18 smithi171 ceph-mon[37789]: pgmap v3608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:18:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:18.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:20 smithi116 ceph-mon[37097]: pgmap v3609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:20 smithi171 ceph-mon[37789]: pgmap v3609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:20.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:20 smithi044 ceph-mon[32404]: pgmap v3609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:21 smithi044 ceph-mon[32404]: pgmap v3610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:21 smithi044 ceph-mon[32404]: pgmap v3611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5845.719749 secs 2024-07-26T10:18:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:21.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:21 smithi171 ceph-mon[37789]: pgmap v3610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:21.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:21 smithi171 ceph-mon[37789]: pgmap v3611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:21.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:21.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5845.719749 secs 2024-07-26T10:18:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:21 smithi116 ceph-mon[37097]: pgmap v3610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:18:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:21 smithi116 ceph-mon[37097]: pgmap v3611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:18:21.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5845.719749 secs 2024-07-26T10:18:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:23 smithi044 ceph-mon[32404]: pgmap v3612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:18:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:23 smithi116 ceph-mon[37097]: pgmap v3612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:18:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:23 smithi171 ceph-mon[37789]: pgmap v3612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:18:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:25 smithi044 ceph-mon[32404]: pgmap v3613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:25 smithi116 ceph-mon[37097]: pgmap v3613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:25 smithi171 ceph-mon[37789]: pgmap v3613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5850.719860 secs 2024-07-26T10:18:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5850.719860 secs 2024-07-26T10:18:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5850.719860 secs 2024-07-26T10:18:27.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:27.167+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:18:27.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:27.167+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:18:27.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:27.167+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:27.421 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:27 smithi044 ceph-mon[32404]: pgmap v3614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:18:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:27 smithi116 ceph-mon[37097]: pgmap v3614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:18:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:27 smithi171 ceph-mon[37789]: pgmap v3614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:18:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:29 smithi044 ceph-mon[32404]: pgmap v3615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:29.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:29 smithi116 ceph-mon[37097]: pgmap v3615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:29 smithi171 ceph-mon[37789]: pgmap v3615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:18:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:31 smithi044 ceph-mon[32404]: pgmap v3616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:18:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5855.719976 secs 2024-07-26T10:18:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:31 smithi116 ceph-mon[37097]: pgmap v3616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:18:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5855.719976 secs 2024-07-26T10:18:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:31 smithi171 ceph-mon[37789]: pgmap v3616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:18:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5855.719976 secs 2024-07-26T10:18:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:33 smithi044 ceph-mon[32404]: pgmap v3617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:33 smithi116 ceph-mon[37097]: pgmap v3617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:33 smithi171 ceph-mon[37789]: pgmap v3617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:35 smithi044 ceph-mon[32404]: pgmap v3618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:35 smithi116 ceph-mon[37097]: pgmap v3618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:35 smithi171 ceph-mon[37789]: pgmap v3618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5860.720131 secs 2024-07-26T10:18:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5860.720131 secs 2024-07-26T10:18:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5860.720131 secs 2024-07-26T10:18:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:37 smithi044 ceph-mon[32404]: pgmap v3619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:37 smithi116 ceph-mon[37097]: pgmap v3619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:37 smithi171 ceph-mon[37789]: pgmap v3619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:39 smithi116 ceph-mon[37097]: pgmap v3620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:39 smithi171 ceph-mon[37789]: pgmap v3620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:39 smithi044 ceph-mon[32404]: pgmap v3620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:41 smithi171 ceph-mon[37789]: pgmap v3621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5865.720301 secs 2024-07-26T10:18:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:41 smithi044 ceph-mon[32404]: pgmap v3621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5865.720301 secs 2024-07-26T10:18:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:41 smithi116 ceph-mon[37097]: pgmap v3621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5865.720301 secs 2024-07-26T10:18:42.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:42.170+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:18:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:42.170+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:18:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:42.170+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:43 smithi171 ceph-mon[37789]: pgmap v3622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:43 smithi044 ceph-mon[32404]: pgmap v3622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:43 smithi116 ceph-mon[37097]: pgmap v3622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:45.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:45 smithi116 ceph-mon[37097]: pgmap v3623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:45 smithi171 ceph-mon[37789]: pgmap v3623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:45 smithi044 ceph-mon[32404]: pgmap v3623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5870.720446 secs 2024-07-26T10:18:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5870.720446 secs 2024-07-26T10:18:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5870.720446 secs 2024-07-26T10:18:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:47 smithi171 ceph-mon[37789]: pgmap v3624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:47 smithi044 ceph-mon[32404]: pgmap v3624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:47 smithi116 ceph-mon[37097]: pgmap v3624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:49 smithi044 ceph-mon[32404]: pgmap v3625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:49 smithi116 ceph-mon[37097]: pgmap v3625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:49 smithi171 ceph-mon[37789]: pgmap v3625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:18:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5875.720656 secs 2024-07-26T10:18:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5875.720656 secs 2024-07-26T10:18:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5875.720656 secs 2024-07-26T10:18:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:51 smithi044 ceph-mon[32404]: pgmap v3626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:51 smithi116 ceph-mon[37097]: pgmap v3626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:51 smithi171 ceph-mon[37789]: pgmap v3626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:53 smithi044 ceph-mon[32404]: pgmap v3627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:53 smithi116 ceph-mon[37097]: pgmap v3627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:53 smithi171 ceph-mon[37789]: pgmap v3627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:55 smithi044 ceph-mon[32404]: pgmap v3628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5880.720779 secs 2024-07-26T10:18:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:55 smithi116 ceph-mon[37097]: pgmap v3628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5880.720779 secs 2024-07-26T10:18:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:55 smithi171 ceph-mon[37789]: pgmap v3628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:18:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5880.720779 secs 2024-07-26T10:18:57.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:57.180+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:18:57.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:57.180+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:18:57.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:18:57.180+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:57 smithi044 ceph-mon[32404]: pgmap v3629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:57 smithi116 ceph-mon[37097]: pgmap v3629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:18:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:57 smithi171 ceph-mon[37789]: pgmap v3629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:18:59 smithi044 ceph-mon[32404]: pgmap v3630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:18:59 smithi116 ceph-mon[37097]: pgmap v3630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:18:59 smithi171 ceph-mon[37789]: pgmap v3630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5885.720943 secs 2024-07-26T10:19:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5885.720943 secs 2024-07-26T10:19:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5885.720943 secs 2024-07-26T10:19:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:01 smithi044 ceph-mon[32404]: pgmap v3631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:01 smithi116 ceph-mon[37097]: pgmap v3631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:01 smithi171 ceph-mon[37789]: pgmap v3631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:03 smithi044 ceph-mon[32404]: pgmap v3632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:03 smithi116 ceph-mon[37097]: pgmap v3632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:03 smithi171 ceph-mon[37789]: pgmap v3632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:05 smithi044 ceph-mon[32404]: pgmap v3633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5890.721086 secs 2024-07-26T10:19:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:05 smithi116 ceph-mon[37097]: pgmap v3633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5890.721086 secs 2024-07-26T10:19:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:05 smithi171 ceph-mon[37789]: pgmap v3633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5890.721086 secs 2024-07-26T10:19:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:06 smithi044 ceph-mon[32404]: pgmap v3634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:06 smithi116 ceph-mon[37097]: pgmap v3634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:06 smithi171 ceph-mon[37789]: pgmap v3634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:09 smithi044 ceph-mon[32404]: pgmap v3635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:09 smithi116 ceph-mon[37097]: pgmap v3635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:09 smithi171 ceph-mon[37789]: pgmap v3635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:11 smithi044 ceph-mon[32404]: pgmap v3636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:11.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5895.721264 secs 2024-07-26T10:19:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:11 smithi116 ceph-mon[37097]: pgmap v3636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5895.721264 secs 2024-07-26T10:19:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:11 smithi171 ceph-mon[37789]: pgmap v3636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5895.721264 secs 2024-07-26T10:19:12.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:12.183+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:19:12.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:12.183+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:19:12.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:12.183+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:13 smithi044 ceph-mon[32404]: pgmap v3637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:13 smithi116 ceph-mon[37097]: pgmap v3637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:13 smithi171 ceph-mon[37789]: pgmap v3637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:15 smithi044 ceph-mon[32404]: pgmap v3638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:15 smithi116 ceph-mon[37097]: pgmap v3638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:15 smithi171 ceph-mon[37789]: pgmap v3638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5900.721427 secs 2024-07-26T10:19:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5900.721427 secs 2024-07-26T10:19:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5900.721427 secs 2024-07-26T10:19:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:17 smithi044 ceph-mon[32404]: pgmap v3639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:17 smithi116 ceph-mon[37097]: pgmap v3639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:17 smithi171 ceph-mon[37789]: pgmap v3639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:19 smithi044 ceph-mon[32404]: pgmap v3640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:19 smithi116 ceph-mon[37097]: pgmap v3640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:19 smithi171 ceph-mon[37789]: pgmap v3640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:19:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:21 smithi044 ceph-mon[32404]: pgmap v3641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5905.721526 secs 2024-07-26T10:19:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:21 smithi116 ceph-mon[37097]: pgmap v3641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5905.721526 secs 2024-07-26T10:19:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:21 smithi171 ceph-mon[37789]: pgmap v3641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5905.721526 secs 2024-07-26T10:19:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:22 smithi116 ceph-mon[37097]: pgmap v3642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:22 smithi116 ceph-mon[37097]: pgmap v3643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:19:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:19:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:22 smithi171 ceph-mon[37789]: pgmap v3642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:22 smithi171 ceph-mon[37789]: pgmap v3643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:19:22.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:19:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:22 smithi044 ceph-mon[32404]: pgmap v3642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:22 smithi044 ceph-mon[32404]: pgmap v3643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:19:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:19:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:24 smithi044 ceph-mon[32404]: pgmap v3644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:24 smithi116 ceph-mon[37097]: pgmap v3644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:24 smithi171 ceph-mon[37789]: pgmap v3644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5910.721685 secs 2024-07-26T10:19:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5910.721685 secs 2024-07-26T10:19:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5910.721685 secs 2024-07-26T10:19:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:26 smithi044 ceph-mon[32404]: pgmap v3645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:26 smithi116 ceph-mon[37097]: pgmap v3645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:26 smithi171 ceph-mon[37789]: pgmap v3645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:27.192+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:19:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:27.192+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:19:27.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:27.192+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:28 smithi044 ceph-mon[32404]: pgmap v3646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:19:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:28 smithi116 ceph-mon[37097]: pgmap v3646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:19:28.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:28 smithi171 ceph-mon[37789]: pgmap v3646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:19:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:30 smithi044 ceph-mon[32404]: pgmap v3647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5915.721786 secs 2024-07-26T10:19:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:30 smithi116 ceph-mon[37097]: pgmap v3647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5915.721786 secs 2024-07-26T10:19:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:30 smithi171 ceph-mon[37789]: pgmap v3647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:19:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5915.721786 secs 2024-07-26T10:19:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:32 smithi044 ceph-mon[32404]: pgmap v3648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:32 smithi116 ceph-mon[37097]: pgmap v3648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:32 smithi171 ceph-mon[37789]: pgmap v3648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:19:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:34 smithi044 ceph-mon[32404]: pgmap v3649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:34 smithi116 ceph-mon[37097]: pgmap v3649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:34 smithi171 ceph-mon[37789]: pgmap v3649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5920.721929 secs 2024-07-26T10:19:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5920.721929 secs 2024-07-26T10:19:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5920.721929 secs 2024-07-26T10:19:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:36 smithi116 ceph-mon[37097]: pgmap v3650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:36 smithi171 ceph-mon[37789]: pgmap v3650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:36 smithi044 ceph-mon[32404]: pgmap v3650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:38 smithi171 ceph-mon[37789]: pgmap v3651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:38 smithi044 ceph-mon[32404]: pgmap v3651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:38 smithi116 ceph-mon[37097]: pgmap v3651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:40 smithi171 ceph-mon[37789]: pgmap v3652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5925.722094 secs 2024-07-26T10:19:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:40 smithi044 ceph-mon[32404]: pgmap v3652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5925.722094 secs 2024-07-26T10:19:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:40 smithi116 ceph-mon[37097]: pgmap v3652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5925.722094 secs 2024-07-26T10:19:42.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:42.196+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:19:42.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:42.196+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:19:42.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:42.196+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:42 smithi171 ceph-mon[37789]: pgmap v3653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:42 smithi044 ceph-mon[32404]: pgmap v3653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:42 smithi116 ceph-mon[37097]: pgmap v3653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:44 smithi171 ceph-mon[37789]: pgmap v3654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:44 smithi044 ceph-mon[32404]: pgmap v3654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:44 smithi116 ceph-mon[37097]: pgmap v3654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5930.722239 secs 2024-07-26T10:19:45.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5930.722239 secs 2024-07-26T10:19:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5930.722239 secs 2024-07-26T10:19:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:46 smithi171 ceph-mon[37789]: pgmap v3655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:46 smithi044 ceph-mon[32404]: pgmap v3655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:46 smithi116 ceph-mon[37097]: pgmap v3655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:48 smithi171 ceph-mon[37789]: pgmap v3656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:48 smithi044 ceph-mon[32404]: pgmap v3656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:48 smithi116 ceph-mon[37097]: pgmap v3656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:50 smithi171 ceph-mon[37789]: pgmap v3657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5935.722388 secs 2024-07-26T10:19:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:50 smithi044 ceph-mon[32404]: pgmap v3657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5935.722388 secs 2024-07-26T10:19:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:50 smithi116 ceph-mon[37097]: pgmap v3657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:19:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5935.722388 secs 2024-07-26T10:19:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:52 smithi171 ceph-mon[37789]: pgmap v3658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:52 smithi044 ceph-mon[32404]: pgmap v3658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:53.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:52 smithi116 ceph-mon[37097]: pgmap v3658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:19:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:54 smithi171 ceph-mon[37789]: pgmap v3659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:54 smithi044 ceph-mon[32404]: pgmap v3659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:54 smithi116 ceph-mon[37097]: pgmap v3659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:19:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5940.722547 secs 2024-07-26T10:19:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5940.722547 secs 2024-07-26T10:19:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5940.722547 secs 2024-07-26T10:19:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:56 smithi044 ceph-mon[32404]: pgmap v3660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:57.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:56 smithi116 ceph-mon[37097]: pgmap v3660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:56 smithi171 ceph-mon[37789]: pgmap v3660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:19:57.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:57.204+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:19:57.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:57.204+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:19:57.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:19:57.204+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:19:58 smithi044 ceph-mon[32404]: pgmap v3661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:19:58 smithi116 ceph-mon[37097]: pgmap v3661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:19:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:19:58 smithi171 ceph-mon[37789]: pgmap v3661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:00 smithi044 ceph-mon[32404]: pgmap v3662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:20:01.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5945.722714 secs 2024-07-26T10:20:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:00 smithi116 ceph-mon[37097]: pgmap v3662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:20:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5945.722714 secs 2024-07-26T10:20:01.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:00 smithi171 ceph-mon[37789]: pgmap v3662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:01.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:20:01.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5945.722714 secs 2024-07-26T10:20:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:02 smithi044 ceph-mon[32404]: pgmap v3663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:02 smithi116 ceph-mon[37097]: pgmap v3663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:02 smithi171 ceph-mon[37789]: pgmap v3663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:04 smithi044 ceph-mon[32404]: pgmap v3664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:04 smithi116 ceph-mon[37097]: pgmap v3664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:04 smithi171 ceph-mon[37789]: pgmap v3664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5950.722875 secs 2024-07-26T10:20:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5950.722875 secs 2024-07-26T10:20:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5950.722875 secs 2024-07-26T10:20:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:06 smithi044 ceph-mon[32404]: pgmap v3665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:06 smithi116 ceph-mon[37097]: pgmap v3665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:06 smithi171 ceph-mon[37789]: pgmap v3665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:08 smithi044 ceph-mon[32404]: pgmap v3666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:08 smithi116 ceph-mon[37097]: pgmap v3666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:08 smithi171 ceph-mon[37789]: pgmap v3666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:10 smithi044 ceph-mon[32404]: pgmap v3667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5955.723038 secs 2024-07-26T10:20:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:10 smithi116 ceph-mon[37097]: pgmap v3667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5955.723038 secs 2024-07-26T10:20:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:10 smithi171 ceph-mon[37789]: pgmap v3667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5955.723038 secs 2024-07-26T10:20:12.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:12.208+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:20:12.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:12.208+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:20:12.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:12.208+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:12 smithi044 ceph-mon[32404]: pgmap v3668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:12 smithi116 ceph-mon[37097]: pgmap v3668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:12 smithi171 ceph-mon[37789]: pgmap v3668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:14 smithi044 ceph-mon[32404]: pgmap v3669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:14 smithi116 ceph-mon[37097]: pgmap v3669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:14 smithi171 ceph-mon[37789]: pgmap v3669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:15.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.731+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:20:15.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.731+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:20:15.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.731+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:20:15.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.732+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:20:15.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.732+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:20:15.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.732+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:20:15.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.733+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:20:15.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:15.733+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:20:15.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5960.723186 secs 2024-07-26T10:20:15.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5960.723186 secs 2024-07-26T10:20:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5960.723186 secs 2024-07-26T10:20:17.004 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:16 smithi171 ceph-mon[37789]: pgmap v3670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:16 smithi044 ceph-mon[32404]: pgmap v3670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:16 smithi116 ceph-mon[37097]: pgmap v3670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:18 smithi044 ceph-mon[32404]: pgmap v3671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:18 smithi116 ceph-mon[37097]: pgmap v3671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:19.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:18 smithi171 ceph-mon[37789]: pgmap v3671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:20 smithi044 ceph-mon[32404]: pgmap v3672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5965.723351 secs 2024-07-26T10:20:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:20 smithi116 ceph-mon[37097]: pgmap v3672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5965.723351 secs 2024-07-26T10:20:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:20 smithi171 ceph-mon[37789]: pgmap v3672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:20:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5965.723351 secs 2024-07-26T10:20:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:22 smithi044 ceph-mon[32404]: pgmap v3673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:22 smithi116 ceph-mon[37097]: pgmap v3673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:22 smithi171 ceph-mon[37789]: pgmap v3673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:23 smithi044 ceph-mon[32404]: pgmap v3674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:20:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:23 smithi044 ceph-mon[32404]: pgmap v3675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:20:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:20:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:23 smithi116 ceph-mon[37097]: pgmap v3674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:20:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:23 smithi116 ceph-mon[37097]: pgmap v3675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:20:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:20:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:23 smithi171 ceph-mon[37789]: pgmap v3674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:20:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:23 smithi171 ceph-mon[37789]: pgmap v3675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:20:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:20:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:25 smithi116 ceph-mon[37097]: pgmap v3676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5970.723491 secs 2024-07-26T10:20:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:25 smithi171 ceph-mon[37789]: pgmap v3676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:26.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5970.723491 secs 2024-07-26T10:20:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:25 smithi044 ceph-mon[32404]: pgmap v3676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5970.723491 secs 2024-07-26T10:20:27.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:27.217+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:20:27.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:27.217+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:20:27.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:27.217+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:27 smithi116 ceph-mon[37097]: pgmap v3677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:20:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:27 smithi171 ceph-mon[37789]: pgmap v3677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:20:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:27 smithi044 ceph-mon[32404]: pgmap v3677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:20:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:29 smithi171 ceph-mon[37789]: pgmap v3678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:29 smithi044 ceph-mon[32404]: pgmap v3678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:29 smithi116 ceph-mon[37097]: pgmap v3678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:20:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5975.723751 secs 2024-07-26T10:20:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5975.723751 secs 2024-07-26T10:20:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5975.723751 secs 2024-07-26T10:20:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:31 smithi171 ceph-mon[37789]: pgmap v3679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:20:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:31 smithi044 ceph-mon[32404]: pgmap v3679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:20:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:31 smithi116 ceph-mon[37097]: pgmap v3679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:20:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:33 smithi171 ceph-mon[37789]: pgmap v3680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:20:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:33 smithi044 ceph-mon[32404]: pgmap v3680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:20:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:33 smithi116 ceph-mon[37097]: pgmap v3680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:20:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:35 smithi171 ceph-mon[37789]: pgmap v3681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:20:36.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5980.723915 secs 2024-07-26T10:20:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:35 smithi044 ceph-mon[32404]: pgmap v3681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:20:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5980.723915 secs 2024-07-26T10:20:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:35 smithi116 ceph-mon[37097]: pgmap v3681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:20:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5980.723915 secs 2024-07-26T10:20:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:37 smithi171 ceph-mon[37789]: pgmap v3682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:37 smithi044 ceph-mon[32404]: pgmap v3682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:37 smithi116 ceph-mon[37097]: pgmap v3682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:39 smithi044 ceph-mon[32404]: pgmap v3683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:20:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:39 smithi116 ceph-mon[37097]: pgmap v3683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:20:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:39 smithi171 ceph-mon[37789]: pgmap v3683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:20:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5985.724061 secs 2024-07-26T10:20:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5985.724061 secs 2024-07-26T10:20:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5985.724061 secs 2024-07-26T10:20:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:41 smithi044 ceph-mon[32404]: pgmap v3684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:20:42.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:42.222+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:20:42.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:42.222+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:20:42.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:42.222+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:41 smithi116 ceph-mon[37097]: pgmap v3684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:20:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:41 smithi171 ceph-mon[37789]: pgmap v3684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:20:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:43 smithi044 ceph-mon[32404]: pgmap v3685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:20:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:43 smithi116 ceph-mon[37097]: pgmap v3685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:20:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:43 smithi171 ceph-mon[37789]: pgmap v3685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:20:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:45 smithi044 ceph-mon[32404]: pgmap v3686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5990.724196 secs 2024-07-26T10:20:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:45 smithi116 ceph-mon[37097]: pgmap v3686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5990.724196 secs 2024-07-26T10:20:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:45 smithi171 ceph-mon[37789]: pgmap v3686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:20:46.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5990.724196 secs 2024-07-26T10:20:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:46 smithi044 ceph-mon[32404]: pgmap v3687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:46 smithi116 ceph-mon[37097]: pgmap v3687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:46 smithi171 ceph-mon[37789]: pgmap v3687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:49 smithi171 ceph-mon[37789]: pgmap v3688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:49 smithi044 ceph-mon[32404]: pgmap v3688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:49 smithi116 ceph-mon[37097]: pgmap v3688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:20:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 5995.724345 secs 2024-07-26T10:20:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 5995.724345 secs 2024-07-26T10:20:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 5995.724345 secs 2024-07-26T10:20:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:51 smithi171 ceph-mon[37789]: pgmap v3689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:51 smithi044 ceph-mon[32404]: pgmap v3689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:51 smithi116 ceph-mon[37097]: pgmap v3689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:20:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:53 smithi171 ceph-mon[37789]: pgmap v3690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:20:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:53 smithi044 ceph-mon[32404]: pgmap v3690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:20:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:53 smithi116 ceph-mon[37097]: pgmap v3690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:20:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:55 smithi171 ceph-mon[37789]: pgmap v3691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:20:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6000.724490 secs 2024-07-26T10:20:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:55 smithi044 ceph-mon[32404]: pgmap v3691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:20:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6000.724490 secs 2024-07-26T10:20:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:55 smithi116 ceph-mon[37097]: pgmap v3691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:20:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6000.724490 secs 2024-07-26T10:20:57.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:57.231+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:20:57.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:57.231+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:20:57.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:20:57.231+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:57 smithi171 ceph-mon[37789]: pgmap v3692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:57 smithi044 ceph-mon[32404]: pgmap v3692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:20:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:57 smithi116 ceph-mon[37097]: pgmap v3692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:20:59 smithi171 ceph-mon[37789]: pgmap v3693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:21:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:20:59 smithi044 ceph-mon[32404]: pgmap v3693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:21:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:20:59 smithi116 ceph-mon[37097]: pgmap v3693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:21:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6005.724682 secs 2024-07-26T10:21:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6005.724682 secs 2024-07-26T10:21:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6005.724682 secs 2024-07-26T10:21:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:01 smithi171 ceph-mon[37789]: pgmap v3694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:21:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:01 smithi044 ceph-mon[32404]: pgmap v3694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:21:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:01 smithi116 ceph-mon[37097]: pgmap v3694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:21:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:03 smithi044 ceph-mon[32404]: pgmap v3695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:03 smithi116 ceph-mon[37097]: pgmap v3695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:03 smithi171 ceph-mon[37789]: pgmap v3695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:21:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:05 smithi044 ceph-mon[32404]: pgmap v3696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6010.724836 secs 2024-07-26T10:21:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:05 smithi116 ceph-mon[37097]: pgmap v3696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6010.724836 secs 2024-07-26T10:21:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:05 smithi171 ceph-mon[37789]: pgmap v3696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6010.724836 secs 2024-07-26T10:21:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:07 smithi044 ceph-mon[32404]: pgmap v3697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:08.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:07 smithi116 ceph-mon[37097]: pgmap v3697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:07 smithi171 ceph-mon[37789]: pgmap v3697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:09 smithi044 ceph-mon[32404]: pgmap v3698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:10.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:09 smithi116 ceph-mon[37097]: pgmap v3698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:09 smithi171 ceph-mon[37789]: pgmap v3698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6015.724938 secs 2024-07-26T10:21:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6015.724938 secs 2024-07-26T10:21:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6015.724938 secs 2024-07-26T10:21:12.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:11 smithi044 ceph-mon[32404]: pgmap v3699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:12.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:12.235+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:21:12.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:12.235+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:21:12.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:12.235+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:11 smithi116 ceph-mon[37097]: pgmap v3699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:11 smithi171 ceph-mon[37789]: pgmap v3699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:13 smithi044 ceph-mon[32404]: pgmap v3700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:13 smithi116 ceph-mon[37097]: pgmap v3700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:13 smithi171 ceph-mon[37789]: pgmap v3700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:16.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:15 smithi171 ceph-mon[37789]: pgmap v3701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:16.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6020.725081 secs 2024-07-26T10:21:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:15 smithi044 ceph-mon[32404]: pgmap v3701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6020.725081 secs 2024-07-26T10:21:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:15 smithi116 ceph-mon[37097]: pgmap v3701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:21:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6020.725081 secs 2024-07-26T10:21:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:17 smithi044 ceph-mon[32404]: pgmap v3702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:17 smithi116 ceph-mon[37097]: pgmap v3702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:17 smithi171 ceph-mon[37789]: pgmap v3702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:21:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:19 smithi044 ceph-mon[32404]: pgmap v3703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:19 smithi116 ceph-mon[37097]: pgmap v3703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:19 smithi171 ceph-mon[37789]: pgmap v3703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:21:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6025.725244 secs 2024-07-26T10:21:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6025.725244 secs 2024-07-26T10:21:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6025.725244 secs 2024-07-26T10:21:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:21 smithi044 ceph-mon[32404]: pgmap v3704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:21 smithi116 ceph-mon[37097]: pgmap v3704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:21 smithi171 ceph-mon[37789]: pgmap v3704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:23 smithi044 ceph-mon[32404]: pgmap v3705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:23 smithi116 ceph-mon[37097]: pgmap v3705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:24.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:23 smithi171 ceph-mon[37789]: pgmap v3705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:25.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:25.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:21:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:25.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:25.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:21:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:21:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:25 smithi044 ceph-mon[32404]: pgmap v3706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:21:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:25 smithi044 ceph-mon[32404]: pgmap v3707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6030.725398 secs 2024-07-26T10:21:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:25 smithi116 ceph-mon[37097]: pgmap v3706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:21:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:25 smithi116 ceph-mon[37097]: pgmap v3707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6030.725398 secs 2024-07-26T10:21:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:25 smithi171 ceph-mon[37789]: pgmap v3706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:21:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:25 smithi171 ceph-mon[37789]: pgmap v3707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6030.725398 secs 2024-07-26T10:21:27.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:27.244+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:21:27.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:27.244+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:21:27.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:27.244+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:27 smithi044 ceph-mon[32404]: pgmap v3708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:21:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:27 smithi116 ceph-mon[37097]: pgmap v3708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:21:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:27 smithi171 ceph-mon[37789]: pgmap v3708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:21:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:29 smithi044 ceph-mon[32404]: pgmap v3709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:29 smithi116 ceph-mon[37097]: pgmap v3709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:29 smithi171 ceph-mon[37789]: pgmap v3709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:30 smithi044 ceph-mon[32404]: pgmap v3710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6035.725548 secs 2024-07-26T10:21:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:30 smithi116 ceph-mon[37097]: pgmap v3710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6035.725548 secs 2024-07-26T10:21:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:30 smithi171 ceph-mon[37789]: pgmap v3710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6035.725548 secs 2024-07-26T10:21:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:33 smithi171 ceph-mon[37789]: pgmap v3711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:33 smithi044 ceph-mon[32404]: pgmap v3711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:33 smithi116 ceph-mon[37097]: pgmap v3711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:21:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:35 smithi171 ceph-mon[37789]: pgmap v3712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:21:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:35 smithi044 ceph-mon[32404]: pgmap v3712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:21:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:35 smithi116 ceph-mon[37097]: pgmap v3712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:21:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6040.725687 secs 2024-07-26T10:21:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6040.725687 secs 2024-07-26T10:21:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6040.725687 secs 2024-07-26T10:21:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:37 smithi171 ceph-mon[37789]: pgmap v3713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:37 smithi044 ceph-mon[32404]: pgmap v3713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:37 smithi116 ceph-mon[37097]: pgmap v3713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:39 smithi171 ceph-mon[37789]: pgmap v3714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:39 smithi044 ceph-mon[32404]: pgmap v3714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:39 smithi116 ceph-mon[37097]: pgmap v3714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:41 smithi171 ceph-mon[37789]: pgmap v3715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6045.725849 secs 2024-07-26T10:21:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:41 smithi044 ceph-mon[32404]: pgmap v3715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6045.725849 secs 2024-07-26T10:21:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:41 smithi116 ceph-mon[37097]: pgmap v3715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6045.725849 secs 2024-07-26T10:21:42.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:42.245+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:21:42.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:42.245+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:21:42.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:42.245+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:43 smithi171 ceph-mon[37789]: pgmap v3716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:43 smithi044 ceph-mon[32404]: pgmap v3716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:43 smithi116 ceph-mon[37097]: pgmap v3716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:45 smithi171 ceph-mon[37789]: pgmap v3717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:45 smithi044 ceph-mon[32404]: pgmap v3717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:45 smithi116 ceph-mon[37097]: pgmap v3717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6050.725995 secs 2024-07-26T10:21:46.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6050.725995 secs 2024-07-26T10:21:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6050.725995 secs 2024-07-26T10:21:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:47 smithi044 ceph-mon[32404]: pgmap v3718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:47 smithi116 ceph-mon[37097]: pgmap v3718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:47 smithi171 ceph-mon[37789]: pgmap v3718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:49 smithi044 ceph-mon[32404]: pgmap v3719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:49 smithi116 ceph-mon[37097]: pgmap v3719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:49 smithi171 ceph-mon[37789]: pgmap v3719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:51 smithi044 ceph-mon[32404]: pgmap v3720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6055.726165 secs 2024-07-26T10:21:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:51 smithi116 ceph-mon[37097]: pgmap v3720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6055.726165 secs 2024-07-26T10:21:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:51 smithi171 ceph-mon[37789]: pgmap v3720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6055.726165 secs 2024-07-26T10:21:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:53 smithi044 ceph-mon[32404]: pgmap v3721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:53 smithi116 ceph-mon[37097]: pgmap v3721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:53 smithi171 ceph-mon[37789]: pgmap v3721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:55 smithi044 ceph-mon[32404]: pgmap v3722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:55 smithi116 ceph-mon[37097]: pgmap v3722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:55 smithi171 ceph-mon[37789]: pgmap v3722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:21:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6060.726338 secs 2024-07-26T10:21:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6060.726338 secs 2024-07-26T10:21:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6060.726338 secs 2024-07-26T10:21:57.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:57.253+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:21:57.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:57.253+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:21:57.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:21:57.253+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:57.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:57 smithi044 ceph-mon[32404]: pgmap v3723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:57 smithi116 ceph-mon[37097]: pgmap v3723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:57 smithi171 ceph-mon[37789]: pgmap v3723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:21:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:21:59 smithi044 ceph-mon[32404]: pgmap v3724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:21:59 smithi116 ceph-mon[37097]: pgmap v3724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:21:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:21:59 smithi171 ceph-mon[37789]: pgmap v3724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:01 smithi044 ceph-mon[32404]: pgmap v3725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6065.726499 secs 2024-07-26T10:22:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:01 smithi116 ceph-mon[37097]: pgmap v3725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6065.726499 secs 2024-07-26T10:22:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:01 smithi171 ceph-mon[37789]: pgmap v3725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:01.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6065.726499 secs 2024-07-26T10:22:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:03 smithi044 ceph-mon[32404]: pgmap v3726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:03 smithi116 ceph-mon[37097]: pgmap v3726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:03 smithi171 ceph-mon[37789]: pgmap v3726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:05 smithi044 ceph-mon[32404]: pgmap v3727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:05 smithi116 ceph-mon[37097]: pgmap v3727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:05 smithi171 ceph-mon[37789]: pgmap v3727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6070.726663 secs 2024-07-26T10:22:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6070.726663 secs 2024-07-26T10:22:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6070.726663 secs 2024-07-26T10:22:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:07 smithi044 ceph-mon[32404]: pgmap v3728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:07 smithi116 ceph-mon[37097]: pgmap v3728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:07 smithi171 ceph-mon[37789]: pgmap v3728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:09 smithi044 ceph-mon[32404]: pgmap v3729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:09 smithi116 ceph-mon[37097]: pgmap v3729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:09 smithi171 ceph-mon[37789]: pgmap v3729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:11 smithi044 ceph-mon[32404]: pgmap v3730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:11.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6075.726820 secs 2024-07-26T10:22:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:11 smithi116 ceph-mon[37097]: pgmap v3730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6075.726820 secs 2024-07-26T10:22:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:11 smithi171 ceph-mon[37789]: pgmap v3730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6075.726820 secs 2024-07-26T10:22:12.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:12.258+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:22:12.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:12.258+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:22:12.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:12.258+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:13 smithi044 ceph-mon[32404]: pgmap v3731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:13 smithi116 ceph-mon[37097]: pgmap v3731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:13 smithi171 ceph-mon[37789]: pgmap v3731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:15 smithi044 ceph-mon[32404]: pgmap v3732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:15 smithi116 ceph-mon[37097]: pgmap v3732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:15 smithi171 ceph-mon[37789]: pgmap v3732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:16.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6080.726976 secs 2024-07-26T10:22:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6080.726976 secs 2024-07-26T10:22:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6080.726976 secs 2024-07-26T10:22:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:17 smithi044 ceph-mon[32404]: pgmap v3733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:17 smithi116 ceph-mon[37097]: pgmap v3733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:17 smithi171 ceph-mon[37789]: pgmap v3733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:19 smithi044 ceph-mon[32404]: pgmap v3734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:19 smithi116 ceph-mon[37097]: pgmap v3734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:19 smithi171 ceph-mon[37789]: pgmap v3734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:21 smithi044 ceph-mon[32404]: pgmap v3735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6085.727107 secs 2024-07-26T10:22:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:21 smithi116 ceph-mon[37097]: pgmap v3735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6085.727107 secs 2024-07-26T10:22:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:21 smithi171 ceph-mon[37789]: pgmap v3735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6085.727107 secs 2024-07-26T10:22:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:23 smithi044 ceph-mon[32404]: pgmap v3736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:23 smithi116 ceph-mon[37097]: pgmap v3736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:23 smithi171 ceph-mon[37789]: pgmap v3736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:25 smithi044 ceph-mon[32404]: pgmap v3737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:25 smithi116 ceph-mon[37097]: pgmap v3737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:25 smithi171 ceph-mon[37789]: pgmap v3737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:26 smithi044 ceph-mon[32404]: pgmap v3738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:26 smithi044 ceph-mon[32404]: pgmap v3739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:22:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6090.727200 secs 2024-07-26T10:22:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:26 smithi116 ceph-mon[37097]: pgmap v3738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:22:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:26 smithi116 ceph-mon[37097]: pgmap v3739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:22:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6090.727200 secs 2024-07-26T10:22:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:26 smithi171 ceph-mon[37789]: pgmap v3738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:22:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:26 smithi171 ceph-mon[37789]: pgmap v3739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:22:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6090.727200 secs 2024-07-26T10:22:27.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:27.265+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:22:27.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:27.265+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:27.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:27.265+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:22:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:28 smithi171 ceph-mon[37789]: pgmap v3740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:22:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:28 smithi044 ceph-mon[32404]: pgmap v3740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:22:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:28 smithi116 ceph-mon[37097]: pgmap v3740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:22:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:30 smithi171 ceph-mon[37789]: pgmap v3741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:30 smithi044 ceph-mon[32404]: pgmap v3741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:30 smithi116 ceph-mon[37097]: pgmap v3741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6095.727361 secs 2024-07-26T10:22:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6095.727361 secs 2024-07-26T10:22:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6095.727361 secs 2024-07-26T10:22:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:32 smithi044 ceph-mon[32404]: pgmap v3742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:22:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:32 smithi116 ceph-mon[37097]: pgmap v3742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:22:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:32 smithi171 ceph-mon[37789]: pgmap v3742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:22:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:34 smithi044 ceph-mon[32404]: pgmap v3743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:34 smithi116 ceph-mon[37097]: pgmap v3743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:34 smithi171 ceph-mon[37789]: pgmap v3743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:22:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:36 smithi044 ceph-mon[32404]: pgmap v3744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:22:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6100.727505 secs 2024-07-26T10:22:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:36 smithi116 ceph-mon[37097]: pgmap v3744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:22:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6100.727505 secs 2024-07-26T10:22:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:36 smithi171 ceph-mon[37789]: pgmap v3744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:22:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6100.727505 secs 2024-07-26T10:22:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:38 smithi044 ceph-mon[32404]: pgmap v3745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:38 smithi116 ceph-mon[37097]: pgmap v3745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:38 smithi171 ceph-mon[37789]: pgmap v3745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:40 smithi044 ceph-mon[32404]: pgmap v3746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:40 smithi116 ceph-mon[37097]: pgmap v3746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:40 smithi171 ceph-mon[37789]: pgmap v3746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6105.727612 secs 2024-07-26T10:22:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6105.727612 secs 2024-07-26T10:22:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6105.727612 secs 2024-07-26T10:22:42.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:42.269+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:22:42.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:42.269+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:42.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:42.269+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:22:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:42 smithi044 ceph-mon[32404]: pgmap v3747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:42 smithi116 ceph-mon[37097]: pgmap v3747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:42 smithi171 ceph-mon[37789]: pgmap v3747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:44 smithi044 ceph-mon[32404]: pgmap v3748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:44 smithi116 ceph-mon[37097]: pgmap v3748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:44 smithi171 ceph-mon[37789]: pgmap v3748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:46 smithi044 ceph-mon[32404]: pgmap v3749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6110.727777 secs 2024-07-26T10:22:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:46 smithi116 ceph-mon[37097]: pgmap v3749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6110.727777 secs 2024-07-26T10:22:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:46 smithi171 ceph-mon[37789]: pgmap v3749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6110.727777 secs 2024-07-26T10:22:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:48 smithi044 ceph-mon[32404]: pgmap v3750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:48 smithi116 ceph-mon[37097]: pgmap v3750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:48 smithi171 ceph-mon[37789]: pgmap v3750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:50 smithi044 ceph-mon[32404]: pgmap v3751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:50 smithi116 ceph-mon[37097]: pgmap v3751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:50 smithi171 ceph-mon[37789]: pgmap v3751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:22:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6115.727946 secs 2024-07-26T10:22:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6115.727946 secs 2024-07-26T10:22:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6115.727946 secs 2024-07-26T10:22:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:52 smithi044 ceph-mon[32404]: pgmap v3752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:52 smithi116 ceph-mon[37097]: pgmap v3752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:52 smithi171 ceph-mon[37789]: pgmap v3752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:54 smithi044 ceph-mon[32404]: pgmap v3753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:54 smithi116 ceph-mon[37097]: pgmap v3753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:54 smithi171 ceph-mon[37789]: pgmap v3753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:22:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:56 smithi044 ceph-mon[32404]: pgmap v3754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6120.728104 secs 2024-07-26T10:22:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:56 smithi116 ceph-mon[37097]: pgmap v3754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6120.728104 secs 2024-07-26T10:22:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:56 smithi171 ceph-mon[37789]: pgmap v3754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:22:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6120.728104 secs 2024-07-26T10:22:57.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:57.275+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:22:57.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:57.275+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:57.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:22:57.275+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:22:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:22:58 smithi044 ceph-mon[32404]: pgmap v3755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:22:58 smithi116 ceph-mon[37097]: pgmap v3755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:22:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:22:58 smithi171 ceph-mon[37789]: pgmap v3755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:00 smithi044 ceph-mon[32404]: pgmap v3756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:00 smithi116 ceph-mon[37097]: pgmap v3756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:00 smithi171 ceph-mon[37789]: pgmap v3756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6125.728281 secs 2024-07-26T10:23:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6125.728281 secs 2024-07-26T10:23:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6125.728281 secs 2024-07-26T10:23:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:02 smithi044 ceph-mon[32404]: pgmap v3757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:02 smithi116 ceph-mon[37097]: pgmap v3757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:02 smithi171 ceph-mon[37789]: pgmap v3757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:04 smithi044 ceph-mon[32404]: pgmap v3758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:04 smithi116 ceph-mon[37097]: pgmap v3758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:04 smithi171 ceph-mon[37789]: pgmap v3758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:06 smithi044 ceph-mon[32404]: pgmap v3759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6130.728438 secs 2024-07-26T10:23:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:06 smithi116 ceph-mon[37097]: pgmap v3759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6130.728438 secs 2024-07-26T10:23:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:06 smithi171 ceph-mon[37789]: pgmap v3759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6130.728438 secs 2024-07-26T10:23:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:08 smithi044 ceph-mon[32404]: pgmap v3760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:08 smithi116 ceph-mon[37097]: pgmap v3760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:08 smithi171 ceph-mon[37789]: pgmap v3760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:10 smithi044 ceph-mon[32404]: pgmap v3761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:10 smithi116 ceph-mon[37097]: pgmap v3761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:10 smithi171 ceph-mon[37789]: pgmap v3761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6135.728601 secs 2024-07-26T10:23:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6135.728601 secs 2024-07-26T10:23:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6135.728601 secs 2024-07-26T10:23:12.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:12.279+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:23:12.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:12.279+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:12.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:12.279+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:23:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:12 smithi044 ceph-mon[32404]: pgmap v3762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:12 smithi116 ceph-mon[37097]: pgmap v3762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:12 smithi171 ceph-mon[37789]: pgmap v3762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:14 smithi044 ceph-mon[32404]: pgmap v3763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:14 smithi116 ceph-mon[37097]: pgmap v3763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:14 smithi171 ceph-mon[37789]: pgmap v3763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:15.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:15.764+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:23:15.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:15.764+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:23:15.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:15.765+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:23:15.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:15.765+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:23:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6140.728724 secs 2024-07-26T10:23:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6140.728724 secs 2024-07-26T10:23:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6140.728724 secs 2024-07-26T10:23:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:16 smithi116 ceph-mon[37097]: pgmap v3764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:16 smithi171 ceph-mon[37789]: pgmap v3764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:16.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:16 smithi044 ceph-mon[32404]: pgmap v3764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:18 smithi171 ceph-mon[37789]: pgmap v3765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:18 smithi044 ceph-mon[32404]: pgmap v3765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:18 smithi116 ceph-mon[37097]: pgmap v3765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:20 smithi171 ceph-mon[37789]: pgmap v3766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6145.728906 secs 2024-07-26T10:23:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:20 smithi044 ceph-mon[32404]: pgmap v3766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:21.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6145.728906 secs 2024-07-26T10:23:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:20 smithi116 ceph-mon[37097]: pgmap v3766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6145.728906 secs 2024-07-26T10:23:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:22 smithi171 ceph-mon[37789]: pgmap v3767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:22 smithi044 ceph-mon[32404]: pgmap v3767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:22 smithi116 ceph-mon[37097]: pgmap v3767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:24 smithi171 ceph-mon[37789]: pgmap v3768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:24 smithi044 ceph-mon[32404]: pgmap v3768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:24 smithi116 ceph-mon[37097]: pgmap v3768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:23:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6150.729044 secs 2024-07-26T10:23:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6150.729044 secs 2024-07-26T10:23:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6150.729044 secs 2024-07-26T10:23:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:26 smithi171 ceph-mon[37789]: pgmap v3769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:23:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:26 smithi044 ceph-mon[32404]: pgmap v3769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:23:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:26 smithi116 ceph-mon[37097]: pgmap v3769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:23:27.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:27.288+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:23:27.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:27.288+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:23:27.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:27.288+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:27 smithi171 ceph-mon[37789]: pgmap v3770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:27 smithi171 ceph-mon[37789]: pgmap v3771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:23:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:27 smithi044 ceph-mon[32404]: pgmap v3770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:27 smithi044 ceph-mon[32404]: pgmap v3771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:23:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:27 smithi116 ceph-mon[37097]: pgmap v3770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:23:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:27 smithi116 ceph-mon[37097]: pgmap v3771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:23:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:29 smithi171 ceph-mon[37789]: pgmap v3772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:29 smithi044 ceph-mon[32404]: pgmap v3772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:29 smithi116 ceph-mon[37097]: pgmap v3772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6155.729195 secs 2024-07-26T10:23:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6155.729195 secs 2024-07-26T10:23:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6155.729195 secs 2024-07-26T10:23:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:31 smithi171 ceph-mon[37789]: pgmap v3773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:31 smithi044 ceph-mon[32404]: pgmap v3773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:31 smithi116 ceph-mon[37097]: pgmap v3773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:33 smithi044 ceph-mon[32404]: pgmap v3774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:33 smithi116 ceph-mon[37097]: pgmap v3774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:33 smithi171 ceph-mon[37789]: pgmap v3774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:35 smithi044 ceph-mon[32404]: pgmap v3775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6160.729368 secs 2024-07-26T10:23:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:35 smithi116 ceph-mon[37097]: pgmap v3775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6160.729368 secs 2024-07-26T10:23:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:35 smithi171 ceph-mon[37789]: pgmap v3775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:23:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6160.729368 secs 2024-07-26T10:23:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:37 smithi044 ceph-mon[32404]: pgmap v3776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:23:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:37 smithi116 ceph-mon[37097]: pgmap v3776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:23:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:37 smithi171 ceph-mon[37789]: pgmap v3776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:23:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:39 smithi044 ceph-mon[32404]: pgmap v3777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:39 smithi116 ceph-mon[37097]: pgmap v3777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:39 smithi171 ceph-mon[37789]: pgmap v3777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6165.729517 secs 2024-07-26T10:23:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6165.729517 secs 2024-07-26T10:23:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6165.729517 secs 2024-07-26T10:23:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:41 smithi044 ceph-mon[32404]: pgmap v3778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:41 smithi116 ceph-mon[37097]: pgmap v3778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:41 smithi171 ceph-mon[37789]: pgmap v3778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:42.292+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:23:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:42.292+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:42.292+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:23:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:43 smithi044 ceph-mon[32404]: pgmap v3779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:44.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:43 smithi116 ceph-mon[37097]: pgmap v3779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:43 smithi171 ceph-mon[37789]: pgmap v3779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:45 smithi044 ceph-mon[32404]: pgmap v3780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6170.729670 secs 2024-07-26T10:23:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:45 smithi116 ceph-mon[37097]: pgmap v3780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6170.729670 secs 2024-07-26T10:23:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:45 smithi171 ceph-mon[37789]: pgmap v3780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6170.729670 secs 2024-07-26T10:23:47.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:47 smithi171 ceph-mon[37789]: pgmap v3781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:47 smithi044 ceph-mon[32404]: pgmap v3781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:47 smithi116 ceph-mon[37097]: pgmap v3781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:49 smithi044 ceph-mon[32404]: pgmap v3782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:49 smithi116 ceph-mon[37097]: pgmap v3782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:49 smithi171 ceph-mon[37789]: pgmap v3782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:23:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6175.729815 secs 2024-07-26T10:23:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6175.729815 secs 2024-07-26T10:23:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6175.729815 secs 2024-07-26T10:23:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:51 smithi171 ceph-mon[37789]: pgmap v3783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:52.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:51 smithi044 ceph-mon[32404]: pgmap v3783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:51 smithi116 ceph-mon[37097]: pgmap v3783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:23:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:53 smithi044 ceph-mon[32404]: pgmap v3784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:53 smithi116 ceph-mon[37097]: pgmap v3784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:53 smithi171 ceph-mon[37789]: pgmap v3784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:23:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:55 smithi044 ceph-mon[32404]: pgmap v3785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6180.729966 secs 2024-07-26T10:23:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:55 smithi116 ceph-mon[37097]: pgmap v3785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6180.729966 secs 2024-07-26T10:23:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:55 smithi171 ceph-mon[37789]: pgmap v3785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:23:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6180.729966 secs 2024-07-26T10:23:57.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:57.299+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:23:57.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:57.299+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:57.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:23:57.299+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:23:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:57 smithi044 ceph-mon[32404]: pgmap v3786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:57 smithi116 ceph-mon[37097]: pgmap v3786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:23:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:57 smithi171 ceph-mon[37789]: pgmap v3786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:23:59 smithi044 ceph-mon[32404]: pgmap v3787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:23:59 smithi116 ceph-mon[37097]: pgmap v3787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:23:59 smithi171 ceph-mon[37789]: pgmap v3787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6185.730127 secs 2024-07-26T10:24:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6185.730127 secs 2024-07-26T10:24:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6185.730127 secs 2024-07-26T10:24:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:01 smithi044 ceph-mon[32404]: pgmap v3788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:01 smithi116 ceph-mon[37097]: pgmap v3788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:01 smithi171 ceph-mon[37789]: pgmap v3788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:03 smithi044 ceph-mon[32404]: pgmap v3789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:03 smithi116 ceph-mon[37097]: pgmap v3789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:03 smithi171 ceph-mon[37789]: pgmap v3789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:05 smithi044 ceph-mon[32404]: pgmap v3790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6190.730290 secs 2024-07-26T10:24:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:05 smithi116 ceph-mon[37097]: pgmap v3790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6190.730290 secs 2024-07-26T10:24:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:05 smithi171 ceph-mon[37789]: pgmap v3790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6190.730290 secs 2024-07-26T10:24:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:07 smithi044 ceph-mon[32404]: pgmap v3791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:07 smithi116 ceph-mon[37097]: pgmap v3791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:07 smithi171 ceph-mon[37789]: pgmap v3791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:09 smithi044 ceph-mon[32404]: pgmap v3792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:09 smithi116 ceph-mon[37097]: pgmap v3792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:09 smithi171 ceph-mon[37789]: pgmap v3792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6195.730411 secs 2024-07-26T10:24:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6195.730411 secs 2024-07-26T10:24:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6195.730411 secs 2024-07-26T10:24:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:11 smithi044 ceph-mon[32404]: pgmap v3793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:11 smithi116 ceph-mon[37097]: pgmap v3793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:11 smithi171 ceph-mon[37789]: pgmap v3793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:12.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:12.301+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:24:12.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:12.301+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:24:12.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:12.301+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:13 smithi044 ceph-mon[32404]: pgmap v3794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:13 smithi116 ceph-mon[37097]: pgmap v3794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:13 smithi171 ceph-mon[37789]: pgmap v3794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:15 smithi044 ceph-mon[32404]: pgmap v3795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6200.730591 secs 2024-07-26T10:24:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:15 smithi116 ceph-mon[37097]: pgmap v3795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6200.730591 secs 2024-07-26T10:24:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:15 smithi171 ceph-mon[37789]: pgmap v3795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6200.730591 secs 2024-07-26T10:24:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:17 smithi044 ceph-mon[32404]: pgmap v3796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:24:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:17 smithi116 ceph-mon[37097]: pgmap v3796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:24:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:17 smithi171 ceph-mon[37789]: pgmap v3796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:24:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:19 smithi044 ceph-mon[32404]: pgmap v3797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:19 smithi116 ceph-mon[37097]: pgmap v3797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:19 smithi171 ceph-mon[37789]: pgmap v3797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6205.730773 secs 2024-07-26T10:24:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6205.730773 secs 2024-07-26T10:24:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6205.730773 secs 2024-07-26T10:24:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:21 smithi044 ceph-mon[32404]: pgmap v3798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:24:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:21 smithi116 ceph-mon[37097]: pgmap v3798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:24:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:21 smithi171 ceph-mon[37789]: pgmap v3798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:24:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:23 smithi044 ceph-mon[32404]: pgmap v3799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:23 smithi116 ceph-mon[37097]: pgmap v3799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:23 smithi171 ceph-mon[37789]: pgmap v3799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:25 smithi044 ceph-mon[32404]: pgmap v3800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:26.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6210.730959 secs 2024-07-26T10:24:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:25 smithi116 ceph-mon[37097]: pgmap v3800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6210.730959 secs 2024-07-26T10:24:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:25 smithi171 ceph-mon[37789]: pgmap v3800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6210.730959 secs 2024-07-26T10:24:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:27.310+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:24:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:27.310+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:27.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:27.310+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:24:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: pgmap v3801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: pgmap v3801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: pgmap v3801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:28.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: pgmap v3802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: pgmap v3803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:24:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:24:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:24:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: pgmap v3802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: pgmap v3803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:24:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:24:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:24:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: pgmap v3802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:24:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: pgmap v3803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:24:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:30 smithi171 ceph-mon[37789]: pgmap v3804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6215.731144 secs 2024-07-26T10:24:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:30 smithi044 ceph-mon[32404]: pgmap v3804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6215.731144 secs 2024-07-26T10:24:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:30 smithi116 ceph-mon[37097]: pgmap v3804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6215.731144 secs 2024-07-26T10:24:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:32 smithi171 ceph-mon[37789]: pgmap v3805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:33.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:32 smithi044 ceph-mon[32404]: pgmap v3805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:33.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:32 smithi116 ceph-mon[37097]: pgmap v3805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:34 smithi044 ceph-mon[32404]: pgmap v3806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:34 smithi171 ceph-mon[37789]: pgmap v3806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:34 smithi116 ceph-mon[37097]: pgmap v3806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:24:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6220.731304 secs 2024-07-26T10:24:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6220.731304 secs 2024-07-26T10:24:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6220.731304 secs 2024-07-26T10:24:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:36 smithi171 ceph-mon[37789]: pgmap v3807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:36 smithi044 ceph-mon[32404]: pgmap v3807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:36 smithi116 ceph-mon[37097]: pgmap v3807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:24:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:38 smithi171 ceph-mon[37789]: pgmap v3808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:38 smithi044 ceph-mon[32404]: pgmap v3808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:38 smithi116 ceph-mon[37097]: pgmap v3808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:24:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:40 smithi171 ceph-mon[37789]: pgmap v3809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6225.731488 secs 2024-07-26T10:24:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:40 smithi044 ceph-mon[32404]: pgmap v3809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6225.731488 secs 2024-07-26T10:24:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:40 smithi116 ceph-mon[37097]: pgmap v3809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6225.731488 secs 2024-07-26T10:24:42.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:42.314+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:42.314+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:24:42.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:42.314+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:42 smithi171 ceph-mon[37789]: pgmap v3810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:42 smithi044 ceph-mon[32404]: pgmap v3810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:42 smithi116 ceph-mon[37097]: pgmap v3810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:44 smithi044 ceph-mon[32404]: pgmap v3811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:44 smithi116 ceph-mon[37097]: pgmap v3811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:44 smithi171 ceph-mon[37789]: pgmap v3811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6230.731611 secs 2024-07-26T10:24:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6230.731611 secs 2024-07-26T10:24:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6230.731611 secs 2024-07-26T10:24:47.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:46 smithi116 ceph-mon[37097]: pgmap v3812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:47.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:46 smithi171 ceph-mon[37789]: pgmap v3812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:46 smithi044 ceph-mon[32404]: pgmap v3812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:48 smithi044 ceph-mon[32404]: pgmap v3813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:48 smithi116 ceph-mon[37097]: pgmap v3813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:48 smithi171 ceph-mon[37789]: pgmap v3813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:50 smithi044 ceph-mon[32404]: pgmap v3814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6235.731778 secs 2024-07-26T10:24:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:50 smithi116 ceph-mon[37097]: pgmap v3814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6235.731778 secs 2024-07-26T10:24:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:50 smithi171 ceph-mon[37789]: pgmap v3814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:24:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6235.731778 secs 2024-07-26T10:24:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:52 smithi044 ceph-mon[32404]: pgmap v3815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:52 smithi116 ceph-mon[37097]: pgmap v3815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:52 smithi171 ceph-mon[37789]: pgmap v3815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:24:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:54 smithi044 ceph-mon[32404]: pgmap v3816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:54 smithi116 ceph-mon[37097]: pgmap v3816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:54 smithi171 ceph-mon[37789]: pgmap v3816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:24:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6240.731945 secs 2024-07-26T10:24:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6240.731945 secs 2024-07-26T10:24:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6240.731945 secs 2024-07-26T10:24:57.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:56 smithi044 ceph-mon[32404]: pgmap v3817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:56 smithi116 ceph-mon[37097]: pgmap v3817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:57.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:57.318+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:24:57.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:57.318+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:24:57.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:24:57.318+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:56 smithi171 ceph-mon[37789]: pgmap v3817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:24:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:24:58 smithi044 ceph-mon[32404]: pgmap v3818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:24:58 smithi116 ceph-mon[37097]: pgmap v3818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:24:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:24:58 smithi171 ceph-mon[37789]: pgmap v3818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:00 smithi044 ceph-mon[32404]: pgmap v3819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6245.732119 secs 2024-07-26T10:25:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:00 smithi116 ceph-mon[37097]: pgmap v3819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6245.732119 secs 2024-07-26T10:25:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:00 smithi171 ceph-mon[37789]: pgmap v3819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6245.732119 secs 2024-07-26T10:25:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:02 smithi044 ceph-mon[32404]: pgmap v3820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:02 smithi116 ceph-mon[37097]: pgmap v3820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:02 smithi171 ceph-mon[37789]: pgmap v3820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:04 smithi044 ceph-mon[32404]: pgmap v3821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:04 smithi116 ceph-mon[37097]: pgmap v3821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:04 smithi171 ceph-mon[37789]: pgmap v3821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6250.732305 secs 2024-07-26T10:25:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6250.732305 secs 2024-07-26T10:25:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6250.732305 secs 2024-07-26T10:25:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:06 smithi044 ceph-mon[32404]: pgmap v3822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:06 smithi116 ceph-mon[37097]: pgmap v3822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:06 smithi171 ceph-mon[37789]: pgmap v3822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:08 smithi044 ceph-mon[32404]: pgmap v3823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:08 smithi116 ceph-mon[37097]: pgmap v3823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:08 smithi171 ceph-mon[37789]: pgmap v3823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:10 smithi044 ceph-mon[32404]: pgmap v3824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6255.732470 secs 2024-07-26T10:25:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:10 smithi116 ceph-mon[37097]: pgmap v3824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6255.732470 secs 2024-07-26T10:25:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:10 smithi171 ceph-mon[37789]: pgmap v3824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6255.732470 secs 2024-07-26T10:25:12.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:12.322+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:25:12.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:12.322+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:25:12.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:12.322+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:12 smithi044 ceph-mon[32404]: pgmap v3825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:12 smithi116 ceph-mon[37097]: pgmap v3825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:12 smithi171 ceph-mon[37789]: pgmap v3825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:14 smithi044 ceph-mon[32404]: pgmap v3826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:14 smithi116 ceph-mon[37097]: pgmap v3826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:14 smithi171 ceph-mon[37789]: pgmap v3826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6260.732689 secs 2024-07-26T10:25:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6260.732689 secs 2024-07-26T10:25:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6260.732689 secs 2024-07-26T10:25:17.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:16 smithi171 ceph-mon[37789]: pgmap v3827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:16 smithi044 ceph-mon[32404]: pgmap v3827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:16 smithi116 ceph-mon[37097]: pgmap v3827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:18 smithi044 ceph-mon[32404]: pgmap v3828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:18 smithi116 ceph-mon[37097]: pgmap v3828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:18 smithi171 ceph-mon[37789]: pgmap v3828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:20 smithi044 ceph-mon[32404]: pgmap v3829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6265.732871 secs 2024-07-26T10:25:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:20 smithi116 ceph-mon[37097]: pgmap v3829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6265.732871 secs 2024-07-26T10:25:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:20 smithi171 ceph-mon[37789]: pgmap v3829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:25:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6265.732871 secs 2024-07-26T10:25:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:22 smithi044 ceph-mon[32404]: pgmap v3830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:22 smithi116 ceph-mon[37097]: pgmap v3830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:22 smithi171 ceph-mon[37789]: pgmap v3830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:25:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:24 smithi044 ceph-mon[32404]: pgmap v3831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:24.302 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:24 smithi116 ceph-mon[37097]: pgmap v3831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:24 smithi171 ceph-mon[37789]: pgmap v3831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:25:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6270.733050 secs 2024-07-26T10:25:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6270.733050 secs 2024-07-26T10:25:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6270.733050 secs 2024-07-26T10:25:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:26 smithi171 ceph-mon[37789]: pgmap v3832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:26 smithi044 ceph-mon[32404]: pgmap v3832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:26 smithi116 ceph-mon[37097]: pgmap v3832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:27.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:27.331+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:25:27.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:27.331+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:25:27.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:27.331+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: pgmap v3833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: pgmap v3833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: pgmap v3833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:25:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:25:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:25:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:25:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:25:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:25:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:25:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:25:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:25:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:25:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:25:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:30 smithi044 ceph-mon[32404]: pgmap v3834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:25:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:30 smithi044 ceph-mon[32404]: pgmap v3835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6275.733204 secs 2024-07-26T10:25:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:30 smithi116 ceph-mon[37097]: pgmap v3834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:25:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:30 smithi116 ceph-mon[37097]: pgmap v3835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6275.733204 secs 2024-07-26T10:25:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:30 smithi171 ceph-mon[37789]: pgmap v3834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:25:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:30 smithi171 ceph-mon[37789]: pgmap v3835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6275.733204 secs 2024-07-26T10:25:33.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:32 smithi044 ceph-mon[32404]: pgmap v3836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:25:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:32 smithi116 ceph-mon[37097]: pgmap v3836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:25:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:32 smithi171 ceph-mon[37789]: pgmap v3836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:25:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:34 smithi044 ceph-mon[32404]: pgmap v3837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:34 smithi116 ceph-mon[37097]: pgmap v3837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:34 smithi171 ceph-mon[37789]: pgmap v3837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6280.733387 secs 2024-07-26T10:25:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6280.733387 secs 2024-07-26T10:25:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6280.733387 secs 2024-07-26T10:25:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:36 smithi044 ceph-mon[32404]: pgmap v3838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:36 smithi116 ceph-mon[37097]: pgmap v3838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:36 smithi171 ceph-mon[37789]: pgmap v3838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:38 smithi044 ceph-mon[32404]: pgmap v3839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:38 smithi116 ceph-mon[37097]: pgmap v3839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:38 smithi171 ceph-mon[37789]: pgmap v3839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:25:41.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:40 smithi044 ceph-mon[32404]: pgmap v3840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:25:41.271 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6285.733587 secs 2024-07-26T10:25:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:40 smithi116 ceph-mon[37097]: pgmap v3840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:25:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6285.733587 secs 2024-07-26T10:25:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:40 smithi171 ceph-mon[37789]: pgmap v3840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:25:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6285.733587 secs 2024-07-26T10:25:42.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:42.335+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:25:42.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:42.335+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:25:42.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:42.335+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:42 smithi044 ceph-mon[32404]: pgmap v3841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:42 smithi116 ceph-mon[37097]: pgmap v3841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:42 smithi171 ceph-mon[37789]: pgmap v3841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:25:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:44 smithi044 ceph-mon[32404]: pgmap v3842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:44 smithi116 ceph-mon[37097]: pgmap v3842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:44 smithi171 ceph-mon[37789]: pgmap v3842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6290.733738 secs 2024-07-26T10:25:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6290.733738 secs 2024-07-26T10:25:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6290.733738 secs 2024-07-26T10:25:47.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:46 smithi116 ceph-mon[37097]: pgmap v3843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:47.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:46 smithi171 ceph-mon[37789]: pgmap v3843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:46 smithi044 ceph-mon[32404]: pgmap v3843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:48 smithi044 ceph-mon[32404]: pgmap v3844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:48 smithi116 ceph-mon[37097]: pgmap v3844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:48 smithi171 ceph-mon[37789]: pgmap v3844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:50 smithi044 ceph-mon[32404]: pgmap v3845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6295.733849 secs 2024-07-26T10:25:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:50 smithi116 ceph-mon[37097]: pgmap v3845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6295.733849 secs 2024-07-26T10:25:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:50 smithi171 ceph-mon[37789]: pgmap v3845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6295.733849 secs 2024-07-26T10:25:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:52 smithi044 ceph-mon[32404]: pgmap v3846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:52 smithi116 ceph-mon[37097]: pgmap v3846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:52 smithi171 ceph-mon[37789]: pgmap v3846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:54 smithi044 ceph-mon[32404]: pgmap v3847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:54 smithi116 ceph-mon[37097]: pgmap v3847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:54 smithi171 ceph-mon[37789]: pgmap v3847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:25:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6300.734030 secs 2024-07-26T10:25:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6300.734030 secs 2024-07-26T10:25:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6300.734030 secs 2024-07-26T10:25:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:56 smithi044 ceph-mon[32404]: pgmap v3848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:56 smithi116 ceph-mon[37097]: pgmap v3848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:56 smithi171 ceph-mon[37789]: pgmap v3848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:25:57.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:57.344+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:25:57.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:57.344+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:57.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:25:57.344+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:25:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:25:58 smithi044 ceph-mon[32404]: pgmap v3849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:25:58 smithi116 ceph-mon[37097]: pgmap v3849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:25:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:25:58 smithi171 ceph-mon[37789]: pgmap v3849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:00 smithi044 ceph-mon[32404]: pgmap v3850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6305.734172 secs 2024-07-26T10:26:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:00 smithi116 ceph-mon[37097]: pgmap v3850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6305.734172 secs 2024-07-26T10:26:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:00 smithi171 ceph-mon[37789]: pgmap v3850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6305.734172 secs 2024-07-26T10:26:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:02 smithi044 ceph-mon[32404]: pgmap v3851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:02 smithi116 ceph-mon[37097]: pgmap v3851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:02 smithi171 ceph-mon[37789]: pgmap v3851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:04 smithi044 ceph-mon[32404]: pgmap v3852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:04 smithi116 ceph-mon[37097]: pgmap v3852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:04 smithi171 ceph-mon[37789]: pgmap v3852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6310.734328 secs 2024-07-26T10:26:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6310.734328 secs 2024-07-26T10:26:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6310.734328 secs 2024-07-26T10:26:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:06 smithi044 ceph-mon[32404]: pgmap v3853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:06 smithi116 ceph-mon[37097]: pgmap v3853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:06 smithi171 ceph-mon[37789]: pgmap v3853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:08 smithi044 ceph-mon[32404]: pgmap v3854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:08 smithi116 ceph-mon[37097]: pgmap v3854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:08 smithi171 ceph-mon[37789]: pgmap v3854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:10 smithi044 ceph-mon[32404]: pgmap v3855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:10 smithi116 ceph-mon[37097]: pgmap v3855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:10 smithi171 ceph-mon[37789]: pgmap v3855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6315.734506 secs 2024-07-26T10:26:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6315.734506 secs 2024-07-26T10:26:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6315.734506 secs 2024-07-26T10:26:12.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:12.348+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:26:12.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:12.348+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:12.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:12.348+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:26:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:12 smithi044 ceph-mon[32404]: pgmap v3856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:12 smithi116 ceph-mon[37097]: pgmap v3856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:12 smithi171 ceph-mon[37789]: pgmap v3856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:14 smithi044 ceph-mon[32404]: pgmap v3857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:14 smithi116 ceph-mon[37097]: pgmap v3857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:14 smithi171 ceph-mon[37789]: pgmap v3857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:14.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 msgr2=0x7fcf540b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 msgr2=0x7fcf540b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 0x7fcf540b5300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcf5805c000 tx=0x7fcf58064000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 0x7fcf540b5300 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcf5805c000 tx=0x7fcf58064000 comp rx=0 tx=0).stop 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 msgr2=0x7fcf540b5300 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 msgr2=0x7fcf540b5300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b4ef0 0x7fcf540b5300 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:26:14.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.703+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:26:14.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.704+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:26:14.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:14.706+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf58069040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:26:15.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:15.796+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:26:15.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:15.796+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:26:15.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:15.797+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:26:15.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:15.797+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:26:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:16 smithi044 ceph-mon[32404]: pgmap v3858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6320.734659 secs 2024-07-26T10:26:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:16 smithi116 ceph-mon[37097]: pgmap v3858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6320.734659 secs 2024-07-26T10:26:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:16 smithi171 ceph-mon[37789]: pgmap v3858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6320.734659 secs 2024-07-26T10:26:18.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:18 smithi171 ceph-mon[37789]: pgmap v3859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:18 smithi044 ceph-mon[32404]: pgmap v3859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:18 smithi116 ceph-mon[37097]: pgmap v3859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:20 smithi044 ceph-mon[32404]: pgmap v3860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:20 smithi116 ceph-mon[37097]: pgmap v3860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:20 smithi171 ceph-mon[37789]: pgmap v3860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6325.734778 secs 2024-07-26T10:26:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6325.734778 secs 2024-07-26T10:26:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6325.734778 secs 2024-07-26T10:26:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:22 smithi044 ceph-mon[32404]: pgmap v3861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:26:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:22 smithi116 ceph-mon[37097]: pgmap v3861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:26:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:22 smithi171 ceph-mon[37789]: pgmap v3861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:26:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:24 smithi044 ceph-mon[32404]: pgmap v3862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:26:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:24 smithi116 ceph-mon[37097]: pgmap v3862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:26:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:24 smithi171 ceph-mon[37789]: pgmap v3862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:26:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:26 smithi044 ceph-mon[32404]: pgmap v3863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:26:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6330.734950 secs 2024-07-26T10:26:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:26 smithi116 ceph-mon[37097]: pgmap v3863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:26:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6330.734950 secs 2024-07-26T10:26:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:26 smithi171 ceph-mon[37789]: pgmap v3863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:26:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6330.734950 secs 2024-07-26T10:26:27.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:27.353+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:26:27.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:27.353+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:27.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:27.353+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:26:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:28 smithi044 ceph-mon[32404]: pgmap v3864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:28 smithi116 ceph-mon[37097]: pgmap v3864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:28 smithi171 ceph-mon[37789]: pgmap v3864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:26:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:30 smithi044 ceph-mon[32404]: pgmap v3865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:30 smithi116 ceph-mon[37097]: pgmap v3865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:30 smithi171 ceph-mon[37789]: pgmap v3865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:31 smithi044 ceph-mon[32404]: pgmap v3866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:26:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:31 smithi044 ceph-mon[32404]: pgmap v3867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:26:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6335.735059 secs 2024-07-26T10:26:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:31 smithi116 ceph-mon[37097]: pgmap v3866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:26:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:31 smithi116 ceph-mon[37097]: pgmap v3867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:26:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6335.735059 secs 2024-07-26T10:26:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:31 smithi171 ceph-mon[37789]: pgmap v3866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:26:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:31 smithi171 ceph-mon[37789]: pgmap v3867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:26:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6335.735059 secs 2024-07-26T10:26:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:33 smithi171 ceph-mon[37789]: pgmap v3868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:26:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:33 smithi044 ceph-mon[32404]: pgmap v3868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:26:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:33 smithi116 ceph-mon[37097]: pgmap v3868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:26:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:35 smithi171 ceph-mon[37789]: pgmap v3869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:35 smithi044 ceph-mon[32404]: pgmap v3869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:35 smithi116 ceph-mon[37097]: pgmap v3869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6340.735231 secs 2024-07-26T10:26:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6340.735231 secs 2024-07-26T10:26:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6340.735231 secs 2024-07-26T10:26:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:37 smithi044 ceph-mon[32404]: pgmap v3870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:37 smithi116 ceph-mon[37097]: pgmap v3870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:37 smithi171 ceph-mon[37789]: pgmap v3870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:39 smithi044 ceph-mon[32404]: pgmap v3871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:39 smithi116 ceph-mon[37097]: pgmap v3871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:39 smithi171 ceph-mon[37789]: pgmap v3871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:26:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:41 smithi044 ceph-mon[32404]: pgmap v3872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:26:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6345.735385 secs 2024-07-26T10:26:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:41 smithi116 ceph-mon[37097]: pgmap v3872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:26:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6345.735385 secs 2024-07-26T10:26:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:41 smithi171 ceph-mon[37789]: pgmap v3872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:26:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6345.735385 secs 2024-07-26T10:26:42.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:42.357+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:26:42.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:42.357+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:42.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:42.357+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:26:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:43 smithi044 ceph-mon[32404]: pgmap v3873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:43 smithi116 ceph-mon[37097]: pgmap v3873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:43 smithi171 ceph-mon[37789]: pgmap v3873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:45 smithi044 ceph-mon[32404]: pgmap v3874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:45 smithi116 ceph-mon[37097]: pgmap v3874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:45 smithi171 ceph-mon[37789]: pgmap v3874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6350.735559 secs 2024-07-26T10:26:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6350.735559 secs 2024-07-26T10:26:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6350.735559 secs 2024-07-26T10:26:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:47 smithi044 ceph-mon[32404]: pgmap v3875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:47 smithi116 ceph-mon[37097]: pgmap v3875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:47 smithi171 ceph-mon[37789]: pgmap v3875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:49 smithi171 ceph-mon[37789]: pgmap v3876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:49 smithi044 ceph-mon[32404]: pgmap v3876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:49 smithi116 ceph-mon[37097]: pgmap v3876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:51 smithi171 ceph-mon[37789]: pgmap v3877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6355.735720 secs 2024-07-26T10:26:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:51 smithi044 ceph-mon[32404]: pgmap v3877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6355.735720 secs 2024-07-26T10:26:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:51 smithi116 ceph-mon[37097]: pgmap v3877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6355.735720 secs 2024-07-26T10:26:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:53 smithi044 ceph-mon[32404]: pgmap v3878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:53 smithi116 ceph-mon[37097]: pgmap v3878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:53 smithi171 ceph-mon[37789]: pgmap v3878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:26:55.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:55 smithi044 ceph-mon[32404]: pgmap v3879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:55 smithi116 ceph-mon[37097]: pgmap v3879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:55 smithi171 ceph-mon[37789]: pgmap v3879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:26:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6360.735835 secs 2024-07-26T10:26:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6360.735835 secs 2024-07-26T10:26:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6360.735835 secs 2024-07-26T10:26:57.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:57.366+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:26:57.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:57.366+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:26:57.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:26:57.366+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:57.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:57 smithi044 ceph-mon[32404]: pgmap v3880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:57 smithi116 ceph-mon[37097]: pgmap v3880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:57 smithi171 ceph-mon[37789]: pgmap v3880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:26:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:26:59 smithi044 ceph-mon[32404]: pgmap v3881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:26:59 smithi116 ceph-mon[37097]: pgmap v3881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:26:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:26:59 smithi171 ceph-mon[37789]: pgmap v3881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:01 smithi044 ceph-mon[32404]: pgmap v3882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6365.735989 secs 2024-07-26T10:27:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:01 smithi116 ceph-mon[37097]: pgmap v3882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6365.735989 secs 2024-07-26T10:27:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:01 smithi171 ceph-mon[37789]: pgmap v3882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6365.735989 secs 2024-07-26T10:27:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:03 smithi044 ceph-mon[32404]: pgmap v3883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:03 smithi116 ceph-mon[37097]: pgmap v3883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:03 smithi171 ceph-mon[37789]: pgmap v3883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:05 smithi044 ceph-mon[32404]: pgmap v3884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:05 smithi116 ceph-mon[37097]: pgmap v3884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:05 smithi171 ceph-mon[37789]: pgmap v3884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6370.736130 secs 2024-07-26T10:27:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6370.736130 secs 2024-07-26T10:27:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6370.736130 secs 2024-07-26T10:27:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:07 smithi044 ceph-mon[32404]: pgmap v3885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:07 smithi116 ceph-mon[37097]: pgmap v3885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:07 smithi171 ceph-mon[37789]: pgmap v3885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:09 smithi044 ceph-mon[32404]: pgmap v3886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:09 smithi116 ceph-mon[37097]: pgmap v3886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:09 smithi171 ceph-mon[37789]: pgmap v3886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:11 smithi044 ceph-mon[32404]: pgmap v3887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6375.736261 secs 2024-07-26T10:27:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:11 smithi116 ceph-mon[37097]: pgmap v3887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6375.736261 secs 2024-07-26T10:27:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:11 smithi171 ceph-mon[37789]: pgmap v3887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6375.736261 secs 2024-07-26T10:27:12.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:12.370+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:27:12.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:12.370+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:27:12.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:12.370+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:13 smithi044 ceph-mon[32404]: pgmap v3888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:13 smithi116 ceph-mon[37097]: pgmap v3888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:13 smithi171 ceph-mon[37789]: pgmap v3888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:15 smithi044 ceph-mon[32404]: pgmap v3889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:15 smithi116 ceph-mon[37097]: pgmap v3889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:15 smithi171 ceph-mon[37789]: pgmap v3889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6380.736426 secs 2024-07-26T10:27:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6380.736426 secs 2024-07-26T10:27:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6380.736426 secs 2024-07-26T10:27:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:17 smithi044 ceph-mon[32404]: pgmap v3890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:17 smithi116 ceph-mon[37097]: pgmap v3890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:17 smithi171 ceph-mon[37789]: pgmap v3890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:19 smithi044 ceph-mon[32404]: pgmap v3891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:19 smithi116 ceph-mon[37097]: pgmap v3891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:19 smithi171 ceph-mon[37789]: pgmap v3891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:21 smithi044 ceph-mon[32404]: pgmap v3892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6385.736561 secs 2024-07-26T10:27:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:21 smithi116 ceph-mon[37097]: pgmap v3892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6385.736561 secs 2024-07-26T10:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:21 smithi171 ceph-mon[37789]: pgmap v3892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6385.736561 secs 2024-07-26T10:27:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:23 smithi044 ceph-mon[32404]: pgmap v3893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:23 smithi116 ceph-mon[37097]: pgmap v3893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:23 smithi171 ceph-mon[37789]: pgmap v3893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:25.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:25 smithi044 ceph-mon[32404]: pgmap v3894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:25 smithi116 ceph-mon[37097]: pgmap v3894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:25 smithi171 ceph-mon[37789]: pgmap v3894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6390.736671 secs 2024-07-26T10:27:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6390.736671 secs 2024-07-26T10:27:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6390.736671 secs 2024-07-26T10:27:27.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:27.379+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:27:27.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:27.379+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:27:27.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:27.379+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:27.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:27 smithi044 ceph-mon[32404]: pgmap v3895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:27 smithi116 ceph-mon[37097]: pgmap v3895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:27 smithi171 ceph-mon[37789]: pgmap v3895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:29 smithi044 ceph-mon[32404]: pgmap v3896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:29 smithi116 ceph-mon[37097]: pgmap v3896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:29 smithi171 ceph-mon[37789]: pgmap v3896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:31 smithi044 ceph-mon[32404]: pgmap v3897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6395.736825 secs 2024-07-26T10:27:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:31 smithi116 ceph-mon[37097]: pgmap v3897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6395.736825 secs 2024-07-26T10:27:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:31 smithi171 ceph-mon[37789]: pgmap v3897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6395.736825 secs 2024-07-26T10:27:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:27:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:33 smithi044 ceph-mon[32404]: pgmap v3898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:33 smithi044 ceph-mon[32404]: pgmap v3899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:33 smithi116 ceph-mon[37097]: pgmap v3898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:33 smithi116 ceph-mon[37097]: pgmap v3899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:33 smithi171 ceph-mon[37789]: pgmap v3898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:33 smithi171 ceph-mon[37789]: pgmap v3899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:35 smithi044 ceph-mon[32404]: pgmap v3900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:35 smithi116 ceph-mon[37097]: pgmap v3900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:35 smithi171 ceph-mon[37789]: pgmap v3900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6400.736970 secs 2024-07-26T10:27:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6400.736970 secs 2024-07-26T10:27:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6400.736970 secs 2024-07-26T10:27:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:37 smithi116 ceph-mon[37097]: pgmap v3901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:37 smithi171 ceph-mon[37789]: pgmap v3901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:37 smithi044 ceph-mon[32404]: pgmap v3901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:39 smithi171 ceph-mon[37789]: pgmap v3902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:39 smithi044 ceph-mon[32404]: pgmap v3902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:39 smithi116 ceph-mon[37097]: pgmap v3902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:41 smithi171 ceph-mon[37789]: pgmap v3903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6405.737128 secs 2024-07-26T10:27:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:41 smithi044 ceph-mon[32404]: pgmap v3903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6405.737128 secs 2024-07-26T10:27:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:41 smithi116 ceph-mon[37097]: pgmap v3903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:27:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6405.737128 secs 2024-07-26T10:27:42.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:42.383+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:27:42.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:42.383+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:27:42.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:42.383+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:43 smithi171 ceph-mon[37789]: pgmap v3904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:27:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:43 smithi044 ceph-mon[32404]: pgmap v3904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:27:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:43 smithi116 ceph-mon[37097]: pgmap v3904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:27:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:45 smithi171 ceph-mon[37789]: pgmap v3905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:45 smithi044 ceph-mon[32404]: pgmap v3905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:45 smithi116 ceph-mon[37097]: pgmap v3905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6410.737223 secs 2024-07-26T10:27:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6410.737223 secs 2024-07-26T10:27:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6410.737223 secs 2024-07-26T10:27:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:47 smithi171 ceph-mon[37789]: pgmap v3906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:47 smithi044 ceph-mon[32404]: pgmap v3906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:47 smithi116 ceph-mon[37097]: pgmap v3906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:49.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:49 smithi116 ceph-mon[37097]: pgmap v3907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:49 smithi171 ceph-mon[37789]: pgmap v3907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:49 smithi044 ceph-mon[32404]: pgmap v3907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:27:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6415.737373 secs 2024-07-26T10:27:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6415.737373 secs 2024-07-26T10:27:51.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6415.737373 secs 2024-07-26T10:27:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:51 smithi171 ceph-mon[37789]: pgmap v3908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:51 smithi044 ceph-mon[32404]: pgmap v3908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:51 smithi116 ceph-mon[37097]: pgmap v3908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:53 smithi171 ceph-mon[37789]: pgmap v3909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:53 smithi044 ceph-mon[32404]: pgmap v3909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:53 smithi116 ceph-mon[37097]: pgmap v3909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:27:55.938 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:55 smithi044 ceph-mon[32404]: pgmap v3910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:55.938 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6420.737523 secs 2024-07-26T10:27:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:55 smithi116 ceph-mon[37097]: pgmap v3910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6420.737523 secs 2024-07-26T10:27:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:55 smithi171 ceph-mon[37789]: pgmap v3910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:27:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6420.737523 secs 2024-07-26T10:27:57.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:57.392+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:27:57.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:57.392+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:57.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:27:57.392+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:27:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:57 smithi044 ceph-mon[32404]: pgmap v3911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:57 smithi116 ceph-mon[37097]: pgmap v3911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:27:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:57 smithi171 ceph-mon[37789]: pgmap v3911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:27:59 smithi044 ceph-mon[32404]: pgmap v3912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:27:59 smithi116 ceph-mon[37097]: pgmap v3912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:27:59 smithi171 ceph-mon[37789]: pgmap v3912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6425.737705 secs 2024-07-26T10:28:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6425.737705 secs 2024-07-26T10:28:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6425.737705 secs 2024-07-26T10:28:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:01 smithi044 ceph-mon[32404]: pgmap v3913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:01 smithi116 ceph-mon[37097]: pgmap v3913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:01 smithi171 ceph-mon[37789]: pgmap v3913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:03 smithi044 ceph-mon[32404]: pgmap v3914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:03 smithi116 ceph-mon[37097]: pgmap v3914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:03 smithi171 ceph-mon[37789]: pgmap v3914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:05 smithi044 ceph-mon[32404]: pgmap v3915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6430.737870 secs 2024-07-26T10:28:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:05 smithi116 ceph-mon[37097]: pgmap v3915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6430.737870 secs 2024-07-26T10:28:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:05 smithi171 ceph-mon[37789]: pgmap v3915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6430.737870 secs 2024-07-26T10:28:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:07 smithi044 ceph-mon[32404]: pgmap v3916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:07 smithi116 ceph-mon[37097]: pgmap v3916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:07 smithi171 ceph-mon[37789]: pgmap v3916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:09 smithi044 ceph-mon[32404]: pgmap v3917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:09 smithi116 ceph-mon[37097]: pgmap v3917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:09 smithi171 ceph-mon[37789]: pgmap v3917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6435.737971 secs 2024-07-26T10:28:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6435.737971 secs 2024-07-26T10:28:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6435.737971 secs 2024-07-26T10:28:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:11 smithi044 ceph-mon[32404]: pgmap v3918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:11 smithi116 ceph-mon[37097]: pgmap v3918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:11 smithi171 ceph-mon[37789]: pgmap v3918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:12.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:12.394+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:28:12.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:12.394+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:12.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:12.394+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:28:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:13 smithi044 ceph-mon[32404]: pgmap v3919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:13 smithi116 ceph-mon[37097]: pgmap v3919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:13 smithi171 ceph-mon[37789]: pgmap v3919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:15 smithi044 ceph-mon[32404]: pgmap v3920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:16.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6440.738120 secs 2024-07-26T10:28:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:15 smithi116 ceph-mon[37097]: pgmap v3920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6440.738120 secs 2024-07-26T10:28:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:15 smithi171 ceph-mon[37789]: pgmap v3920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6440.738120 secs 2024-07-26T10:28:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:17 smithi044 ceph-mon[32404]: pgmap v3921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:17 smithi116 ceph-mon[37097]: pgmap v3921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:17 smithi171 ceph-mon[37789]: pgmap v3921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:19 smithi044 ceph-mon[32404]: pgmap v3922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:19 smithi116 ceph-mon[37097]: pgmap v3922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:19 smithi171 ceph-mon[37789]: pgmap v3922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:28:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6445.738267 secs 2024-07-26T10:28:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6445.738267 secs 2024-07-26T10:28:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6445.738267 secs 2024-07-26T10:28:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:21 smithi044 ceph-mon[32404]: pgmap v3923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:21 smithi116 ceph-mon[37097]: pgmap v3923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:21 smithi171 ceph-mon[37789]: pgmap v3923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:23 smithi044 ceph-mon[32404]: pgmap v3924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:28:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:23 smithi116 ceph-mon[37097]: pgmap v3924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:28:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:23 smithi171 ceph-mon[37789]: pgmap v3924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:28:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:25 smithi044 ceph-mon[32404]: pgmap v3925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:28:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6450.738433 secs 2024-07-26T10:28:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:25 smithi116 ceph-mon[37097]: pgmap v3925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:28:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6450.738433 secs 2024-07-26T10:28:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:25 smithi171 ceph-mon[37789]: pgmap v3925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:28:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6450.738433 secs 2024-07-26T10:28:27.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:27.397+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:28:27.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:27.397+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:27.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:27.397+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:28:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:27 smithi044 ceph-mon[32404]: pgmap v3926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:28:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:27 smithi116 ceph-mon[37097]: pgmap v3926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:28:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:27 smithi171 ceph-mon[37789]: pgmap v3926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:28:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:29 smithi044 ceph-mon[32404]: pgmap v3927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:28:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:29 smithi116 ceph-mon[37097]: pgmap v3927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:28:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:29 smithi171 ceph-mon[37789]: pgmap v3927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:28:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:30.822+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:28:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:30.822+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:28:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:30.823+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf400064b0 con 0x7fcf6c1057b0 2024-07-26T10:28:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:30.823+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:28:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6455.738585 secs 2024-07-26T10:28:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6455.738585 secs 2024-07-26T10:28:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6455.738585 secs 2024-07-26T10:28:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:31 smithi044 ceph-mon[32404]: pgmap v3928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:28:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:31 smithi116 ceph-mon[37097]: pgmap v3928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:28:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:31 smithi171 ceph-mon[37789]: pgmap v3928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:28:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:33 smithi044 ceph-mon[32404]: pgmap v3929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:28:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:33 smithi116 ceph-mon[37097]: pgmap v3929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:28:34.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:33 smithi171 ceph-mon[37789]: pgmap v3929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:28:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: pgmap v3930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:35.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: pgmap v3930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: pgmap v3930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6460.738727 secs 2024-07-26T10:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6460.738727 secs 2024-07-26T10:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6460.738727 secs 2024-07-26T10:28:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:38 smithi171 ceph-mon[37789]: pgmap v3931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:38 smithi044 ceph-mon[32404]: pgmap v3931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:38 smithi116 ceph-mon[37097]: pgmap v3931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:28:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:39 smithi171 ceph-mon[37789]: pgmap v3932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:39 smithi044 ceph-mon[32404]: pgmap v3932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:39 smithi116 ceph-mon[37097]: pgmap v3932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:40 smithi171 ceph-mon[37789]: pgmap v3933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:28:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:40 smithi044 ceph-mon[32404]: pgmap v3933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:28:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:40 smithi116 ceph-mon[37097]: pgmap v3933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:28:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6465.738829 secs 2024-07-26T10:28:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6465.738829 secs 2024-07-26T10:28:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6465.738829 secs 2024-07-26T10:28:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:42 smithi171 ceph-mon[37789]: pgmap v3934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:28:42.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:42.401+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:28:42.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:42.401+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:42.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:42.401+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:28:42.404 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:42 smithi044 ceph-mon[32404]: pgmap v3934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:28:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:42 smithi116 ceph-mon[37097]: pgmap v3934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:28:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:44 smithi044 ceph-mon[32404]: pgmap v3935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:44 smithi116 ceph-mon[37097]: pgmap v3935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:44 smithi171 ceph-mon[37789]: pgmap v3935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6470.738992 secs 2024-07-26T10:28:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6470.738992 secs 2024-07-26T10:28:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6470.738992 secs 2024-07-26T10:28:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:46 smithi171 ceph-mon[37789]: pgmap v3936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:28:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:46 smithi044 ceph-mon[32404]: pgmap v3936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:28:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:46 smithi116 ceph-mon[37097]: pgmap v3936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:28:49.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:48 smithi116 ceph-mon[37097]: pgmap v3937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:48 smithi171 ceph-mon[37789]: pgmap v3937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:48 smithi044 ceph-mon[32404]: pgmap v3937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:28:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:50 smithi171 ceph-mon[37789]: pgmap v3938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6475.739152 secs 2024-07-26T10:28:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:50 smithi044 ceph-mon[32404]: pgmap v3938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6475.739152 secs 2024-07-26T10:28:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:50 smithi116 ceph-mon[37097]: pgmap v3938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:28:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6475.739152 secs 2024-07-26T10:28:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:52 smithi171 ceph-mon[37789]: pgmap v3939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:52 smithi044 ceph-mon[32404]: pgmap v3939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:52 smithi116 ceph-mon[37097]: pgmap v3939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:28:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:54 smithi044 ceph-mon[32404]: pgmap v3940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:54.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:54 smithi116 ceph-mon[37097]: pgmap v3940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:54 smithi171 ceph-mon[37789]: pgmap v3940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:28:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6480.739312 secs 2024-07-26T10:28:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6480.739312 secs 2024-07-26T10:28:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6480.739312 secs 2024-07-26T10:28:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:56 smithi044 ceph-mon[32404]: pgmap v3941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:56 smithi116 ceph-mon[37097]: pgmap v3941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:56 smithi171 ceph-mon[37789]: pgmap v3941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:28:57.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:57.410+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:28:57.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:57.410+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:57.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:28:57.410+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:28:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:28:58 smithi116 ceph-mon[37097]: pgmap v3942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:28:58 smithi171 ceph-mon[37789]: pgmap v3942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:28:59.169 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:28:58 smithi044 ceph-mon[32404]: pgmap v3942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:00 smithi171 ceph-mon[37789]: pgmap v3943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6485.739430 secs 2024-07-26T10:29:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:00 smithi044 ceph-mon[32404]: pgmap v3943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6485.739430 secs 2024-07-26T10:29:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:00 smithi116 ceph-mon[37097]: pgmap v3943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6485.739430 secs 2024-07-26T10:29:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:02 smithi171 ceph-mon[37789]: pgmap v3944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:02 smithi044 ceph-mon[32404]: pgmap v3944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:02 smithi116 ceph-mon[37097]: pgmap v3944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:05.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:04 smithi171 ceph-mon[37789]: pgmap v3945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:04 smithi044 ceph-mon[32404]: pgmap v3945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:04 smithi116 ceph-mon[37097]: pgmap v3945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6490.739575 secs 2024-07-26T10:29:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6490.739575 secs 2024-07-26T10:29:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6490.739575 secs 2024-07-26T10:29:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:06 smithi171 ceph-mon[37789]: pgmap v3946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:06 smithi044 ceph-mon[32404]: pgmap v3946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:06 smithi116 ceph-mon[37097]: pgmap v3946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:08 smithi171 ceph-mon[37789]: pgmap v3947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:08 smithi044 ceph-mon[32404]: pgmap v3947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:08 smithi116 ceph-mon[37097]: pgmap v3947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:10 smithi171 ceph-mon[37789]: pgmap v3948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6495.739723 secs 2024-07-26T10:29:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:10 smithi044 ceph-mon[32404]: pgmap v3948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6495.739723 secs 2024-07-26T10:29:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:10 smithi116 ceph-mon[37097]: pgmap v3948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6495.739723 secs 2024-07-26T10:29:12.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:12.415+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:29:12.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:12.415+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:29:12.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:12.415+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:12 smithi171 ceph-mon[37789]: pgmap v3949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:12 smithi044 ceph-mon[32404]: pgmap v3949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:12 smithi116 ceph-mon[37097]: pgmap v3949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:14 smithi171 ceph-mon[37789]: pgmap v3950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:14 smithi044 ceph-mon[32404]: pgmap v3950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:15.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:14 smithi116 ceph-mon[37097]: pgmap v3950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:15.830+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:29:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:15.830+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:29:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:15.831+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:29:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:15.833+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:29:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6500.739883 secs 2024-07-26T10:29:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6500.739883 secs 2024-07-26T10:29:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6500.739883 secs 2024-07-26T10:29:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:16 smithi044 ceph-mon[32404]: pgmap v3951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:16 smithi116 ceph-mon[37097]: pgmap v3951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:16 smithi171 ceph-mon[37789]: pgmap v3951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:18 smithi044 ceph-mon[32404]: pgmap v3952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:18 smithi116 ceph-mon[37097]: pgmap v3952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:18 smithi171 ceph-mon[37789]: pgmap v3952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:21.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:20 smithi171 ceph-mon[37789]: pgmap v3953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:21.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6505.740047 secs 2024-07-26T10:29:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:20 smithi044 ceph-mon[32404]: pgmap v3953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6505.740047 secs 2024-07-26T10:29:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:20 smithi116 ceph-mon[37097]: pgmap v3953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6505.740047 secs 2024-07-26T10:29:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:22 smithi044 ceph-mon[32404]: pgmap v3954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:22 smithi116 ceph-mon[37097]: pgmap v3954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:22 smithi171 ceph-mon[37789]: pgmap v3954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:24 smithi044 ceph-mon[32404]: pgmap v3955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:24 smithi116 ceph-mon[37097]: pgmap v3955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:24 smithi171 ceph-mon[37789]: pgmap v3955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6510.740179 secs 2024-07-26T10:29:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6510.740179 secs 2024-07-26T10:29:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6510.740179 secs 2024-07-26T10:29:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:26 smithi044 ceph-mon[32404]: pgmap v3956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:29:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:26 smithi116 ceph-mon[37097]: pgmap v3956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:29:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:26 smithi171 ceph-mon[37789]: pgmap v3956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:29:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:27.418+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:29:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:27.418+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:27.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:27.418+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:29:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:28 smithi044 ceph-mon[32404]: pgmap v3957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:28 smithi171 ceph-mon[37789]: pgmap v3957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:28 smithi116 ceph-mon[37097]: pgmap v3957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:30 smithi171 ceph-mon[37789]: pgmap v3958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6515.740341 secs 2024-07-26T10:29:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:30 smithi044 ceph-mon[32404]: pgmap v3958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6515.740341 secs 2024-07-26T10:29:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:30 smithi116 ceph-mon[37097]: pgmap v3958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6515.740341 secs 2024-07-26T10:29:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:32 smithi171 ceph-mon[37789]: pgmap v3959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:32 smithi044 ceph-mon[32404]: pgmap v3959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:33.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:32 smithi116 ceph-mon[37097]: pgmap v3959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:29:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:34 smithi171 ceph-mon[37789]: pgmap v3960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:34 smithi044 ceph-mon[32404]: pgmap v3960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:34 smithi116 ceph-mon[37097]: pgmap v3960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:29:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6520.740505 secs 2024-07-26T10:29:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6520.740505 secs 2024-07-26T10:29:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6520.740505 secs 2024-07-26T10:29:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:36 smithi171 ceph-mon[37789]: pgmap v3961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:36 smithi044 ceph-mon[32404]: pgmap v3961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:36 smithi116 ceph-mon[37097]: pgmap v3961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:38 smithi171 ceph-mon[37789]: pgmap v3962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:38 smithi044 ceph-mon[32404]: pgmap v3962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:38 smithi116 ceph-mon[37097]: pgmap v3962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:39 smithi171 ceph-mon[37789]: pgmap v3963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:29:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:39 smithi171 ceph-mon[37789]: pgmap v3964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:29:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:39 smithi044 ceph-mon[32404]: pgmap v3963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:29:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:39 smithi044 ceph-mon[32404]: pgmap v3964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:29:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:39 smithi116 ceph-mon[37097]: pgmap v3963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:29:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:39 smithi116 ceph-mon[37097]: pgmap v3964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:29:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6525.740707 secs 2024-07-26T10:29:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6525.740707 secs 2024-07-26T10:29:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6525.740707 secs 2024-07-26T10:29:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:41 smithi044 ceph-mon[32404]: pgmap v3965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:29:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:41 smithi116 ceph-mon[37097]: pgmap v3965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:29:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:41 smithi171 ceph-mon[37789]: pgmap v3965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:29:42.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:42.423+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:29:42.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:42.423+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:29:42.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:42.423+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:43 smithi044 ceph-mon[32404]: pgmap v3966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:43 smithi116 ceph-mon[37097]: pgmap v3966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:43 smithi171 ceph-mon[37789]: pgmap v3966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:29:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:45 smithi044 ceph-mon[32404]: pgmap v3967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:45 smithi116 ceph-mon[37097]: pgmap v3967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:45 smithi171 ceph-mon[37789]: pgmap v3967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6530.740859 secs 2024-07-26T10:29:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6530.740859 secs 2024-07-26T10:29:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6530.740859 secs 2024-07-26T10:29:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:47 smithi044 ceph-mon[32404]: pgmap v3968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:47 smithi116 ceph-mon[37097]: pgmap v3968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:47 smithi171 ceph-mon[37789]: pgmap v3968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:29:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:49 smithi044 ceph-mon[32404]: pgmap v3969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:29:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:49 smithi116 ceph-mon[37097]: pgmap v3969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:29:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:49 smithi171 ceph-mon[37789]: pgmap v3969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:29:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6535.741023 secs 2024-07-26T10:29:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6535.741023 secs 2024-07-26T10:29:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6535.741023 secs 2024-07-26T10:29:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:51 smithi044 ceph-mon[32404]: pgmap v3970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:51 smithi116 ceph-mon[37097]: pgmap v3970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:51 smithi171 ceph-mon[37789]: pgmap v3970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:29:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:53 smithi044 ceph-mon[32404]: pgmap v3971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:53 smithi116 ceph-mon[37097]: pgmap v3971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:53 smithi171 ceph-mon[37789]: pgmap v3971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:55 smithi044 ceph-mon[32404]: pgmap v3972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:55 smithi116 ceph-mon[37097]: pgmap v3972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:55 smithi171 ceph-mon[37789]: pgmap v3972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:29:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6540.741180 secs 2024-07-26T10:29:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6540.741180 secs 2024-07-26T10:29:56.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6540.741180 secs 2024-07-26T10:29:57.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:57 smithi044 ceph-mon[32404]: pgmap v3973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:57.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:57.432+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:29:57.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:57.432+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:57.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:29:57.432+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:29:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:57 smithi116 ceph-mon[37097]: pgmap v3973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:57 smithi171 ceph-mon[37789]: pgmap v3973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:29:59 smithi044 ceph-mon[32404]: pgmap v3974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:29:59 smithi116 ceph-mon[37097]: pgmap v3974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:29:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:29:59 smithi171 ceph-mon[37789]: pgmap v3974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:30:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:30:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:30:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6545.741356 secs 2024-07-26T10:30:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:01 smithi044 ceph-mon[32404]: pgmap v3975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6545.741356 secs 2024-07-26T10:30:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:01 smithi116 ceph-mon[37097]: pgmap v3975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6545.741356 secs 2024-07-26T10:30:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:01 smithi171 ceph-mon[37789]: pgmap v3975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:03 smithi044 ceph-mon[32404]: pgmap v3976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:03 smithi116 ceph-mon[37097]: pgmap v3976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:03 smithi171 ceph-mon[37789]: pgmap v3976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:05 smithi044 ceph-mon[32404]: pgmap v3977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:30:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:05 smithi116 ceph-mon[37097]: pgmap v3977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:30:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:05 smithi171 ceph-mon[37789]: pgmap v3977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:30:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6550.741524 secs 2024-07-26T10:30:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6550.741524 secs 2024-07-26T10:30:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6550.741524 secs 2024-07-26T10:30:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:07 smithi044 ceph-mon[32404]: pgmap v3978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:07 smithi116 ceph-mon[37097]: pgmap v3978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:07 smithi171 ceph-mon[37789]: pgmap v3978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:09 smithi044 ceph-mon[32404]: pgmap v3979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:09 smithi116 ceph-mon[37097]: pgmap v3979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:09 smithi171 ceph-mon[37789]: pgmap v3979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6555.741724 secs 2024-07-26T10:30:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6555.741724 secs 2024-07-26T10:30:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6555.741724 secs 2024-07-26T10:30:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:11 smithi044 ceph-mon[32404]: pgmap v3980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:30:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:11 smithi116 ceph-mon[37097]: pgmap v3980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:30:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:11 smithi171 ceph-mon[37789]: pgmap v3980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:30:12.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:12.436+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:30:12.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:12.436+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:12.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:12.436+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:30:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:13 smithi044 ceph-mon[32404]: pgmap v3981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:13 smithi116 ceph-mon[37097]: pgmap v3981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:13 smithi171 ceph-mon[37789]: pgmap v3981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:30:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:15 smithi044 ceph-mon[32404]: pgmap v3982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:15 smithi116 ceph-mon[37097]: pgmap v3982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:15 smithi171 ceph-mon[37789]: pgmap v3982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6560.741875 secs 2024-07-26T10:30:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6560.741875 secs 2024-07-26T10:30:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6560.741875 secs 2024-07-26T10:30:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:17 smithi044 ceph-mon[32404]: pgmap v3983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:17 smithi116 ceph-mon[37097]: pgmap v3983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:17 smithi171 ceph-mon[37789]: pgmap v3983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:19.294 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:19 smithi116 ceph-mon[37097]: pgmap v3984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:19 smithi044 ceph-mon[32404]: pgmap v3984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:19 smithi171 ceph-mon[37789]: pgmap v3984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:21.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6565.742044 secs 2024-07-26T10:30:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6565.742044 secs 2024-07-26T10:30:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6565.742044 secs 2024-07-26T10:30:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:21 smithi044 ceph-mon[32404]: pgmap v3985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:21 smithi116 ceph-mon[37097]: pgmap v3985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:21 smithi171 ceph-mon[37789]: pgmap v3985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:23 smithi044 ceph-mon[32404]: pgmap v3986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:23 smithi116 ceph-mon[37097]: pgmap v3986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:23 smithi171 ceph-mon[37789]: pgmap v3986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:25 smithi044 ceph-mon[32404]: pgmap v3987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:25 smithi116 ceph-mon[37097]: pgmap v3987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:25 smithi171 ceph-mon[37789]: pgmap v3987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6570.742195 secs 2024-07-26T10:30:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6570.742195 secs 2024-07-26T10:30:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6570.742195 secs 2024-07-26T10:30:27.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:27 smithi044 ceph-mon[32404]: pgmap v3988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:27.438+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:30:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:27.438+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:27.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:27.438+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:30:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:27 smithi116 ceph-mon[37097]: pgmap v3988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:27 smithi171 ceph-mon[37789]: pgmap v3988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:29 smithi044 ceph-mon[32404]: pgmap v3989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:29 smithi116 ceph-mon[37097]: pgmap v3989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:29 smithi171 ceph-mon[37789]: pgmap v3989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6575.742350 secs 2024-07-26T10:30:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6575.742350 secs 2024-07-26T10:30:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6575.742350 secs 2024-07-26T10:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:31 smithi044 ceph-mon[32404]: pgmap v3990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:31 smithi116 ceph-mon[37097]: pgmap v3990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:31 smithi171 ceph-mon[37789]: pgmap v3990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:33 smithi044 ceph-mon[32404]: pgmap v3991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:33 smithi116 ceph-mon[37097]: pgmap v3991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:33 smithi171 ceph-mon[37789]: pgmap v3991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:35 smithi044 ceph-mon[32404]: pgmap v3992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:35 smithi116 ceph-mon[37097]: pgmap v3992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:35 smithi171 ceph-mon[37789]: pgmap v3992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6580.742492 secs 2024-07-26T10:30:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6580.742492 secs 2024-07-26T10:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6580.742492 secs 2024-07-26T10:30:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:37 smithi044 ceph-mon[32404]: pgmap v3993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:37 smithi116 ceph-mon[37097]: pgmap v3993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:37 smithi171 ceph-mon[37789]: pgmap v3993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:39 smithi044 ceph-mon[32404]: pgmap v3994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:39 smithi116 ceph-mon[37097]: pgmap v3994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:39 smithi171 ceph-mon[37789]: pgmap v3994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:41 smithi044 ceph-mon[32404]: pgmap v3995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:30:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:41 smithi044 ceph-mon[32404]: pgmap v3996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:30:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6585.742656 secs 2024-07-26T10:30:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:41 smithi116 ceph-mon[37097]: pgmap v3995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:30:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:41 smithi116 ceph-mon[37097]: pgmap v3996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:30:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6585.742656 secs 2024-07-26T10:30:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:41 smithi171 ceph-mon[37789]: pgmap v3995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:30:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:41 smithi171 ceph-mon[37789]: pgmap v3996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:30:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6585.742656 secs 2024-07-26T10:30:42.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:42.442+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:30:42.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:42.442+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:42.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:42.442+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:30:43.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:43 smithi171 ceph-mon[37789]: pgmap v3997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:43 smithi044 ceph-mon[32404]: pgmap v3997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:43 smithi116 ceph-mon[37097]: pgmap v3997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:45 smithi171 ceph-mon[37789]: pgmap v3998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:45 smithi044 ceph-mon[32404]: pgmap v3998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:45 smithi116 ceph-mon[37097]: pgmap v3998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6590.742809 secs 2024-07-26T10:30:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6590.742809 secs 2024-07-26T10:30:46.895 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6590.742809 secs 2024-07-26T10:30:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:47 smithi171 ceph-mon[37789]: pgmap v3999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:47 smithi044 ceph-mon[32404]: pgmap v3999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:47 smithi116 ceph-mon[37097]: pgmap v3999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:49 smithi044 ceph-mon[32404]: pgmap v4000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:49 smithi116 ceph-mon[37097]: pgmap v4000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:49 smithi171 ceph-mon[37789]: pgmap v4000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:30:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:51 smithi171 ceph-mon[37789]: pgmap v4001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6595.742986 secs 2024-07-26T10:30:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:51 smithi044 ceph-mon[32404]: pgmap v4001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6595.742986 secs 2024-07-26T10:30:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:51 smithi116 ceph-mon[37097]: pgmap v4001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:30:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6595.742986 secs 2024-07-26T10:30:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:53 smithi171 ceph-mon[37789]: pgmap v4002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:53 smithi044 ceph-mon[32404]: pgmap v4002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:53 smithi116 ceph-mon[37097]: pgmap v4002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:30:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:55 smithi171 ceph-mon[37789]: pgmap v4003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:55 smithi044 ceph-mon[32404]: pgmap v4003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:55 smithi116 ceph-mon[37097]: pgmap v4003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:30:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6600.743153 secs 2024-07-26T10:30:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6600.743153 secs 2024-07-26T10:30:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6600.743153 secs 2024-07-26T10:30:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:57.449+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:30:57.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:57.449+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:30:57.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:30:57.449+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:57.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:57 smithi044 ceph-mon[32404]: pgmap v4004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:57 smithi116 ceph-mon[37097]: pgmap v4004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:57 smithi171 ceph-mon[37789]: pgmap v4004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:30:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:30:59 smithi044 ceph-mon[32404]: pgmap v4005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:30:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:30:59 smithi116 ceph-mon[37097]: pgmap v4005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:30:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:30:59 smithi171 ceph-mon[37789]: pgmap v4005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:01 smithi044 ceph-mon[32404]: pgmap v4006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6605.743325 secs 2024-07-26T10:31:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:01 smithi116 ceph-mon[37097]: pgmap v4006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6605.743325 secs 2024-07-26T10:31:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:01 smithi171 ceph-mon[37789]: pgmap v4006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6605.743325 secs 2024-07-26T10:31:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:03 smithi044 ceph-mon[32404]: pgmap v4007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:03 smithi116 ceph-mon[37097]: pgmap v4007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:03 smithi171 ceph-mon[37789]: pgmap v4007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:05 smithi044 ceph-mon[32404]: pgmap v4008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:05 smithi116 ceph-mon[37097]: pgmap v4008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:05 smithi171 ceph-mon[37789]: pgmap v4008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6610.743478 secs 2024-07-26T10:31:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6610.743478 secs 2024-07-26T10:31:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6610.743478 secs 2024-07-26T10:31:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:07 smithi044 ceph-mon[32404]: pgmap v4009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:07 smithi116 ceph-mon[37097]: pgmap v4009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:07 smithi171 ceph-mon[37789]: pgmap v4009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:09 smithi044 ceph-mon[32404]: pgmap v4010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:09 smithi116 ceph-mon[37097]: pgmap v4010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:09 smithi171 ceph-mon[37789]: pgmap v4010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:11 smithi044 ceph-mon[32404]: pgmap v4011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6615.743601 secs 2024-07-26T10:31:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:11 smithi116 ceph-mon[37097]: pgmap v4011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6615.743601 secs 2024-07-26T10:31:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:11 smithi171 ceph-mon[37789]: pgmap v4011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6615.743601 secs 2024-07-26T10:31:12.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:12.453+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:31:12.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:12.453+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:12.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:12.453+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:31:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:13 smithi044 ceph-mon[32404]: pgmap v4012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:13 smithi116 ceph-mon[37097]: pgmap v4012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:13 smithi171 ceph-mon[37789]: pgmap v4012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:15 smithi044 ceph-mon[32404]: pgmap v4013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:15 smithi116 ceph-mon[37097]: pgmap v4013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:15 smithi171 ceph-mon[37789]: pgmap v4013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6620.743773 secs 2024-07-26T10:31:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6620.743773 secs 2024-07-26T10:31:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6620.743773 secs 2024-07-26T10:31:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:17 smithi044 ceph-mon[32404]: pgmap v4014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:17 smithi116 ceph-mon[37097]: pgmap v4014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:17 smithi171 ceph-mon[37789]: pgmap v4014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:19 smithi044 ceph-mon[32404]: pgmap v4015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:19 smithi116 ceph-mon[37097]: pgmap v4015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:19 smithi171 ceph-mon[37789]: pgmap v4015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:21 smithi044 ceph-mon[32404]: pgmap v4016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6625.743926 secs 2024-07-26T10:31:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:21 smithi116 ceph-mon[37097]: pgmap v4016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6625.743926 secs 2024-07-26T10:31:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:21 smithi171 ceph-mon[37789]: pgmap v4016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6625.743926 secs 2024-07-26T10:31:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:23 smithi044 ceph-mon[32404]: pgmap v4017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:23 smithi116 ceph-mon[37097]: pgmap v4017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:23 smithi171 ceph-mon[37789]: pgmap v4017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:25 smithi044 ceph-mon[32404]: pgmap v4018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:25 smithi116 ceph-mon[37097]: pgmap v4018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:25 smithi171 ceph-mon[37789]: pgmap v4018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6630.744071 secs 2024-07-26T10:31:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6630.744071 secs 2024-07-26T10:31:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6630.744071 secs 2024-07-26T10:31:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:27.454+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:31:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:27.454+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:27.454+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:31:27.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:27 smithi044 ceph-mon[32404]: pgmap v4019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:27 smithi116 ceph-mon[37097]: pgmap v4019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:27 smithi171 ceph-mon[37789]: pgmap v4019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:29 smithi044 ceph-mon[32404]: pgmap v4020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:29 smithi116 ceph-mon[37097]: pgmap v4020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:29 smithi171 ceph-mon[37789]: pgmap v4020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:31 smithi044 ceph-mon[32404]: pgmap v4021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6635.744240 secs 2024-07-26T10:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:31 smithi116 ceph-mon[37097]: pgmap v4021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6635.744240 secs 2024-07-26T10:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:31 smithi171 ceph-mon[37789]: pgmap v4021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6635.744240 secs 2024-07-26T10:31:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:33 smithi044 ceph-mon[32404]: pgmap v4022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:33 smithi116 ceph-mon[37097]: pgmap v4022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:33 smithi171 ceph-mon[37789]: pgmap v4022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:31:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:35 smithi044 ceph-mon[32404]: pgmap v4023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:35 smithi116 ceph-mon[37097]: pgmap v4023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:35 smithi171 ceph-mon[37789]: pgmap v4023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6640.744414 secs 2024-07-26T10:31:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6640.744414 secs 2024-07-26T10:31:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6640.744414 secs 2024-07-26T10:31:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:37 smithi044 ceph-mon[32404]: pgmap v4024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:37 smithi116 ceph-mon[37097]: pgmap v4024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:37 smithi171 ceph-mon[37789]: pgmap v4024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:39 smithi044 ceph-mon[32404]: pgmap v4025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:39 smithi116 ceph-mon[37097]: pgmap v4025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:39 smithi171 ceph-mon[37789]: pgmap v4025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:31:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:41 smithi116 ceph-mon[37097]: pgmap v4026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6645.744580 secs 2024-07-26T10:31:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:41 smithi171 ceph-mon[37789]: pgmap v4026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6645.744580 secs 2024-07-26T10:31:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:41 smithi044 ceph-mon[32404]: pgmap v4026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6645.744580 secs 2024-07-26T10:31:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:42.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:42.457+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:31:42.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:42.457+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:42.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:42.457+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:31:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:42 smithi044 ceph-mon[32404]: pgmap v4027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:31:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:42 smithi044 ceph-mon[32404]: pgmap v4028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:31:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:31:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:42 smithi171 ceph-mon[37789]: pgmap v4027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:31:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:42 smithi171 ceph-mon[37789]: pgmap v4028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:31:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:31:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:42 smithi116 ceph-mon[37097]: pgmap v4027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:31:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:42 smithi116 ceph-mon[37097]: pgmap v4028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:31:43.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:31:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:44 smithi171 ceph-mon[37789]: pgmap v4029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:44 smithi044 ceph-mon[32404]: pgmap v4029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:44 smithi116 ceph-mon[37097]: pgmap v4029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6650.744744 secs 2024-07-26T10:31:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:46 smithi171 ceph-mon[37789]: pgmap v4030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6650.744744 secs 2024-07-26T10:31:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:46 smithi044 ceph-mon[32404]: pgmap v4030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6650.744744 secs 2024-07-26T10:31:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:46 smithi116 ceph-mon[37097]: pgmap v4030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:48 smithi171 ceph-mon[37789]: pgmap v4031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:31:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:48 smithi044 ceph-mon[32404]: pgmap v4031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:31:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:48 smithi116 ceph-mon[37097]: pgmap v4031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:31:50.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:50 smithi116 ceph-mon[37097]: pgmap v4032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:50 smithi171 ceph-mon[37789]: pgmap v4032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:50 smithi044 ceph-mon[32404]: pgmap v4032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:31:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6655.744843 secs 2024-07-26T10:31:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6655.744843 secs 2024-07-26T10:31:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6655.744843 secs 2024-07-26T10:31:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:52 smithi171 ceph-mon[37789]: pgmap v4033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:31:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:52 smithi044 ceph-mon[32404]: pgmap v4033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:31:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:52 smithi116 ceph-mon[37097]: pgmap v4033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:31:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:54 smithi171 ceph-mon[37789]: pgmap v4034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:31:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:54 smithi044 ceph-mon[32404]: pgmap v4034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:31:55.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:54 smithi116 ceph-mon[37097]: pgmap v4034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:31:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6660.745008 secs 2024-07-26T10:31:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:56 smithi171 ceph-mon[37789]: pgmap v4035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6660.745008 secs 2024-07-26T10:31:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:56 smithi044 ceph-mon[32404]: pgmap v4035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6660.745008 secs 2024-07-26T10:31:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:56 smithi116 ceph-mon[37097]: pgmap v4035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:31:57.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:57.460+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:31:57.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:57.460+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:57.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:31:57.460+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:31:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:31:58 smithi171 ceph-mon[37789]: pgmap v4036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:31:58 smithi044 ceph-mon[32404]: pgmap v4036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:31:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:31:58 smithi116 ceph-mon[37097]: pgmap v4036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:00 smithi044 ceph-mon[32404]: pgmap v4037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:00 smithi116 ceph-mon[37097]: pgmap v4037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:01.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:00 smithi171 ceph-mon[37789]: pgmap v4037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6665.745177 secs 2024-07-26T10:32:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6665.745177 secs 2024-07-26T10:32:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6665.745177 secs 2024-07-26T10:32:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:02 smithi044 ceph-mon[32404]: pgmap v4038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:03.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:02 smithi116 ceph-mon[37097]: pgmap v4038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:02 smithi171 ceph-mon[37789]: pgmap v4038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:04 smithi044 ceph-mon[32404]: pgmap v4039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:04 smithi116 ceph-mon[37097]: pgmap v4039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:04 smithi171 ceph-mon[37789]: pgmap v4039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:05.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6670.745329 secs 2024-07-26T10:32:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6670.745329 secs 2024-07-26T10:32:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6670.745329 secs 2024-07-26T10:32:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:06 smithi044 ceph-mon[32404]: pgmap v4040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:06 smithi116 ceph-mon[37097]: pgmap v4040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:06 smithi171 ceph-mon[37789]: pgmap v4040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:08 smithi044 ceph-mon[32404]: pgmap v4041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:08 smithi116 ceph-mon[37097]: pgmap v4041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:08 smithi171 ceph-mon[37789]: pgmap v4041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:10 smithi044 ceph-mon[32404]: pgmap v4042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6675.745494 secs 2024-07-26T10:32:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:10 smithi116 ceph-mon[37097]: pgmap v4042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6675.745494 secs 2024-07-26T10:32:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:10 smithi171 ceph-mon[37789]: pgmap v4042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6675.745494 secs 2024-07-26T10:32:12.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:12.464+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:32:12.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:12.464+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:12.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:12.464+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:32:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:12 smithi044 ceph-mon[32404]: pgmap v4043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:12 smithi116 ceph-mon[37097]: pgmap v4043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:12 smithi171 ceph-mon[37789]: pgmap v4043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:14 smithi044 ceph-mon[32404]: pgmap v4044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:14 smithi116 ceph-mon[37097]: pgmap v4044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:14 smithi171 ceph-mon[37789]: pgmap v4044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:15.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:15.862+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:32:15.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:15.862+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:32:15.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:15.863+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:32:15.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:15.863+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:32:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6680.745659 secs 2024-07-26T10:32:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6680.745659 secs 2024-07-26T10:32:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6680.745659 secs 2024-07-26T10:32:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:16 smithi044 ceph-mon[32404]: pgmap v4045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:16 smithi116 ceph-mon[37097]: pgmap v4045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:16 smithi171 ceph-mon[37789]: pgmap v4045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:18 smithi044 ceph-mon[32404]: pgmap v4046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:18 smithi116 ceph-mon[37097]: pgmap v4046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:18 smithi171 ceph-mon[37789]: pgmap v4046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:20 smithi044 ceph-mon[32404]: pgmap v4047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:21.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6685.745824 secs 2024-07-26T10:32:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:20 smithi116 ceph-mon[37097]: pgmap v4047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6685.745824 secs 2024-07-26T10:32:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:20 smithi171 ceph-mon[37789]: pgmap v4047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6685.745824 secs 2024-07-26T10:32:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:22 smithi044 ceph-mon[32404]: pgmap v4048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:22 smithi116 ceph-mon[37097]: pgmap v4048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:22 smithi171 ceph-mon[37789]: pgmap v4048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:24 smithi044 ceph-mon[32404]: pgmap v4049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:24 smithi116 ceph-mon[37097]: pgmap v4049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:24 smithi171 ceph-mon[37789]: pgmap v4049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6690.745985 secs 2024-07-26T10:32:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6690.745985 secs 2024-07-26T10:32:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6690.745985 secs 2024-07-26T10:32:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:26 smithi044 ceph-mon[32404]: pgmap v4050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:26 smithi116 ceph-mon[37097]: pgmap v4050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:26 smithi171 ceph-mon[37789]: pgmap v4050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:27.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:27.473+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:32:27.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:27.473+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:27.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:27.473+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:32:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:28 smithi044 ceph-mon[32404]: pgmap v4051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:28 smithi116 ceph-mon[37097]: pgmap v4051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:28 smithi171 ceph-mon[37789]: pgmap v4051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:30 smithi044 ceph-mon[32404]: pgmap v4052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6695.746102 secs 2024-07-26T10:32:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:30 smithi116 ceph-mon[37097]: pgmap v4052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6695.746102 secs 2024-07-26T10:32:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:30 smithi171 ceph-mon[37789]: pgmap v4052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6695.746102 secs 2024-07-26T10:32:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:32 smithi044 ceph-mon[32404]: pgmap v4053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:32 smithi116 ceph-mon[37097]: pgmap v4053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:32 smithi171 ceph-mon[37789]: pgmap v4053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:34 smithi044 ceph-mon[32404]: pgmap v4054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:34 smithi116 ceph-mon[37097]: pgmap v4054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:34 smithi171 ceph-mon[37789]: pgmap v4054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:32:36.002 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6700.746243 secs 2024-07-26T10:32:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6700.746243 secs 2024-07-26T10:32:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6700.746243 secs 2024-07-26T10:32:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:36 smithi044 ceph-mon[32404]: pgmap v4055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:36 smithi116 ceph-mon[37097]: pgmap v4055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:36 smithi171 ceph-mon[37789]: pgmap v4055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:38 smithi044 ceph-mon[32404]: pgmap v4056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:38 smithi116 ceph-mon[37097]: pgmap v4056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:38 smithi171 ceph-mon[37789]: pgmap v4056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:32:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:40 smithi044 ceph-mon[32404]: pgmap v4057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6705.746347 secs 2024-07-26T10:32:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:40 smithi116 ceph-mon[37097]: pgmap v4057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6705.746347 secs 2024-07-26T10:32:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:40 smithi171 ceph-mon[37789]: pgmap v4057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:32:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6705.746347 secs 2024-07-26T10:32:42.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:42.476+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:32:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:42.476+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:42.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:42.476+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:32:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:42 smithi171 ceph-mon[37789]: pgmap v4058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:42 smithi044 ceph-mon[32404]: pgmap v4058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:43.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:42 smithi116 ceph-mon[37097]: pgmap v4058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:32:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:43 smithi171 ceph-mon[37789]: pgmap v4059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:32:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:43 smithi171 ceph-mon[37789]: pgmap v4060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:44.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:32:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:43 smithi044 ceph-mon[32404]: pgmap v4059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:32:44.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:43 smithi044 ceph-mon[32404]: pgmap v4060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:44.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:32:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:43 smithi116 ceph-mon[37097]: pgmap v4059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:32:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:43 smithi116 ceph-mon[37097]: pgmap v4060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:32:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:32:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:45 smithi171 ceph-mon[37789]: pgmap v4061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6710.746482 secs 2024-07-26T10:32:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:45 smithi044 ceph-mon[32404]: pgmap v4061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6710.746482 secs 2024-07-26T10:32:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:45 smithi116 ceph-mon[37097]: pgmap v4061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6710.746482 secs 2024-07-26T10:32:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:47 smithi171 ceph-mon[37789]: pgmap v4062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:47 smithi044 ceph-mon[32404]: pgmap v4062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:47 smithi116 ceph-mon[37097]: pgmap v4062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:50.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:49 smithi116 ceph-mon[37097]: pgmap v4063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:49 smithi171 ceph-mon[37789]: pgmap v4063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:49 smithi044 ceph-mon[32404]: pgmap v4063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:32:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6715.746595 secs 2024-07-26T10:32:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6715.746595 secs 2024-07-26T10:32:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6715.746595 secs 2024-07-26T10:32:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:51 smithi171 ceph-mon[37789]: pgmap v4064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:32:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:51 smithi044 ceph-mon[32404]: pgmap v4064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:32:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:51 smithi116 ceph-mon[37097]: pgmap v4064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:32:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:53 smithi171 ceph-mon[37789]: pgmap v4065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:53 smithi044 ceph-mon[32404]: pgmap v4065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:53 smithi116 ceph-mon[37097]: pgmap v4065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:55 smithi171 ceph-mon[37789]: pgmap v4066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6720.746753 secs 2024-07-26T10:32:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:55 smithi044 ceph-mon[32404]: pgmap v4066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6720.746753 secs 2024-07-26T10:32:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:55 smithi116 ceph-mon[37097]: pgmap v4066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:32:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6720.746753 secs 2024-07-26T10:32:57.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:57.481+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:32:57.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:57.481+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:32:57.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:32:57.481+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:57 smithi171 ceph-mon[37789]: pgmap v4067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:32:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:57 smithi044 ceph-mon[32404]: pgmap v4067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:32:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:57 smithi116 ceph-mon[37097]: pgmap v4067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:32:59 smithi044 ceph-mon[32404]: pgmap v4068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:32:59 smithi116 ceph-mon[37097]: pgmap v4068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:32:59 smithi171 ceph-mon[37789]: pgmap v4068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6725.746917 secs 2024-07-26T10:33:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6725.746917 secs 2024-07-26T10:33:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6725.746917 secs 2024-07-26T10:33:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:01 smithi044 ceph-mon[32404]: pgmap v4069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:33:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:01 smithi116 ceph-mon[37097]: pgmap v4069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:33:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:01 smithi171 ceph-mon[37789]: pgmap v4069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:33:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:03 smithi044 ceph-mon[32404]: pgmap v4070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:03 smithi116 ceph-mon[37097]: pgmap v4070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:03 smithi171 ceph-mon[37789]: pgmap v4070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:33:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:05 smithi044 ceph-mon[32404]: pgmap v4071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6730.747068 secs 2024-07-26T10:33:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:05 smithi116 ceph-mon[37097]: pgmap v4071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6730.747068 secs 2024-07-26T10:33:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:05 smithi171 ceph-mon[37789]: pgmap v4071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6730.747068 secs 2024-07-26T10:33:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:07 smithi044 ceph-mon[32404]: pgmap v4072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:07 smithi116 ceph-mon[37097]: pgmap v4072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:07 smithi171 ceph-mon[37789]: pgmap v4072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:09 smithi044 ceph-mon[32404]: pgmap v4073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:09 smithi116 ceph-mon[37097]: pgmap v4073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:09 smithi171 ceph-mon[37789]: pgmap v4073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6735.747186 secs 2024-07-26T10:33:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6735.747186 secs 2024-07-26T10:33:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6735.747186 secs 2024-07-26T10:33:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:11 smithi044 ceph-mon[32404]: pgmap v4074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:11 smithi116 ceph-mon[37097]: pgmap v4074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:11 smithi171 ceph-mon[37789]: pgmap v4074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:12.485+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:33:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:12.485+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:12.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:12.485+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:33:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:13 smithi044 ceph-mon[32404]: pgmap v4075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:13 smithi116 ceph-mon[37097]: pgmap v4075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:13 smithi171 ceph-mon[37789]: pgmap v4075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:15 smithi044 ceph-mon[32404]: pgmap v4076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6740.747352 secs 2024-07-26T10:33:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:15 smithi116 ceph-mon[37097]: pgmap v4076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6740.747352 secs 2024-07-26T10:33:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:15 smithi171 ceph-mon[37789]: pgmap v4076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6740.747352 secs 2024-07-26T10:33:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:17 smithi044 ceph-mon[32404]: pgmap v4077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:17 smithi116 ceph-mon[37097]: pgmap v4077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:17 smithi171 ceph-mon[37789]: pgmap v4077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:19 smithi044 ceph-mon[32404]: pgmap v4078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:19 smithi116 ceph-mon[37097]: pgmap v4078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:19 smithi171 ceph-mon[37789]: pgmap v4078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6745.747513 secs 2024-07-26T10:33:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6745.747513 secs 2024-07-26T10:33:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6745.747513 secs 2024-07-26T10:33:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:21 smithi044 ceph-mon[32404]: pgmap v4079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:21 smithi116 ceph-mon[37097]: pgmap v4079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:21 smithi171 ceph-mon[37789]: pgmap v4079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:23 smithi044 ceph-mon[32404]: pgmap v4080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:23 smithi116 ceph-mon[37097]: pgmap v4080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:23 smithi171 ceph-mon[37789]: pgmap v4080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:25 smithi044 ceph-mon[32404]: pgmap v4081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6750.747681 secs 2024-07-26T10:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:25 smithi116 ceph-mon[37097]: pgmap v4081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6750.747681 secs 2024-07-26T10:33:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:25 smithi171 ceph-mon[37789]: pgmap v4081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6750.747681 secs 2024-07-26T10:33:27.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:27.491+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:33:27.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:27.491+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:33:27.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:27.491+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:27 smithi044 ceph-mon[32404]: pgmap v4082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:27 smithi116 ceph-mon[37097]: pgmap v4082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:27 smithi171 ceph-mon[37789]: pgmap v4082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:29 smithi044 ceph-mon[32404]: pgmap v4083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:29 smithi116 ceph-mon[37097]: pgmap v4083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:29 smithi171 ceph-mon[37789]: pgmap v4083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6755.747828 secs 2024-07-26T10:33:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6755.747828 secs 2024-07-26T10:33:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6755.747828 secs 2024-07-26T10:33:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:31 smithi044 ceph-mon[32404]: pgmap v4084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:31 smithi116 ceph-mon[37097]: pgmap v4084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:31 smithi171 ceph-mon[37789]: pgmap v4084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:33 smithi044 ceph-mon[32404]: pgmap v4085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:33 smithi116 ceph-mon[37097]: pgmap v4085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:33 smithi171 ceph-mon[37789]: pgmap v4085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:35 smithi044 ceph-mon[32404]: pgmap v4086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6760.747992 secs 2024-07-26T10:33:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:35 smithi116 ceph-mon[37097]: pgmap v4086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6760.747992 secs 2024-07-26T10:33:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:35 smithi171 ceph-mon[37789]: pgmap v4086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:33:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6760.747992 secs 2024-07-26T10:33:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:37 smithi044 ceph-mon[32404]: pgmap v4087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:37 smithi116 ceph-mon[37097]: pgmap v4087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:37 smithi171 ceph-mon[37789]: pgmap v4087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:39 smithi044 ceph-mon[32404]: pgmap v4088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:39 smithi116 ceph-mon[37097]: pgmap v4088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:39 smithi171 ceph-mon[37789]: pgmap v4088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6765.748135 secs 2024-07-26T10:33:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6765.748135 secs 2024-07-26T10:33:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6765.748135 secs 2024-07-26T10:33:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:42 smithi044 ceph-mon[32404]: pgmap v4089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:42 smithi116 ceph-mon[37097]: pgmap v4089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:42 smithi171 ceph-mon[37789]: pgmap v4089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:42.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:42.494+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:33:42.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:42.494+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:33:42.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:42.494+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:44 smithi171 ceph-mon[37789]: pgmap v4090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:44 smithi044 ceph-mon[32404]: pgmap v4090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:44 smithi116 ceph-mon[37097]: pgmap v4090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:33:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:45.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:45.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:45 smithi171 ceph-mon[37789]: pgmap v4091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:33:45.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:45 smithi171 ceph-mon[37789]: pgmap v4092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:45.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:33:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:45 smithi044 ceph-mon[32404]: pgmap v4091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:33:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:45 smithi044 ceph-mon[32404]: pgmap v4092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:33:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:45 smithi116 ceph-mon[37097]: pgmap v4091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:33:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:45 smithi116 ceph-mon[37097]: pgmap v4092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:33:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6770.748262 secs 2024-07-26T10:33:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6770.748262 secs 2024-07-26T10:33:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6770.748262 secs 2024-07-26T10:33:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:47 smithi171 ceph-mon[37789]: pgmap v4093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:33:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:47 smithi044 ceph-mon[32404]: pgmap v4093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:33:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:47 smithi116 ceph-mon[37097]: pgmap v4093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:33:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:49 smithi171 ceph-mon[37789]: pgmap v4094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:49 smithi044 ceph-mon[32404]: pgmap v4094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:49 smithi116 ceph-mon[37097]: pgmap v4094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:51.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:51 smithi116 ceph-mon[37097]: pgmap v4095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:51.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6775.748380 secs 2024-07-26T10:33:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:51 smithi171 ceph-mon[37789]: pgmap v4095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6775.748380 secs 2024-07-26T10:33:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:51 smithi044 ceph-mon[32404]: pgmap v4095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6775.748380 secs 2024-07-26T10:33:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:53 smithi171 ceph-mon[37789]: pgmap v4096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:53 smithi044 ceph-mon[32404]: pgmap v4096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:53 smithi116 ceph-mon[37097]: pgmap v4096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:33:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:55 smithi171 ceph-mon[37789]: pgmap v4097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:55 smithi044 ceph-mon[32404]: pgmap v4097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:55 smithi116 ceph-mon[37097]: pgmap v4097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:33:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6780.748473 secs 2024-07-26T10:33:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6780.748473 secs 2024-07-26T10:33:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6780.748473 secs 2024-07-26T10:33:57.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:57 smithi044 ceph-mon[32404]: pgmap v4098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:57.504+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:33:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:57.504+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:33:57.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:33:57.504+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:57 smithi116 ceph-mon[37097]: pgmap v4098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:57 smithi171 ceph-mon[37789]: pgmap v4098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:33:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:33:59 smithi044 ceph-mon[32404]: pgmap v4099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:33:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:33:59 smithi116 ceph-mon[37097]: pgmap v4099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:33:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:33:59 smithi171 ceph-mon[37789]: pgmap v4099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:00 smithi044 ceph-mon[32404]: pgmap v4100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:00 smithi116 ceph-mon[37097]: pgmap v4100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:00 smithi171 ceph-mon[37789]: pgmap v4100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6785.748602 secs 2024-07-26T10:34:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6785.748602 secs 2024-07-26T10:34:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6785.748602 secs 2024-07-26T10:34:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:02 smithi044 ceph-mon[32404]: pgmap v4101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:02 smithi116 ceph-mon[37097]: pgmap v4101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:02 smithi171 ceph-mon[37789]: pgmap v4101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:04 smithi044 ceph-mon[32404]: pgmap v4102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:04 smithi116 ceph-mon[37097]: pgmap v4102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:04 smithi171 ceph-mon[37789]: pgmap v4102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6790.748745 secs 2024-07-26T10:34:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6790.748745 secs 2024-07-26T10:34:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6790.748745 secs 2024-07-26T10:34:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:06 smithi044 ceph-mon[32404]: pgmap v4103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:06 smithi116 ceph-mon[37097]: pgmap v4103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:06 smithi171 ceph-mon[37789]: pgmap v4103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:08 smithi044 ceph-mon[32404]: pgmap v4104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:08 smithi116 ceph-mon[37097]: pgmap v4104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:08.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:08 smithi171 ceph-mon[37789]: pgmap v4104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:10 smithi044 ceph-mon[32404]: pgmap v4105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:10 smithi116 ceph-mon[37097]: pgmap v4105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:10 smithi171 ceph-mon[37789]: pgmap v4105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6795.748906 secs 2024-07-26T10:34:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6795.748906 secs 2024-07-26T10:34:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6795.748906 secs 2024-07-26T10:34:12.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:12.508+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:34:12.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:12.509+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:34:12.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:12.509+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:12.512 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:12 smithi044 ceph-mon[32404]: pgmap v4106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:12 smithi116 ceph-mon[37097]: pgmap v4106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:12 smithi171 ceph-mon[37789]: pgmap v4106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:14 smithi044 ceph-mon[32404]: pgmap v4107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:14 smithi116 ceph-mon[37097]: pgmap v4107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:14 smithi171 ceph-mon[37789]: pgmap v4107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6800.749044 secs 2024-07-26T10:34:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6800.749044 secs 2024-07-26T10:34:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6800.749044 secs 2024-07-26T10:34:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:16 smithi044 ceph-mon[32404]: pgmap v4108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:16 smithi116 ceph-mon[37097]: pgmap v4108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:16 smithi171 ceph-mon[37789]: pgmap v4108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:18 smithi044 ceph-mon[32404]: pgmap v4109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:18 smithi116 ceph-mon[37097]: pgmap v4109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:18 smithi171 ceph-mon[37789]: pgmap v4109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:20 smithi044 ceph-mon[32404]: pgmap v4110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:20 smithi116 ceph-mon[37097]: pgmap v4110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:20 smithi171 ceph-mon[37789]: pgmap v4110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:21.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6805.749211 secs 2024-07-26T10:34:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6805.749211 secs 2024-07-26T10:34:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6805.749211 secs 2024-07-26T10:34:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:22 smithi044 ceph-mon[32404]: pgmap v4111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:22 smithi116 ceph-mon[37097]: pgmap v4111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:22 smithi171 ceph-mon[37789]: pgmap v4111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:24 smithi044 ceph-mon[32404]: pgmap v4112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:24 smithi116 ceph-mon[37097]: pgmap v4112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:24 smithi171 ceph-mon[37789]: pgmap v4112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6810.749359 secs 2024-07-26T10:34:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6810.749359 secs 2024-07-26T10:34:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6810.749359 secs 2024-07-26T10:34:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:26 smithi044 ceph-mon[32404]: pgmap v4113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:26 smithi116 ceph-mon[37097]: pgmap v4113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:26 smithi171 ceph-mon[37789]: pgmap v4113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:27.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:27.518+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:34:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:27.518+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:34:27.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:27.518+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:28 smithi044 ceph-mon[32404]: pgmap v4114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:28 smithi116 ceph-mon[37097]: pgmap v4114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:28 smithi171 ceph-mon[37789]: pgmap v4114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:30 smithi044 ceph-mon[32404]: pgmap v4115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:30 smithi116 ceph-mon[37097]: pgmap v4115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:30 smithi171 ceph-mon[37789]: pgmap v4115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6815.749492 secs 2024-07-26T10:34:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6815.749492 secs 2024-07-26T10:34:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6815.749492 secs 2024-07-26T10:34:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:32 smithi044 ceph-mon[32404]: pgmap v4116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:32 smithi116 ceph-mon[37097]: pgmap v4116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:32 smithi171 ceph-mon[37789]: pgmap v4116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:34 smithi044 ceph-mon[32404]: pgmap v4117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:34 smithi116 ceph-mon[37097]: pgmap v4117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:34 smithi171 ceph-mon[37789]: pgmap v4117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6820.749654 secs 2024-07-26T10:34:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6820.749654 secs 2024-07-26T10:34:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6820.749654 secs 2024-07-26T10:34:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:36 smithi044 ceph-mon[32404]: pgmap v4118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:36 smithi116 ceph-mon[37097]: pgmap v4118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:36 smithi171 ceph-mon[37789]: pgmap v4118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:38 smithi044 ceph-mon[32404]: pgmap v4119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:38 smithi116 ceph-mon[37097]: pgmap v4119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:38 smithi171 ceph-mon[37789]: pgmap v4119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:40 smithi044 ceph-mon[32404]: pgmap v4120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:40 smithi116 ceph-mon[37097]: pgmap v4120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:40 smithi171 ceph-mon[37789]: pgmap v4120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6825.749790 secs 2024-07-26T10:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6825.749790 secs 2024-07-26T10:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6825.749790 secs 2024-07-26T10:34:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:42 smithi044 ceph-mon[32404]: pgmap v4121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:42 smithi116 ceph-mon[37097]: pgmap v4121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:42.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:42.522+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:34:42.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:42.522+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:34:42.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:42.522+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:42 smithi171 ceph-mon[37789]: pgmap v4121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:44 smithi044 ceph-mon[32404]: pgmap v4122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:44 smithi116 ceph-mon[37097]: pgmap v4122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:44 smithi171 ceph-mon[37789]: pgmap v4122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:34:45.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:46.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6830.749907 secs 2024-07-26T10:34:46.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:46 smithi044 ceph-mon[32404]: pgmap v4123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6830.749907 secs 2024-07-26T10:34:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:46 smithi116 ceph-mon[37097]: pgmap v4123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6830.749907 secs 2024-07-26T10:34:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:46 smithi171 ceph-mon[37789]: pgmap v4123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:49.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:49.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:49.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:49 smithi044 ceph-mon[32404]: pgmap v4124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:49.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:49 smithi116 ceph-mon[37097]: pgmap v4124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:49 smithi171 ceph-mon[37789]: pgmap v4124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:34:50.502 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:50 smithi044 ceph-mon[32404]: pgmap v4125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:50 smithi116 ceph-mon[37097]: pgmap v4125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:50 smithi171 ceph-mon[37789]: pgmap v4125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:34:51.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6835.750080 secs 2024-07-26T10:34:51.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:51.425 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: pgmap v4126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:51.426 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: pgmap v4127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:34:51.426 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6835.750080 secs 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: pgmap v4126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: pgmap v4127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:34:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6835.750080 secs 2024-07-26T10:34:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: pgmap v4126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:34:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: pgmap v4127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:34:51.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:34:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:54 smithi171 ceph-mon[37789]: pgmap v4128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:34:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:54 smithi044 ceph-mon[32404]: pgmap v4128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:34:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:54 smithi116 ceph-mon[37097]: pgmap v4128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:34:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:56 smithi171 ceph-mon[37789]: pgmap v4129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:34:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6840.750222 secs 2024-07-26T10:34:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:56 smithi044 ceph-mon[32404]: pgmap v4129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:34:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6840.750222 secs 2024-07-26T10:34:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:56 smithi116 ceph-mon[37097]: pgmap v4129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:34:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6840.750222 secs 2024-07-26T10:34:57.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:57.531+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:34:57.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:57.531+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:34:57.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:34:57.532+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:34:58 smithi116 ceph-mon[37097]: pgmap v4130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:34:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:34:58 smithi171 ceph-mon[37789]: pgmap v4130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:34:58.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:34:58 smithi044 ceph-mon[32404]: pgmap v4130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:35:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:00 smithi044 ceph-mon[32404]: pgmap v4131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:35:00.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:00 smithi116 ceph-mon[37097]: pgmap v4131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:35:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:00 smithi171 ceph-mon[37789]: pgmap v4131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:35:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6845.750360 secs 2024-07-26T10:35:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6845.750360 secs 2024-07-26T10:35:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6845.750360 secs 2024-07-26T10:35:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:02 smithi044 ceph-mon[32404]: pgmap v4132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:35:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:02 smithi116 ceph-mon[37097]: pgmap v4132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:35:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:02 smithi171 ceph-mon[37789]: pgmap v4132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:35:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:04 smithi044 ceph-mon[32404]: pgmap v4133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:04 smithi116 ceph-mon[37097]: pgmap v4133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:04 smithi171 ceph-mon[37789]: pgmap v4133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:06 smithi044 ceph-mon[32404]: pgmap v4134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6850.750523 secs 2024-07-26T10:35:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:06 smithi116 ceph-mon[37097]: pgmap v4134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6850.750523 secs 2024-07-26T10:35:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:06 smithi171 ceph-mon[37789]: pgmap v4134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6850.750523 secs 2024-07-26T10:35:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:07 smithi044 ceph-mon[32404]: pgmap v4135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:07 smithi116 ceph-mon[37097]: pgmap v4135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:07 smithi171 ceph-mon[37789]: pgmap v4135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:10 smithi171 ceph-mon[37789]: pgmap v4136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:10 smithi044 ceph-mon[32404]: pgmap v4136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:10 smithi116 ceph-mon[37097]: pgmap v4136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6855.750678 secs 2024-07-26T10:35:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6855.750678 secs 2024-07-26T10:35:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6855.750678 secs 2024-07-26T10:35:12.413 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:12 smithi044 ceph-mon[32404]: pgmap v4137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:12 smithi116 ceph-mon[37097]: pgmap v4137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:12.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:12.536+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:35:12.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:12.536+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:35:12.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:12.536+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:12 smithi171 ceph-mon[37789]: pgmap v4137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:14 smithi044 ceph-mon[32404]: pgmap v4138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:14 smithi116 ceph-mon[37097]: pgmap v4138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:14 smithi171 ceph-mon[37789]: pgmap v4138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:15.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:15.893+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:35:15.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:15.893+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:35:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:15.894+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:35:15.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:15.894+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:35:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:16 smithi044 ceph-mon[32404]: pgmap v4139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6860.750775 secs 2024-07-26T10:35:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:16 smithi116 ceph-mon[37097]: pgmap v4139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6860.750775 secs 2024-07-26T10:35:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:16 smithi171 ceph-mon[37789]: pgmap v4139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6860.750775 secs 2024-07-26T10:35:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:18 smithi044 ceph-mon[32404]: pgmap v4140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:18 smithi116 ceph-mon[37097]: pgmap v4140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:18 smithi171 ceph-mon[37789]: pgmap v4140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:20 smithi044 ceph-mon[32404]: pgmap v4141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:20 smithi116 ceph-mon[37097]: pgmap v4141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:20 smithi171 ceph-mon[37789]: pgmap v4141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:21.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6865.750922 secs 2024-07-26T10:35:21.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:21 smithi116 ceph-mon[37097]: pgmap v4142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6865.750922 secs 2024-07-26T10:35:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:21 smithi044 ceph-mon[32404]: pgmap v4142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6865.750922 secs 2024-07-26T10:35:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:21 smithi171 ceph-mon[37789]: pgmap v4142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:24 smithi044 ceph-mon[32404]: pgmap v4143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:24 smithi116 ceph-mon[37097]: pgmap v4143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:24 smithi171 ceph-mon[37789]: pgmap v4143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:26.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:26 smithi171 ceph-mon[37789]: pgmap v4144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:26.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6870.751069 secs 2024-07-26T10:35:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:26 smithi044 ceph-mon[32404]: pgmap v4144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6870.751069 secs 2024-07-26T10:35:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:26 smithi116 ceph-mon[37097]: pgmap v4144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6870.751069 secs 2024-07-26T10:35:27.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:27.538+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:35:27.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:27.538+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:35:27.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:27.538+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:28 smithi044 ceph-mon[32404]: pgmap v4145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:28 smithi116 ceph-mon[37097]: pgmap v4145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:28 smithi171 ceph-mon[37789]: pgmap v4145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:30 smithi044 ceph-mon[32404]: pgmap v4146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:30 smithi116 ceph-mon[37097]: pgmap v4146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:30 smithi171 ceph-mon[37789]: pgmap v4146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6875.751231 secs 2024-07-26T10:35:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6875.751231 secs 2024-07-26T10:35:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6875.751231 secs 2024-07-26T10:35:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:32 smithi044 ceph-mon[32404]: pgmap v4147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:32 smithi116 ceph-mon[37097]: pgmap v4147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:32 smithi171 ceph-mon[37789]: pgmap v4147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:33 smithi044 ceph-mon[32404]: pgmap v4148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:33 smithi116 ceph-mon[37097]: pgmap v4148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:33 smithi171 ceph-mon[37789]: pgmap v4148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:36 smithi044 ceph-mon[32404]: pgmap v4149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6880.751392 secs 2024-07-26T10:35:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:36 smithi116 ceph-mon[37097]: pgmap v4149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6880.751392 secs 2024-07-26T10:35:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:36 smithi171 ceph-mon[37789]: pgmap v4149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6880.751392 secs 2024-07-26T10:35:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:38 smithi044 ceph-mon[32404]: pgmap v4150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:38 smithi116 ceph-mon[37097]: pgmap v4150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:38 smithi171 ceph-mon[37789]: pgmap v4150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:40 smithi044 ceph-mon[32404]: pgmap v4151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:40 smithi116 ceph-mon[37097]: pgmap v4151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:40 smithi171 ceph-mon[37789]: pgmap v4151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6885.751530 secs 2024-07-26T10:35:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6885.751530 secs 2024-07-26T10:35:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6885.751530 secs 2024-07-26T10:35:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:42 smithi044 ceph-mon[32404]: pgmap v4152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:42 smithi116 ceph-mon[37097]: pgmap v4152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:42.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:42.542+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:35:42.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:42.542+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:42.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:42.542+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:35:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:42 smithi171 ceph-mon[37789]: pgmap v4152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:43 smithi044 ceph-mon[32404]: pgmap v4153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:43 smithi116 ceph-mon[37097]: pgmap v4153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:43 smithi171 ceph-mon[37789]: pgmap v4153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:35:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:46 smithi044 ceph-mon[32404]: pgmap v4154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6890.751676 secs 2024-07-26T10:35:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:46 smithi116 ceph-mon[37097]: pgmap v4154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6890.751676 secs 2024-07-26T10:35:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:46 smithi171 ceph-mon[37789]: pgmap v4154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:35:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6890.751676 secs 2024-07-26T10:35:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:48 smithi044 ceph-mon[32404]: pgmap v4155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:48 smithi116 ceph-mon[37097]: pgmap v4155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:48 smithi171 ceph-mon[37789]: pgmap v4155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:50 smithi044 ceph-mon[32404]: pgmap v4156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:50 smithi116 ceph-mon[37097]: pgmap v4156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:50 smithi171 ceph-mon[37789]: pgmap v4156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:35:51.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6895.751777 secs 2024-07-26T10:35:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6895.751777 secs 2024-07-26T10:35:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6895.751777 secs 2024-07-26T10:35:52.421 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: pgmap v4157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:35:52.421 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:52.421 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:35:52.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:52 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:35:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: pgmap v4157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:35:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:35:52.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:52 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:35:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: pgmap v4157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:35:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:35:52.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:52 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:35:53.450 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:53 smithi171 ceph-mon[37789]: pgmap v4158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:35:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:53 smithi044 ceph-mon[32404]: pgmap v4158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:35:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:53 smithi116 ceph-mon[37097]: pgmap v4158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: pgmap v4159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: pgmap v4160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: pgmap v4161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:35:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6900.751926 secs 2024-07-26T10:35:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: pgmap v4159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:35:56.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:35:56.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: pgmap v4160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:35:56.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: pgmap v4161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:35:56.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6900.751926 secs 2024-07-26T10:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: pgmap v4159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:35:56.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: pgmap v4160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:35:56.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: pgmap v4161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:35:56.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:35:56.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6900.751926 secs 2024-07-26T10:35:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:57 smithi044 ceph-mon[32404]: pgmap v4162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:35:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:57 smithi116 ceph-mon[37097]: pgmap v4162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:35:57.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:57.551+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:35:57.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:57.551+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:57.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:35:57.551+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:35:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:57 smithi171 ceph-mon[37789]: pgmap v4162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:35:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:35:59 smithi044 ceph-mon[32404]: pgmap v4163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T10:35:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:35:59 smithi116 ceph-mon[37097]: pgmap v4163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T10:35:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:35:59 smithi171 ceph-mon[37789]: pgmap v4163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T10:36:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6905.752087 secs 2024-07-26T10:36:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6905.752087 secs 2024-07-26T10:36:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6905.752087 secs 2024-07-26T10:36:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:01 smithi044 ceph-mon[32404]: pgmap v4164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T10:36:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:01 smithi116 ceph-mon[37097]: pgmap v4164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T10:36:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:01 smithi171 ceph-mon[37789]: pgmap v4164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T10:36:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:03 smithi044 ceph-mon[32404]: pgmap v4165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:36:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:03 smithi116 ceph-mon[37097]: pgmap v4165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:36:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:03 smithi171 ceph-mon[37789]: pgmap v4165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:36:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:05 smithi171 ceph-mon[37789]: pgmap v4166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:36:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:05 smithi044 ceph-mon[32404]: pgmap v4166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:36:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:05 smithi116 ceph-mon[37097]: pgmap v4166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:36:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6910.752255 secs 2024-07-26T10:36:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6910.752255 secs 2024-07-26T10:36:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6910.752255 secs 2024-07-26T10:36:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:07 smithi044 ceph-mon[32404]: pgmap v4167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:36:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:07 smithi116 ceph-mon[37097]: pgmap v4167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:36:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:07 smithi171 ceph-mon[37789]: pgmap v4167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:36:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:09 smithi044 ceph-mon[32404]: pgmap v4168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:09 smithi116 ceph-mon[37097]: pgmap v4168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:09 smithi171 ceph-mon[37789]: pgmap v4168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6915.752428 secs 2024-07-26T10:36:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6915.752428 secs 2024-07-26T10:36:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6915.752428 secs 2024-07-26T10:36:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:11 smithi044 ceph-mon[32404]: pgmap v4169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:11 smithi116 ceph-mon[37097]: pgmap v4169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:11 smithi171 ceph-mon[37789]: pgmap v4169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:12.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:12.555+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:36:12.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:12.555+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:12.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:12.555+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:36:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:13 smithi044 ceph-mon[32404]: pgmap v4170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:13 smithi116 ceph-mon[37097]: pgmap v4170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:13 smithi171 ceph-mon[37789]: pgmap v4170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:15 smithi044 ceph-mon[32404]: pgmap v4171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:15 smithi116 ceph-mon[37097]: pgmap v4171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:15 smithi171 ceph-mon[37789]: pgmap v4171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6920.752577 secs 2024-07-26T10:36:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6920.752577 secs 2024-07-26T10:36:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6920.752577 secs 2024-07-26T10:36:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:17 smithi044 ceph-mon[32404]: pgmap v4172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:17 smithi116 ceph-mon[37097]: pgmap v4172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:17 smithi171 ceph-mon[37789]: pgmap v4172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:19 smithi044 ceph-mon[32404]: pgmap v4173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:19 smithi116 ceph-mon[37097]: pgmap v4173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:19 smithi171 ceph-mon[37789]: pgmap v4173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6925.752719 secs 2024-07-26T10:36:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6925.752719 secs 2024-07-26T10:36:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6925.752719 secs 2024-07-26T10:36:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:21 smithi044 ceph-mon[32404]: pgmap v4174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:21 smithi116 ceph-mon[37097]: pgmap v4174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:21 smithi171 ceph-mon[37789]: pgmap v4174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:23 smithi044 ceph-mon[32404]: pgmap v4175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:23 smithi116 ceph-mon[37097]: pgmap v4175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:23 smithi171 ceph-mon[37789]: pgmap v4175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:25 smithi044 ceph-mon[32404]: pgmap v4176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:25 smithi116 ceph-mon[37097]: pgmap v4176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:25 smithi171 ceph-mon[37789]: pgmap v4176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:26.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6930.752875 secs 2024-07-26T10:36:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6930.752875 secs 2024-07-26T10:36:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6930.752875 secs 2024-07-26T10:36:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:27 smithi044 ceph-mon[32404]: pgmap v4177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:27 smithi116 ceph-mon[37097]: pgmap v4177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:27.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:27.557+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:36:27.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:27.557+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:36:27.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:27.557+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:27 smithi171 ceph-mon[37789]: pgmap v4177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:29 smithi044 ceph-mon[32404]: pgmap v4178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:29 smithi116 ceph-mon[37097]: pgmap v4178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:29 smithi171 ceph-mon[37789]: pgmap v4178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6935.752980 secs 2024-07-26T10:36:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6935.752980 secs 2024-07-26T10:36:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6935.752980 secs 2024-07-26T10:36:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:31 smithi044 ceph-mon[32404]: pgmap v4179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:31 smithi116 ceph-mon[37097]: pgmap v4179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:31 smithi171 ceph-mon[37789]: pgmap v4179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:33 smithi044 ceph-mon[32404]: pgmap v4180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:33 smithi116 ceph-mon[37097]: pgmap v4180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:33 smithi171 ceph-mon[37789]: pgmap v4180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:35 smithi044 ceph-mon[32404]: pgmap v4181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:35 smithi116 ceph-mon[37097]: pgmap v4181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:35 smithi171 ceph-mon[37789]: pgmap v4181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6940.753143 secs 2024-07-26T10:36:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6940.753143 secs 2024-07-26T10:36:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6940.753143 secs 2024-07-26T10:36:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:37 smithi044 ceph-mon[32404]: pgmap v4182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:37 smithi116 ceph-mon[37097]: pgmap v4182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:37 smithi171 ceph-mon[37789]: pgmap v4182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:39 smithi044 ceph-mon[32404]: pgmap v4183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:39 smithi116 ceph-mon[37097]: pgmap v4183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:39 smithi171 ceph-mon[37789]: pgmap v4183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6945.753291 secs 2024-07-26T10:36:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6945.753291 secs 2024-07-26T10:36:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6945.753291 secs 2024-07-26T10:36:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:41 smithi044 ceph-mon[32404]: pgmap v4184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:41 smithi116 ceph-mon[37097]: pgmap v4184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:41 smithi171 ceph-mon[37789]: pgmap v4184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:42.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:42.561+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:36:42.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:42.561+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:42.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:42.561+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:36:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:43 smithi044 ceph-mon[32404]: pgmap v4185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:43 smithi116 ceph-mon[37097]: pgmap v4185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:43 smithi171 ceph-mon[37789]: pgmap v4185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:45 smithi044 ceph-mon[32404]: pgmap v4186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:45 smithi116 ceph-mon[37097]: pgmap v4186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:45 smithi171 ceph-mon[37789]: pgmap v4186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:45.909+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:36:45.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:45.909+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:36:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:45.909+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:36:45.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:45.910+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:36:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6950.753438 secs 2024-07-26T10:36:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6950.753438 secs 2024-07-26T10:36:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6950.753438 secs 2024-07-26T10:36:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:47 smithi044 ceph-mon[32404]: pgmap v4187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:47 smithi116 ceph-mon[37097]: pgmap v4187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:47 smithi171 ceph-mon[37789]: pgmap v4187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:49 smithi044 ceph-mon[32404]: pgmap v4188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:49 smithi116 ceph-mon[37097]: pgmap v4188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:49 smithi171 ceph-mon[37789]: pgmap v4188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:36:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6955.753597 secs 2024-07-26T10:36:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6955.753597 secs 2024-07-26T10:36:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6955.753597 secs 2024-07-26T10:36:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:51 smithi044 ceph-mon[32404]: pgmap v4189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:51 smithi116 ceph-mon[37097]: pgmap v4189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:51 smithi171 ceph-mon[37789]: pgmap v4189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:53 smithi044 ceph-mon[32404]: pgmap v4190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:53 smithi116 ceph-mon[37097]: pgmap v4190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:53 smithi171 ceph-mon[37789]: pgmap v4190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:36:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:55 smithi044 ceph-mon[32404]: pgmap v4191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:55 smithi116 ceph-mon[37097]: pgmap v4191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:55 smithi171 ceph-mon[37789]: pgmap v4191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:56.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6960.753682 secs 2024-07-26T10:36:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6960.753682 secs 2024-07-26T10:36:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6960.753682 secs 2024-07-26T10:36:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:57 smithi044 ceph-mon[32404]: pgmap v4192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:36:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:57 smithi044 ceph-mon[32404]: pgmap v4193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:36:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:57 smithi116 ceph-mon[37097]: pgmap v4192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:36:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:57 smithi116 ceph-mon[37097]: pgmap v4193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:36:57.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:57.568+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:36:57.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:57.568+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:57.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:36:57.568+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:36:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:57 smithi171 ceph-mon[37789]: pgmap v4192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:36:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:57 smithi171 ceph-mon[37789]: pgmap v4193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:36:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:36:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:36:59 smithi044 ceph-mon[32404]: pgmap v4194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:36:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:36:59 smithi116 ceph-mon[37097]: pgmap v4194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:36:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:36:59 smithi171 ceph-mon[37789]: pgmap v4194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:01 smithi044 ceph-mon[32404]: pgmap v4195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6965.753827 secs 2024-07-26T10:37:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:01 smithi116 ceph-mon[37097]: pgmap v4195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6965.753827 secs 2024-07-26T10:37:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:01 smithi171 ceph-mon[37789]: pgmap v4195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6965.753827 secs 2024-07-26T10:37:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:03 smithi044 ceph-mon[32404]: pgmap v4196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:03 smithi116 ceph-mon[37097]: pgmap v4196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:03 smithi171 ceph-mon[37789]: pgmap v4196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:05 smithi044 ceph-mon[32404]: pgmap v4197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:05 smithi116 ceph-mon[37097]: pgmap v4197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:05 smithi171 ceph-mon[37789]: pgmap v4197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:37:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6970.753975 secs 2024-07-26T10:37:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6970.753975 secs 2024-07-26T10:37:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6970.753975 secs 2024-07-26T10:37:07.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:07 smithi171 ceph-mon[37789]: pgmap v4198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:37:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:07 smithi044 ceph-mon[32404]: pgmap v4198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:37:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:07 smithi116 ceph-mon[37097]: pgmap v4198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:37:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:09 smithi044 ceph-mon[32404]: pgmap v4199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:09 smithi116 ceph-mon[37097]: pgmap v4199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:09 smithi171 ceph-mon[37789]: pgmap v4199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:11 smithi044 ceph-mon[32404]: pgmap v4200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6975.754126 secs 2024-07-26T10:37:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:11 smithi116 ceph-mon[37097]: pgmap v4200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6975.754126 secs 2024-07-26T10:37:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:11 smithi171 ceph-mon[37789]: pgmap v4200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6975.754126 secs 2024-07-26T10:37:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:12.569+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:37:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:12.569+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:12.569+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:37:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:13 smithi044 ceph-mon[32404]: pgmap v4201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:13 smithi116 ceph-mon[37097]: pgmap v4201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:13 smithi171 ceph-mon[37789]: pgmap v4201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:15 smithi044 ceph-mon[32404]: pgmap v4202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:15 smithi116 ceph-mon[37097]: pgmap v4202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:15 smithi171 ceph-mon[37789]: pgmap v4202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6980.754220 secs 2024-07-26T10:37:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6980.754220 secs 2024-07-26T10:37:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6980.754220 secs 2024-07-26T10:37:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:17 smithi044 ceph-mon[32404]: pgmap v4203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:17 smithi116 ceph-mon[37097]: pgmap v4203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:17 smithi171 ceph-mon[37789]: pgmap v4203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:19 smithi044 ceph-mon[32404]: pgmap v4204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:19 smithi116 ceph-mon[37097]: pgmap v4204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:19 smithi171 ceph-mon[37789]: pgmap v4204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:21 smithi044 ceph-mon[32404]: pgmap v4205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6985.754373 secs 2024-07-26T10:37:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:21 smithi116 ceph-mon[37097]: pgmap v4205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6985.754373 secs 2024-07-26T10:37:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:21 smithi171 ceph-mon[37789]: pgmap v4205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6985.754373 secs 2024-07-26T10:37:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:23 smithi044 ceph-mon[32404]: pgmap v4206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:23 smithi116 ceph-mon[37097]: pgmap v4206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:23 smithi171 ceph-mon[37789]: pgmap v4206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:25 smithi044 ceph-mon[32404]: pgmap v4207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:25 smithi116 ceph-mon[37097]: pgmap v4207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:25 smithi171 ceph-mon[37789]: pgmap v4207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:26.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6990.754523 secs 2024-07-26T10:37:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6990.754523 secs 2024-07-26T10:37:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6990.754523 secs 2024-07-26T10:37:27.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:27.577+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:37:27.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:27.577+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:37:27.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:27.577+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:27.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:27 smithi044 ceph-mon[32404]: pgmap v4208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:37:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:27 smithi116 ceph-mon[37097]: pgmap v4208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:37:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:27 smithi171 ceph-mon[37789]: pgmap v4208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:37:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:29 smithi044 ceph-mon[32404]: pgmap v4209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:29 smithi116 ceph-mon[37097]: pgmap v4209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:29 smithi171 ceph-mon[37789]: pgmap v4209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:31 smithi044 ceph-mon[32404]: pgmap v4210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 6995.754672 secs 2024-07-26T10:37:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:31 smithi116 ceph-mon[37097]: pgmap v4210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 6995.754672 secs 2024-07-26T10:37:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:31 smithi171 ceph-mon[37789]: pgmap v4210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 6995.754672 secs 2024-07-26T10:37:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:33 smithi044 ceph-mon[32404]: pgmap v4211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:33 smithi116 ceph-mon[37097]: pgmap v4211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:33 smithi171 ceph-mon[37789]: pgmap v4211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:35 smithi044 ceph-mon[32404]: pgmap v4212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:35 smithi116 ceph-mon[37097]: pgmap v4212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:35 smithi171 ceph-mon[37789]: pgmap v4212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7000.754830 secs 2024-07-26T10:37:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7000.754830 secs 2024-07-26T10:37:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7000.754830 secs 2024-07-26T10:37:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:37 smithi044 ceph-mon[32404]: pgmap v4213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:37 smithi116 ceph-mon[37097]: pgmap v4213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:37 smithi171 ceph-mon[37789]: pgmap v4213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:39 smithi044 ceph-mon[32404]: pgmap v4214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:39 smithi116 ceph-mon[37097]: pgmap v4214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:39 smithi171 ceph-mon[37789]: pgmap v4214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:41 smithi044 ceph-mon[32404]: pgmap v4215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7005.754993 secs 2024-07-26T10:37:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:41 smithi116 ceph-mon[37097]: pgmap v4215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7005.754993 secs 2024-07-26T10:37:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:41 smithi171 ceph-mon[37789]: pgmap v4215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7005.754993 secs 2024-07-26T10:37:42.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:42.581+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:37:42.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:42.581+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:37:42.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:42.581+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:43 smithi116 ceph-mon[37097]: pgmap v4216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:43 smithi171 ceph-mon[37789]: pgmap v4216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:43 smithi044 ceph-mon[32404]: pgmap v4216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:45 smithi171 ceph-mon[37789]: pgmap v4217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:45 smithi044 ceph-mon[32404]: pgmap v4217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:45 smithi116 ceph-mon[37097]: pgmap v4217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7010.755149 secs 2024-07-26T10:37:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7010.755149 secs 2024-07-26T10:37:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7010.755149 secs 2024-07-26T10:37:47.811 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:47 smithi044 ceph-mon[32404]: pgmap v4218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:47 smithi171 ceph-mon[37789]: pgmap v4218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:48.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:47 smithi116 ceph-mon[37097]: pgmap v4218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:49 smithi171 ceph-mon[37789]: pgmap v4219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:49 smithi044 ceph-mon[32404]: pgmap v4219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:49 smithi116 ceph-mon[37097]: pgmap v4219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:51.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:51 smithi116 ceph-mon[37097]: pgmap v4220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:51.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7015.755283 secs 2024-07-26T10:37:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:51 smithi171 ceph-mon[37789]: pgmap v4220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7015.755283 secs 2024-07-26T10:37:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:51 smithi044 ceph-mon[32404]: pgmap v4220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7015.755283 secs 2024-07-26T10:37:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:53 smithi171 ceph-mon[37789]: pgmap v4221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:53 smithi044 ceph-mon[32404]: pgmap v4221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:53 smithi116 ceph-mon[37097]: pgmap v4221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:37:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:55 smithi171 ceph-mon[37789]: pgmap v4222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:55 smithi044 ceph-mon[32404]: pgmap v4222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:55 smithi116 ceph-mon[37097]: pgmap v4222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:37:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7020.755449 secs 2024-07-26T10:37:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7020.755449 secs 2024-07-26T10:37:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7020.755449 secs 2024-07-26T10:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:57.588+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:37:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:57.588+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:37:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:37:57.588+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:57 smithi171 ceph-mon[37789]: pgmap v4223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:57 smithi044 ceph-mon[32404]: pgmap v4223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:57 smithi116 ceph-mon[37097]: pgmap v4223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:37:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:37:59.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:59 smithi044 ceph-mon[32404]: pgmap v4224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:37:59.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:37:59 smithi044 ceph-mon[32404]: pgmap v4225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:59 smithi116 ceph-mon[37097]: pgmap v4224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:37:59 smithi116 ceph-mon[37097]: pgmap v4225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:59 smithi171 ceph-mon[37789]: pgmap v4224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:37:59 smithi171 ceph-mon[37789]: pgmap v4225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:01 smithi044 ceph-mon[32404]: pgmap v4226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7025.755670 secs 2024-07-26T10:38:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:01 smithi116 ceph-mon[37097]: pgmap v4226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7025.755670 secs 2024-07-26T10:38:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:01 smithi171 ceph-mon[37789]: pgmap v4226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7025.755670 secs 2024-07-26T10:38:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:03 smithi044 ceph-mon[32404]: pgmap v4227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:03 smithi116 ceph-mon[37097]: pgmap v4227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:03 smithi171 ceph-mon[37789]: pgmap v4227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:05 smithi044 ceph-mon[32404]: pgmap v4228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:05 smithi116 ceph-mon[37097]: pgmap v4228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:05 smithi171 ceph-mon[37789]: pgmap v4228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7030.755765 secs 2024-07-26T10:38:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7030.755765 secs 2024-07-26T10:38:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7030.755765 secs 2024-07-26T10:38:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:07 smithi044 ceph-mon[32404]: pgmap v4229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:07 smithi116 ceph-mon[37097]: pgmap v4229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:07 smithi171 ceph-mon[37789]: pgmap v4229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:38:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:09 smithi044 ceph-mon[32404]: pgmap v4230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:09 smithi116 ceph-mon[37097]: pgmap v4230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:09 smithi171 ceph-mon[37789]: pgmap v4230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:38:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:11 smithi044 ceph-mon[32404]: pgmap v4231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7035.755917 secs 2024-07-26T10:38:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:11 smithi116 ceph-mon[37097]: pgmap v4231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7035.755917 secs 2024-07-26T10:38:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:11 smithi171 ceph-mon[37789]: pgmap v4231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7035.755917 secs 2024-07-26T10:38:12.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:12.592+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:38:12.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:12.592+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:38:12.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:12.592+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:13 smithi044 ceph-mon[32404]: pgmap v4232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:13 smithi116 ceph-mon[37097]: pgmap v4232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:13 smithi171 ceph-mon[37789]: pgmap v4232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:15.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:15.925+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:38:15.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:15.925+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:38:15.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:15 smithi044 ceph-mon[32404]: pgmap v4233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:15.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:15.926+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:38:15.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:15.926+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:38:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:15 smithi116 ceph-mon[37097]: pgmap v4233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:15 smithi171 ceph-mon[37789]: pgmap v4233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7040.756080 secs 2024-07-26T10:38:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7040.756080 secs 2024-07-26T10:38:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7040.756080 secs 2024-07-26T10:38:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:17 smithi044 ceph-mon[32404]: pgmap v4234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:17 smithi116 ceph-mon[37097]: pgmap v4234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:17 smithi171 ceph-mon[37789]: pgmap v4234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:19 smithi044 ceph-mon[32404]: pgmap v4235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:19 smithi116 ceph-mon[37097]: pgmap v4235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:19 smithi171 ceph-mon[37789]: pgmap v4235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:21 smithi044 ceph-mon[32404]: pgmap v4236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7045.756243 secs 2024-07-26T10:38:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:21 smithi116 ceph-mon[37097]: pgmap v4236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7045.756243 secs 2024-07-26T10:38:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:21 smithi171 ceph-mon[37789]: pgmap v4236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7045.756243 secs 2024-07-26T10:38:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:23 smithi044 ceph-mon[32404]: pgmap v4237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:24.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:23 smithi116 ceph-mon[37097]: pgmap v4237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:23 smithi171 ceph-mon[37789]: pgmap v4237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:25 smithi044 ceph-mon[32404]: pgmap v4238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7050.756395 secs 2024-07-26T10:38:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:25 smithi116 ceph-mon[37097]: pgmap v4238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7050.756395 secs 2024-07-26T10:38:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:25 smithi171 ceph-mon[37789]: pgmap v4238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7050.756395 secs 2024-07-26T10:38:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:27.601+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:38:27.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:27.601+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:27.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:27.601+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:38:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:27 smithi044 ceph-mon[32404]: pgmap v4239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:27 smithi116 ceph-mon[37097]: pgmap v4239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:27 smithi171 ceph-mon[37789]: pgmap v4239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:29 smithi044 ceph-mon[32404]: pgmap v4240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:29 smithi116 ceph-mon[37097]: pgmap v4240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:29 smithi171 ceph-mon[37789]: pgmap v4240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7055.756513 secs 2024-07-26T10:38:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7055.756513 secs 2024-07-26T10:38:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7055.756513 secs 2024-07-26T10:38:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:31 smithi044 ceph-mon[32404]: pgmap v4241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:31 smithi116 ceph-mon[37097]: pgmap v4241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:31 smithi171 ceph-mon[37789]: pgmap v4241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:33 smithi044 ceph-mon[32404]: pgmap v4242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:33 smithi116 ceph-mon[37097]: pgmap v4242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:33 smithi171 ceph-mon[37789]: pgmap v4242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:35 smithi044 ceph-mon[32404]: pgmap v4243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:36.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7060.756673 secs 2024-07-26T10:38:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:35 smithi116 ceph-mon[37097]: pgmap v4243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7060.756673 secs 2024-07-26T10:38:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:35 smithi171 ceph-mon[37789]: pgmap v4243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7060.756673 secs 2024-07-26T10:38:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:37 smithi044 ceph-mon[32404]: pgmap v4244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:37 smithi116 ceph-mon[37097]: pgmap v4244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:37 smithi171 ceph-mon[37789]: pgmap v4244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:39 smithi044 ceph-mon[32404]: pgmap v4245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:39 smithi116 ceph-mon[37097]: pgmap v4245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:39 smithi171 ceph-mon[37789]: pgmap v4245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7065.756819 secs 2024-07-26T10:38:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7065.756819 secs 2024-07-26T10:38:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7065.756819 secs 2024-07-26T10:38:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:41 smithi044 ceph-mon[32404]: pgmap v4246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:41 smithi116 ceph-mon[37097]: pgmap v4246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:41 smithi171 ceph-mon[37789]: pgmap v4246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:42.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:42.606+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:38:42.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:42.606+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:38:42.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:42.606+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:43 smithi044 ceph-mon[32404]: pgmap v4247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:43 smithi116 ceph-mon[37097]: pgmap v4247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:43 smithi171 ceph-mon[37789]: pgmap v4247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:45 smithi044 ceph-mon[32404]: pgmap v4248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7070.756963 secs 2024-07-26T10:38:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:45 smithi116 ceph-mon[37097]: pgmap v4248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7070.756963 secs 2024-07-26T10:38:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:45 smithi171 ceph-mon[37789]: pgmap v4248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7070.756963 secs 2024-07-26T10:38:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:47 smithi044 ceph-mon[32404]: pgmap v4249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:47 smithi116 ceph-mon[37097]: pgmap v4249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:47 smithi171 ceph-mon[37789]: pgmap v4249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:49 smithi044 ceph-mon[32404]: pgmap v4250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:49 smithi116 ceph-mon[37097]: pgmap v4250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:49 smithi171 ceph-mon[37789]: pgmap v4250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:38:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7075.757101 secs 2024-07-26T10:38:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7075.757101 secs 2024-07-26T10:38:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7075.757101 secs 2024-07-26T10:38:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:51 smithi171 ceph-mon[37789]: pgmap v4251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:52.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:51 smithi116 ceph-mon[37097]: pgmap v4251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:51 smithi044 ceph-mon[32404]: pgmap v4251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:53 smithi171 ceph-mon[37789]: pgmap v4252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:53 smithi044 ceph-mon[32404]: pgmap v4252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:53 smithi116 ceph-mon[37097]: pgmap v4252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:38:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:55 smithi171 ceph-mon[37789]: pgmap v4253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:56.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7080.757256 secs 2024-07-26T10:38:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:55 smithi044 ceph-mon[32404]: pgmap v4253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:56.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7080.757256 secs 2024-07-26T10:38:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:55 smithi116 ceph-mon[37097]: pgmap v4253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:38:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7080.757256 secs 2024-07-26T10:38:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:57.615+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:38:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:57.615+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:38:57.615+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:38:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:57 smithi044 ceph-mon[32404]: pgmap v4254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:57 smithi171 ceph-mon[37789]: pgmap v4254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:57 smithi116 ceph-mon[37097]: pgmap v4254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:38:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:38:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:38:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:59 smithi044 ceph-mon[32404]: pgmap v4255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:38:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:59 smithi116 ceph-mon[37097]: pgmap v4255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:38:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:59 smithi171 ceph-mon[37789]: pgmap v4255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:00.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:38:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:01 smithi171 ceph-mon[37789]: pgmap v4256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7085.757375 secs 2024-07-26T10:39:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:01 smithi044 ceph-mon[32404]: pgmap v4256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7085.757375 secs 2024-07-26T10:39:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:01 smithi116 ceph-mon[37097]: pgmap v4256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7085.757375 secs 2024-07-26T10:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:03 smithi044 ceph-mon[32404]: pgmap v4257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:03 smithi116 ceph-mon[37097]: pgmap v4257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:03 smithi171 ceph-mon[37789]: pgmap v4257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:04 smithi044 ceph-mon[32404]: pgmap v4258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:39:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:04 smithi044 ceph-mon[32404]: pgmap v4259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:04.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:04 smithi116 ceph-mon[37097]: pgmap v4258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:39:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:04 smithi116 ceph-mon[37097]: pgmap v4259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:04.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:04 smithi171 ceph-mon[37789]: pgmap v4258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:39:04.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:04 smithi171 ceph-mon[37789]: pgmap v4259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:04.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:39:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:06 smithi044 ceph-mon[32404]: pgmap v4260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7090.757520 secs 2024-07-26T10:39:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:06 smithi116 ceph-mon[37097]: pgmap v4260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7090.757520 secs 2024-07-26T10:39:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:06 smithi171 ceph-mon[37789]: pgmap v4260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7090.757520 secs 2024-07-26T10:39:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:08 smithi044 ceph-mon[32404]: pgmap v4261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:08 smithi116 ceph-mon[37097]: pgmap v4261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:08 smithi171 ceph-mon[37789]: pgmap v4261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:10.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:10 smithi044 ceph-mon[32404]: pgmap v4262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:10 smithi116 ceph-mon[37097]: pgmap v4262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:10 smithi171 ceph-mon[37789]: pgmap v4262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:39:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7095.757689 secs 2024-07-26T10:39:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7095.757689 secs 2024-07-26T10:39:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7095.757689 secs 2024-07-26T10:39:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:12.619+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:39:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:12.619+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:12.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:12.619+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:39:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:12 smithi044 ceph-mon[32404]: pgmap v4263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:12 smithi116 ceph-mon[37097]: pgmap v4263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:12 smithi171 ceph-mon[37789]: pgmap v4263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:14 smithi044 ceph-mon[32404]: pgmap v4264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:14 smithi116 ceph-mon[37097]: pgmap v4264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:14 smithi171 ceph-mon[37789]: pgmap v4264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:39:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:16 smithi044 ceph-mon[32404]: pgmap v4265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7100.757857 secs 2024-07-26T10:39:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:16 smithi116 ceph-mon[37097]: pgmap v4265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7100.757857 secs 2024-07-26T10:39:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:16 smithi171 ceph-mon[37789]: pgmap v4265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7100.757857 secs 2024-07-26T10:39:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:18 smithi044 ceph-mon[32404]: pgmap v4266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:18 smithi116 ceph-mon[37097]: pgmap v4266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:18 smithi171 ceph-mon[37789]: pgmap v4266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:20 smithi044 ceph-mon[32404]: pgmap v4267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:20.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:20 smithi116 ceph-mon[37097]: pgmap v4267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:20 smithi171 ceph-mon[37789]: pgmap v4267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7105.758001 secs 2024-07-26T10:39:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7105.758001 secs 2024-07-26T10:39:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7105.758001 secs 2024-07-26T10:39:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:22 smithi044 ceph-mon[32404]: pgmap v4268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:22 smithi116 ceph-mon[37097]: pgmap v4268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:22 smithi171 ceph-mon[37789]: pgmap v4268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:24 smithi044 ceph-mon[32404]: pgmap v4269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:24 smithi116 ceph-mon[37097]: pgmap v4269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:24 smithi171 ceph-mon[37789]: pgmap v4269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:26 smithi044 ceph-mon[32404]: pgmap v4270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7110.758142 secs 2024-07-26T10:39:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:26 smithi116 ceph-mon[37097]: pgmap v4270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7110.758142 secs 2024-07-26T10:39:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:26 smithi171 ceph-mon[37789]: pgmap v4270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7110.758142 secs 2024-07-26T10:39:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:27.628+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:39:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:27.628+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:39:27.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:27.628+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:28 smithi044 ceph-mon[32404]: pgmap v4271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:28 smithi116 ceph-mon[37097]: pgmap v4271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:28 smithi171 ceph-mon[37789]: pgmap v4271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:30 smithi044 ceph-mon[32404]: pgmap v4272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:30 smithi116 ceph-mon[37097]: pgmap v4272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:30 smithi171 ceph-mon[37789]: pgmap v4272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7115.758292 secs 2024-07-26T10:39:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7115.758292 secs 2024-07-26T10:39:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7115.758292 secs 2024-07-26T10:39:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:32 smithi044 ceph-mon[32404]: pgmap v4273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:32 smithi116 ceph-mon[37097]: pgmap v4273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:32 smithi171 ceph-mon[37789]: pgmap v4273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:34 smithi044 ceph-mon[32404]: pgmap v4274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:34 smithi116 ceph-mon[37097]: pgmap v4274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:34 smithi171 ceph-mon[37789]: pgmap v4274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:36 smithi044 ceph-mon[32404]: pgmap v4275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7120.758442 secs 2024-07-26T10:39:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:36 smithi116 ceph-mon[37097]: pgmap v4275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7120.758442 secs 2024-07-26T10:39:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:36 smithi171 ceph-mon[37789]: pgmap v4275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7120.758442 secs 2024-07-26T10:39:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:38 smithi044 ceph-mon[32404]: pgmap v4276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:38 smithi116 ceph-mon[37097]: pgmap v4276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:38 smithi171 ceph-mon[37789]: pgmap v4276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:40 smithi044 ceph-mon[32404]: pgmap v4277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:40 smithi116 ceph-mon[37097]: pgmap v4277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:40 smithi171 ceph-mon[37789]: pgmap v4277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7125.758609 secs 2024-07-26T10:39:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7125.758609 secs 2024-07-26T10:39:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7125.758609 secs 2024-07-26T10:39:42.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:42.633+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:39:42.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:42.633+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:39:42.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:42.633+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:42 smithi171 ceph-mon[37789]: pgmap v4278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:42 smithi044 ceph-mon[32404]: pgmap v4278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:42 smithi116 ceph-mon[37097]: pgmap v4278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:44 smithi171 ceph-mon[37789]: pgmap v4279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:44 smithi044 ceph-mon[32404]: pgmap v4279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:45.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:44 smithi116 ceph-mon[37097]: pgmap v4279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:46 smithi171 ceph-mon[37789]: pgmap v4280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7130.758772 secs 2024-07-26T10:39:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:46 smithi044 ceph-mon[32404]: pgmap v4280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7130.758772 secs 2024-07-26T10:39:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:46 smithi116 ceph-mon[37097]: pgmap v4280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7130.758772 secs 2024-07-26T10:39:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:48 smithi171 ceph-mon[37789]: pgmap v4281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:48 smithi044 ceph-mon[32404]: pgmap v4281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:48 smithi116 ceph-mon[37097]: pgmap v4281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:50 smithi171 ceph-mon[37789]: pgmap v4282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:50 smithi044 ceph-mon[32404]: pgmap v4282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:50 smithi116 ceph-mon[37097]: pgmap v4282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:39:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7135.758944 secs 2024-07-26T10:39:51.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7135.758944 secs 2024-07-26T10:39:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7135.758944 secs 2024-07-26T10:39:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:52 smithi171 ceph-mon[37789]: pgmap v4283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:52.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:52 smithi116 ceph-mon[37097]: pgmap v4283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:52 smithi044 ceph-mon[32404]: pgmap v4283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:54 smithi171 ceph-mon[37789]: pgmap v4284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:54 smithi044 ceph-mon[32404]: pgmap v4284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:54 smithi116 ceph-mon[37097]: pgmap v4284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:39:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:56 smithi044 ceph-mon[32404]: pgmap v4285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7140.759104 secs 2024-07-26T10:39:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:56 smithi116 ceph-mon[37097]: pgmap v4285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7140.759104 secs 2024-07-26T10:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:56 smithi171 ceph-mon[37789]: pgmap v4285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7140.759104 secs 2024-07-26T10:39:57.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:57.634+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:39:57.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:57.634+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:39:57.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:39:57.634+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:58.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:39:58 smithi171 ceph-mon[37789]: pgmap v4286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:39:58 smithi044 ceph-mon[32404]: pgmap v4286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:39:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:39:58 smithi116 ceph-mon[37097]: pgmap v4286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:00 smithi044 ceph-mon[32404]: pgmap v4287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:40:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:00 smithi116 ceph-mon[37097]: pgmap v4287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:40:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:00 smithi171 ceph-mon[37789]: pgmap v4287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:40:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7145.759263 secs 2024-07-26T10:40:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7145.759263 secs 2024-07-26T10:40:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7145.759263 secs 2024-07-26T10:40:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:02 smithi044 ceph-mon[32404]: pgmap v4288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:02 smithi116 ceph-mon[37097]: pgmap v4288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:02 smithi171 ceph-mon[37789]: pgmap v4288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:03 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:03 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:03 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:04 smithi044 ceph-mon[32404]: pgmap v4289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:40:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:05.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:40:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:04 smithi116 ceph-mon[37097]: pgmap v4289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:40:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:40:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:04 smithi171 ceph-mon[37789]: pgmap v4289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:40:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:40:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:05 smithi044 ceph-mon[32404]: pgmap v4290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:40:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:05 smithi044 ceph-mon[32404]: pgmap v4291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:05 smithi116 ceph-mon[37097]: pgmap v4290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:40:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:05 smithi116 ceph-mon[37097]: pgmap v4291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:05 smithi171 ceph-mon[37789]: pgmap v4290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:40:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:05 smithi171 ceph-mon[37789]: pgmap v4291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7150.759425 secs 2024-07-26T10:40:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7150.759425 secs 2024-07-26T10:40:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7150.759425 secs 2024-07-26T10:40:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:07 smithi044 ceph-mon[32404]: pgmap v4292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:07 smithi116 ceph-mon[37097]: pgmap v4292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:07 smithi171 ceph-mon[37789]: pgmap v4292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:09 smithi044 ceph-mon[32404]: pgmap v4293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:09 smithi116 ceph-mon[37097]: pgmap v4293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:09 smithi171 ceph-mon[37789]: pgmap v4293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7155.759574 secs 2024-07-26T10:40:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:11 smithi044 ceph-mon[32404]: pgmap v4294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7155.759574 secs 2024-07-26T10:40:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:11 smithi116 ceph-mon[37097]: pgmap v4294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7155.759574 secs 2024-07-26T10:40:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:11 smithi171 ceph-mon[37789]: pgmap v4294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:40:12.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:12.637+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:40:12.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:12.637+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:40:12.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:12.637+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:13 smithi044 ceph-mon[32404]: pgmap v4295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:13 smithi116 ceph-mon[37097]: pgmap v4295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:13 smithi171 ceph-mon[37789]: pgmap v4295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:15 smithi044 ceph-mon[32404]: pgmap v4296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:40:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:15 smithi116 ceph-mon[37097]: pgmap v4296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:40:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:15 smithi171 ceph-mon[37789]: pgmap v4296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:40:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7160.759719 secs 2024-07-26T10:40:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7160.759719 secs 2024-07-26T10:40:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7160.759719 secs 2024-07-26T10:40:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:17 smithi044 ceph-mon[32404]: pgmap v4297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:17 smithi116 ceph-mon[37097]: pgmap v4297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:17 smithi171 ceph-mon[37789]: pgmap v4297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:19 smithi044 ceph-mon[32404]: pgmap v4298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:40:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:19 smithi116 ceph-mon[37097]: pgmap v4298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:40:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:19 smithi171 ceph-mon[37789]: pgmap v4298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:40:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7165.759879 secs 2024-07-26T10:40:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7165.759879 secs 2024-07-26T10:40:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7165.759879 secs 2024-07-26T10:40:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:21 smithi044 ceph-mon[32404]: pgmap v4299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:21 smithi116 ceph-mon[37097]: pgmap v4299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:21 smithi171 ceph-mon[37789]: pgmap v4299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:23 smithi044 ceph-mon[32404]: pgmap v4300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:23 smithi116 ceph-mon[37097]: pgmap v4300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:23 smithi171 ceph-mon[37789]: pgmap v4300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:25 smithi044 ceph-mon[32404]: pgmap v4301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7170.760038 secs 2024-07-26T10:40:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:25 smithi116 ceph-mon[37097]: pgmap v4301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7170.760038 secs 2024-07-26T10:40:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:25 smithi171 ceph-mon[37789]: pgmap v4301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7170.760038 secs 2024-07-26T10:40:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:27.642+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:40:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:27.642+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:27.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:27.642+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:40:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:27 smithi044 ceph-mon[32404]: pgmap v4302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:27 smithi116 ceph-mon[37097]: pgmap v4302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:27 smithi171 ceph-mon[37789]: pgmap v4302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:29 smithi044 ceph-mon[32404]: pgmap v4303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:29 smithi116 ceph-mon[37097]: pgmap v4303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:29 smithi171 ceph-mon[37789]: pgmap v4303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7175.760188 secs 2024-07-26T10:40:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7175.760188 secs 2024-07-26T10:40:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7175.760188 secs 2024-07-26T10:40:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:31 smithi044 ceph-mon[32404]: pgmap v4304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:31 smithi116 ceph-mon[37097]: pgmap v4304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:31 smithi171 ceph-mon[37789]: pgmap v4304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:33 smithi044 ceph-mon[32404]: pgmap v4305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:33 smithi116 ceph-mon[37097]: pgmap v4305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:33 smithi171 ceph-mon[37789]: pgmap v4305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:35 smithi044 ceph-mon[32404]: pgmap v4306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7180.760332 secs 2024-07-26T10:40:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:35 smithi116 ceph-mon[37097]: pgmap v4306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7180.760332 secs 2024-07-26T10:40:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:35 smithi171 ceph-mon[37789]: pgmap v4306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7180.760332 secs 2024-07-26T10:40:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:37 smithi044 ceph-mon[32404]: pgmap v4307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:37 smithi116 ceph-mon[37097]: pgmap v4307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:37 smithi171 ceph-mon[37789]: pgmap v4307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:39 smithi116 ceph-mon[37097]: pgmap v4308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:39 smithi171 ceph-mon[37789]: pgmap v4308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:39 smithi044 ceph-mon[32404]: pgmap v4308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7185.760497 secs 2024-07-26T10:40:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7185.760497 secs 2024-07-26T10:40:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7185.760497 secs 2024-07-26T10:40:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:41 smithi171 ceph-mon[37789]: pgmap v4309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:41 smithi044 ceph-mon[32404]: pgmap v4309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:41 smithi116 ceph-mon[37097]: pgmap v4309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:42.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:42.644+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:40:42.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:42.644+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:40:42.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:42.644+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:43 smithi171 ceph-mon[37789]: pgmap v4310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:43 smithi044 ceph-mon[32404]: pgmap v4310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:43 smithi116 ceph-mon[37097]: pgmap v4310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:45 smithi171 ceph-mon[37789]: pgmap v4311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7190.760680 secs 2024-07-26T10:40:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:45 smithi044 ceph-mon[32404]: pgmap v4311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:46.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7190.760680 secs 2024-07-26T10:40:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:45 smithi116 ceph-mon[37097]: pgmap v4311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7190.760680 secs 2024-07-26T10:40:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:47 smithi171 ceph-mon[37789]: pgmap v4312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:47 smithi044 ceph-mon[32404]: pgmap v4312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:47 smithi116 ceph-mon[37097]: pgmap v4312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:49 smithi171 ceph-mon[37789]: pgmap v4313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:49 smithi044 ceph-mon[32404]: pgmap v4313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:49 smithi116 ceph-mon[37097]: pgmap v4313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:40:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7195.760850 secs 2024-07-26T10:40:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7195.760850 secs 2024-07-26T10:40:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7195.760850 secs 2024-07-26T10:40:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:51 smithi171 ceph-mon[37789]: pgmap v4314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:51 smithi044 ceph-mon[32404]: pgmap v4314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:51 smithi116 ceph-mon[37097]: pgmap v4314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:53 smithi044 ceph-mon[32404]: pgmap v4315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:53 smithi171 ceph-mon[37789]: pgmap v4315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:53 smithi116 ceph-mon[37097]: pgmap v4315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:40:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:55 smithi044 ceph-mon[32404]: pgmap v4316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7200.760977 secs 2024-07-26T10:40:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:55 smithi171 ceph-mon[37789]: pgmap v4316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7200.760977 secs 2024-07-26T10:40:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:55 smithi116 ceph-mon[37097]: pgmap v4316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:40:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7200.760977 secs 2024-07-26T10:40:57.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:57.653+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:40:57.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:57.653+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:57.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:40:57.653+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:40:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:57 smithi044 ceph-mon[32404]: pgmap v4317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:58.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:57 smithi116 ceph-mon[37097]: pgmap v4317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:40:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:57 smithi171 ceph-mon[37789]: pgmap v4317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:00.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:40:59 smithi171 ceph-mon[37789]: pgmap v4318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:41:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:40:59 smithi044 ceph-mon[32404]: pgmap v4318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:41:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:40:59 smithi116 ceph-mon[37097]: pgmap v4318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T10:41:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7205.761144 secs 2024-07-26T10:41:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7205.761144 secs 2024-07-26T10:41:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7205.761144 secs 2024-07-26T10:41:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:01 smithi044 ceph-mon[32404]: pgmap v4319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:01 smithi116 ceph-mon[37097]: pgmap v4319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:01 smithi171 ceph-mon[37789]: pgmap v4319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:03 smithi044 ceph-mon[32404]: pgmap v4320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:04.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:03 smithi116 ceph-mon[37097]: pgmap v4320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:03 smithi171 ceph-mon[37789]: pgmap v4320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:05.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:05 smithi044 ceph-mon[32404]: pgmap v4321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7210.761239 secs 2024-07-26T10:41:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:06.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:05 smithi116 ceph-mon[37097]: pgmap v4321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7210.761239 secs 2024-07-26T10:41:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:05 smithi171 ceph-mon[37789]: pgmap v4321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7210.761239 secs 2024-07-26T10:41:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:06 smithi044 ceph-mon[32404]: pgmap v4322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:41:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:06 smithi044 ceph-mon[32404]: pgmap v4323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:41:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:06 smithi116 ceph-mon[37097]: pgmap v4322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:41:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:06 smithi116 ceph-mon[37097]: pgmap v4323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:41:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:06 smithi171 ceph-mon[37789]: pgmap v4322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:41:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:06 smithi171 ceph-mon[37789]: pgmap v4323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:41:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:08 smithi044 ceph-mon[32404]: pgmap v4324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:41:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:08 smithi116 ceph-mon[37097]: pgmap v4324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:41:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:08 smithi171 ceph-mon[37789]: pgmap v4324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:41:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:10 smithi044 ceph-mon[32404]: pgmap v4325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7215.761402 secs 2024-07-26T10:41:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:10 smithi116 ceph-mon[37097]: pgmap v4325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7215.761402 secs 2024-07-26T10:41:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:10 smithi171 ceph-mon[37789]: pgmap v4325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7215.761402 secs 2024-07-26T10:41:12.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:12.656+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:41:12.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:12.656+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:12.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:12.656+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:41:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:12 smithi044 ceph-mon[32404]: pgmap v4326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:12 smithi116 ceph-mon[37097]: pgmap v4326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:12 smithi171 ceph-mon[37789]: pgmap v4326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:14.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T10:41:14.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:41:14.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf58069040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:41:14.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcf58032af0 tx=0x7fcf58069040 comp rx=0 tx=0).stop 2024-07-26T10:41:14.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T10:41:14.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:41:14.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.707+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:41:14.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.708+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 0x7fcf540c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:41:14.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.708+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 0x7fcf540c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:41:14.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:14.710+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 0x7fcf540c3ec0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf5806d000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:41:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:14 smithi044 ceph-mon[32404]: pgmap v4327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:14 smithi116 ceph-mon[37097]: pgmap v4327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:14 smithi171 ceph-mon[37789]: pgmap v4327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:41:15.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:15.958+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:41:15.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:15.958+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:41:15.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:15.958+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:41:15.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:15.959+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:41:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7220.761568 secs 2024-07-26T10:41:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7220.761568 secs 2024-07-26T10:41:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7220.761568 secs 2024-07-26T10:41:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:16 smithi044 ceph-mon[32404]: pgmap v4328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:41:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:16 smithi116 ceph-mon[37097]: pgmap v4328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:41:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:16 smithi171 ceph-mon[37789]: pgmap v4328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:41:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:18 smithi044 ceph-mon[32404]: pgmap v4329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:18 smithi116 ceph-mon[37097]: pgmap v4329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:18 smithi171 ceph-mon[37789]: pgmap v4329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:20 smithi044 ceph-mon[32404]: pgmap v4330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7225.761714 secs 2024-07-26T10:41:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:20 smithi116 ceph-mon[37097]: pgmap v4330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7225.761714 secs 2024-07-26T10:41:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:20 smithi171 ceph-mon[37789]: pgmap v4330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7225.761714 secs 2024-07-26T10:41:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:22 smithi044 ceph-mon[32404]: pgmap v4331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:22 smithi116 ceph-mon[37097]: pgmap v4331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:22 smithi171 ceph-mon[37789]: pgmap v4331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:24 smithi044 ceph-mon[32404]: pgmap v4332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:24 smithi116 ceph-mon[37097]: pgmap v4332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:24 smithi171 ceph-mon[37789]: pgmap v4332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:26.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7230.761811 secs 2024-07-26T10:41:26.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7230.761811 secs 2024-07-26T10:41:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7230.761811 secs 2024-07-26T10:41:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:26 smithi044 ceph-mon[32404]: pgmap v4333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:26 smithi116 ceph-mon[37097]: pgmap v4333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:26 smithi171 ceph-mon[37789]: pgmap v4333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:27.666+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:41:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:27.666+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:41:27.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:27.666+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:28 smithi044 ceph-mon[32404]: pgmap v4334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:28 smithi116 ceph-mon[37097]: pgmap v4334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:28 smithi171 ceph-mon[37789]: pgmap v4334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:30 smithi044 ceph-mon[32404]: pgmap v4335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7235.761979 secs 2024-07-26T10:41:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:30 smithi116 ceph-mon[37097]: pgmap v4335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7235.761979 secs 2024-07-26T10:41:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:30 smithi171 ceph-mon[37789]: pgmap v4335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7235.761979 secs 2024-07-26T10:41:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:32 smithi044 ceph-mon[32404]: pgmap v4336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:32 smithi116 ceph-mon[37097]: pgmap v4336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:32 smithi171 ceph-mon[37789]: pgmap v4336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:34 smithi044 ceph-mon[32404]: pgmap v4337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:34 smithi116 ceph-mon[37097]: pgmap v4337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:35.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:34 smithi171 ceph-mon[37789]: pgmap v4337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7240.762127 secs 2024-07-26T10:41:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7240.762127 secs 2024-07-26T10:41:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7240.762127 secs 2024-07-26T10:41:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:36 smithi044 ceph-mon[32404]: pgmap v4338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:41:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:36 smithi116 ceph-mon[37097]: pgmap v4338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:41:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:36 smithi171 ceph-mon[37789]: pgmap v4338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:41:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:38 smithi044 ceph-mon[32404]: pgmap v4339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:38 smithi116 ceph-mon[37097]: pgmap v4339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:38 smithi171 ceph-mon[37789]: pgmap v4339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:41 smithi044 ceph-mon[32404]: pgmap v4340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7245.762270 secs 2024-07-26T10:41:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:41 smithi116 ceph-mon[37097]: pgmap v4340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7245.762270 secs 2024-07-26T10:41:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:41 smithi171 ceph-mon[37789]: pgmap v4340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:41:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7245.762270 secs 2024-07-26T10:41:42.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:42.671+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:41:42.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:42.671+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:42.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:42.671+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:41:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:43 smithi044 ceph-mon[32404]: pgmap v4341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:43 smithi116 ceph-mon[37097]: pgmap v4341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:43 smithi171 ceph-mon[37789]: pgmap v4341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:45 smithi044 ceph-mon[32404]: pgmap v4342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:45 smithi116 ceph-mon[37097]: pgmap v4342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:45 smithi171 ceph-mon[37789]: pgmap v4342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7250.762416 secs 2024-07-26T10:41:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7250.762416 secs 2024-07-26T10:41:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7250.762416 secs 2024-07-26T10:41:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:47 smithi171 ceph-mon[37789]: pgmap v4343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:41:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:47 smithi044 ceph-mon[32404]: pgmap v4343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:41:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:47 smithi116 ceph-mon[37097]: pgmap v4343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:41:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:49 smithi171 ceph-mon[37789]: pgmap v4344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:49 smithi044 ceph-mon[32404]: pgmap v4344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:49 smithi116 ceph-mon[37097]: pgmap v4344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:51.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:51 smithi171 ceph-mon[37789]: pgmap v4345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:41:51.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7255.762557 secs 2024-07-26T10:41:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:51 smithi044 ceph-mon[32404]: pgmap v4345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:41:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7255.762557 secs 2024-07-26T10:41:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:51 smithi116 ceph-mon[37097]: pgmap v4345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:41:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7255.762557 secs 2024-07-26T10:41:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:53 smithi171 ceph-mon[37789]: pgmap v4346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:53 smithi044 ceph-mon[32404]: pgmap v4346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:53 smithi116 ceph-mon[37097]: pgmap v4346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:41:55.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:55 smithi116 ceph-mon[37097]: pgmap v4347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:55 smithi171 ceph-mon[37789]: pgmap v4347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:55 smithi044 ceph-mon[32404]: pgmap v4347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7260.762716 secs 2024-07-26T10:41:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7260.762716 secs 2024-07-26T10:41:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7260.762716 secs 2024-07-26T10:41:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:57 smithi171 ceph-mon[37789]: pgmap v4348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:57 smithi044 ceph-mon[32404]: pgmap v4348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:57 smithi116 ceph-mon[37097]: pgmap v4348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:41:57.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:57.680+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:41:57.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:57.680+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:41:57.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:41:57.681+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:41:59 smithi171 ceph-mon[37789]: pgmap v4349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:41:59 smithi044 ceph-mon[32404]: pgmap v4349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:41:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:41:59 smithi116 ceph-mon[37097]: pgmap v4349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:01 smithi171 ceph-mon[37789]: pgmap v4350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7265.762874 secs 2024-07-26T10:42:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:01 smithi044 ceph-mon[32404]: pgmap v4350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7265.762874 secs 2024-07-26T10:42:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:01 smithi116 ceph-mon[37097]: pgmap v4350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7265.762874 secs 2024-07-26T10:42:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:03 smithi171 ceph-mon[37789]: pgmap v4351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:03 smithi044 ceph-mon[32404]: pgmap v4351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:03 smithi116 ceph-mon[37097]: pgmap v4351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:05 smithi171 ceph-mon[37789]: pgmap v4352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:05 smithi044 ceph-mon[32404]: pgmap v4352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:05 smithi116 ceph-mon[37097]: pgmap v4352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7270.762979 secs 2024-07-26T10:42:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7270.762979 secs 2024-07-26T10:42:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7270.762979 secs 2024-07-26T10:42:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:07 smithi044 ceph-mon[32404]: pgmap v4353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:07 smithi116 ceph-mon[37097]: pgmap v4353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:07 smithi171 ceph-mon[37789]: pgmap v4353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:42:08.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:42:08.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:08 smithi044 ceph-mon[32404]: pgmap v4354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:08.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:08 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:42:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:42:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:42:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:08 smithi116 ceph-mon[37097]: pgmap v4354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:08 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:42:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:42:08.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:42:08.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:08 smithi171 ceph-mon[37789]: pgmap v4354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:08.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:08 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:42:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:10 smithi044 ceph-mon[32404]: pgmap v4355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:10 smithi116 ceph-mon[37097]: pgmap v4355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:10 smithi171 ceph-mon[37789]: pgmap v4355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7275.763144 secs 2024-07-26T10:42:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7275.763144 secs 2024-07-26T10:42:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7275.763144 secs 2024-07-26T10:42:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:12 smithi044 ceph-mon[32404]: pgmap v4356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:42:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:12 smithi116 ceph-mon[37097]: pgmap v4356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:42:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:12 smithi171 ceph-mon[37789]: pgmap v4356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:42:12.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:12.684+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:42:12.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:12.684+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:12.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:12.684+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:42:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:13 smithi044 ceph-mon[32404]: pgmap v4357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:13 smithi116 ceph-mon[37097]: pgmap v4357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:13 smithi171 ceph-mon[37789]: pgmap v4357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:16 smithi044 ceph-mon[32404]: pgmap v4358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:42:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7280.763252 secs 2024-07-26T10:42:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:16 smithi116 ceph-mon[37097]: pgmap v4358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:42:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7280.763252 secs 2024-07-26T10:42:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:16 smithi171 ceph-mon[37789]: pgmap v4358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:42:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7280.763252 secs 2024-07-26T10:42:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:18 smithi044 ceph-mon[32404]: pgmap v4359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:18 smithi116 ceph-mon[37097]: pgmap v4359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:18 smithi171 ceph-mon[37789]: pgmap v4359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:42:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:20 smithi044 ceph-mon[32404]: pgmap v4360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:20 smithi116 ceph-mon[37097]: pgmap v4360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:20 smithi171 ceph-mon[37789]: pgmap v4360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7285.763406 secs 2024-07-26T10:42:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7285.763406 secs 2024-07-26T10:42:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7285.763406 secs 2024-07-26T10:42:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:22 smithi044 ceph-mon[32404]: pgmap v4361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:22 smithi116 ceph-mon[37097]: pgmap v4361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:22 smithi171 ceph-mon[37789]: pgmap v4361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:23 smithi044 ceph-mon[32404]: pgmap v4362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:23 smithi116 ceph-mon[37097]: pgmap v4362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:23 smithi171 ceph-mon[37789]: pgmap v4362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:26 smithi044 ceph-mon[32404]: pgmap v4363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7290.763569 secs 2024-07-26T10:42:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:26 smithi116 ceph-mon[37097]: pgmap v4363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7290.763569 secs 2024-07-26T10:42:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:26 smithi171 ceph-mon[37789]: pgmap v4363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7290.763569 secs 2024-07-26T10:42:27.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:27.694+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:42:27.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:27.694+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:27.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:27.694+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:42:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:28 smithi044 ceph-mon[32404]: pgmap v4364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:28 smithi116 ceph-mon[37097]: pgmap v4364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:28 smithi171 ceph-mon[37789]: pgmap v4364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:30 smithi044 ceph-mon[32404]: pgmap v4365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:30 smithi116 ceph-mon[37097]: pgmap v4365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:30 smithi171 ceph-mon[37789]: pgmap v4365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:31.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7295.763725 secs 2024-07-26T10:42:31.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:31 smithi171 ceph-mon[37789]: pgmap v4366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7295.763725 secs 2024-07-26T10:42:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:31 smithi044 ceph-mon[32404]: pgmap v4366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7295.763725 secs 2024-07-26T10:42:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:31 smithi116 ceph-mon[37097]: pgmap v4366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:34 smithi044 ceph-mon[32404]: pgmap v4367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:34 smithi116 ceph-mon[37097]: pgmap v4367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:34 smithi171 ceph-mon[37789]: pgmap v4367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:36 smithi044 ceph-mon[32404]: pgmap v4368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7300.763891 secs 2024-07-26T10:42:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:36 smithi116 ceph-mon[37097]: pgmap v4368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7300.763891 secs 2024-07-26T10:42:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:36 smithi171 ceph-mon[37789]: pgmap v4368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7300.763891 secs 2024-07-26T10:42:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:37 smithi044 ceph-mon[32404]: pgmap v4369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:37 smithi116 ceph-mon[37097]: pgmap v4369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:37 smithi171 ceph-mon[37789]: pgmap v4369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:39 smithi171 ceph-mon[37789]: pgmap v4370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:39 smithi044 ceph-mon[32404]: pgmap v4370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:39 smithi116 ceph-mon[37097]: pgmap v4370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7305.764059 secs 2024-07-26T10:42:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7305.764059 secs 2024-07-26T10:42:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7305.764059 secs 2024-07-26T10:42:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:41 smithi171 ceph-mon[37789]: pgmap v4371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:41 smithi044 ceph-mon[32404]: pgmap v4371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:41 smithi116 ceph-mon[37097]: pgmap v4371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:42.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:42.699+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:42:42.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:42.699+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:42.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:42.699+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:42:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:44 smithi044 ceph-mon[32404]: pgmap v4372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:44 smithi116 ceph-mon[37097]: pgmap v4372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:44 smithi171 ceph-mon[37789]: pgmap v4372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:45 smithi044 ceph-mon[32404]: pgmap v4373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:45 smithi116 ceph-mon[37097]: pgmap v4373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:45 smithi171 ceph-mon[37789]: pgmap v4373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7310.764222 secs 2024-07-26T10:42:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7310.764222 secs 2024-07-26T10:42:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7310.764222 secs 2024-07-26T10:42:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:47 smithi044 ceph-mon[32404]: pgmap v4374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:47 smithi116 ceph-mon[37097]: pgmap v4374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:47 smithi171 ceph-mon[37789]: pgmap v4374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:49 smithi171 ceph-mon[37789]: pgmap v4375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:50.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:49 smithi044 ceph-mon[32404]: pgmap v4375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:49 smithi116 ceph-mon[37097]: pgmap v4375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7315.764362 secs 2024-07-26T10:42:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7315.764362 secs 2024-07-26T10:42:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7315.764362 secs 2024-07-26T10:42:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:51 smithi171 ceph-mon[37789]: pgmap v4376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:51 smithi044 ceph-mon[32404]: pgmap v4376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:51 smithi116 ceph-mon[37097]: pgmap v4376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:53 smithi116 ceph-mon[37097]: pgmap v4377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:53 smithi171 ceph-mon[37789]: pgmap v4377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:53 smithi044 ceph-mon[32404]: pgmap v4377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:42:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:55 smithi116 ceph-mon[37097]: pgmap v4378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:55 smithi171 ceph-mon[37789]: pgmap v4378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:55 smithi044 ceph-mon[32404]: pgmap v4378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:42:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7320.764509 secs 2024-07-26T10:42:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7320.764509 secs 2024-07-26T10:42:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7320.764509 secs 2024-07-26T10:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:57 smithi171 ceph-mon[37789]: pgmap v4379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:57.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:57.708+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:42:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:57.708+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:57.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:42:57.708+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:42:57.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:57 smithi044 ceph-mon[32404]: pgmap v4379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:57 smithi116 ceph-mon[37097]: pgmap v4379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:42:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:42:59 smithi116 ceph-mon[37097]: pgmap v4380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:42:59 smithi171 ceph-mon[37789]: pgmap v4380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:42:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:42:59 smithi044 ceph-mon[32404]: pgmap v4380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7325.764671 secs 2024-07-26T10:43:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7325.764671 secs 2024-07-26T10:43:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7325.764671 secs 2024-07-26T10:43:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:01 smithi171 ceph-mon[37789]: pgmap v4381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:01 smithi044 ceph-mon[32404]: pgmap v4381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:01 smithi116 ceph-mon[37097]: pgmap v4381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:03 smithi116 ceph-mon[37097]: pgmap v4382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:43:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:03 smithi171 ceph-mon[37789]: pgmap v4382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:43:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:03 smithi044 ceph-mon[32404]: pgmap v4382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:43:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:05 smithi116 ceph-mon[37097]: pgmap v4383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:05 smithi171 ceph-mon[37789]: pgmap v4383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:05 smithi044 ceph-mon[32404]: pgmap v4383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7330.764772 secs 2024-07-26T10:43:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7330.764772 secs 2024-07-26T10:43:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7330.764772 secs 2024-07-26T10:43:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:07 smithi171 ceph-mon[37789]: pgmap v4384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:07 smithi044 ceph-mon[32404]: pgmap v4384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:07 smithi116 ceph-mon[37097]: pgmap v4384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:09 smithi171 ceph-mon[37789]: pgmap v4385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:43:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:09 smithi171 ceph-mon[37789]: pgmap v4386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:09.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:43:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:09 smithi044 ceph-mon[32404]: pgmap v4385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:43:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:09 smithi044 ceph-mon[32404]: pgmap v4386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:09.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:43:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:09 smithi116 ceph-mon[37097]: pgmap v4385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:43:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:09 smithi116 ceph-mon[37097]: pgmap v4386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:43:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:11 smithi044 ceph-mon[32404]: pgmap v4387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7335.764932 secs 2024-07-26T10:43:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:11 smithi116 ceph-mon[37097]: pgmap v4387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7335.764932 secs 2024-07-26T10:43:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:11 smithi171 ceph-mon[37789]: pgmap v4387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7335.764932 secs 2024-07-26T10:43:12.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:12.712+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:43:12.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:12.712+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:43:12.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:12.712+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:13 smithi044 ceph-mon[32404]: pgmap v4388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:13 smithi116 ceph-mon[37097]: pgmap v4388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:13 smithi171 ceph-mon[37789]: pgmap v4388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:15 smithi044 ceph-mon[32404]: pgmap v4389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:15 smithi116 ceph-mon[37097]: pgmap v4389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:15 smithi171 ceph-mon[37789]: pgmap v4389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:43:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7340.765067 secs 2024-07-26T10:43:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7340.765067 secs 2024-07-26T10:43:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7340.765067 secs 2024-07-26T10:43:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:17 smithi044 ceph-mon[32404]: pgmap v4390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:17 smithi116 ceph-mon[37097]: pgmap v4390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:17 smithi171 ceph-mon[37789]: pgmap v4390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:19 smithi044 ceph-mon[32404]: pgmap v4391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:19 smithi116 ceph-mon[37097]: pgmap v4391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:19 smithi171 ceph-mon[37789]: pgmap v4391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:21 smithi116 ceph-mon[37097]: pgmap v4392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7345.765209 secs 2024-07-26T10:43:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:21 smithi171 ceph-mon[37789]: pgmap v4392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7345.765209 secs 2024-07-26T10:43:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:21 smithi044 ceph-mon[32404]: pgmap v4392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7345.765209 secs 2024-07-26T10:43:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:23 smithi116 ceph-mon[37097]: pgmap v4393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:23 smithi171 ceph-mon[37789]: pgmap v4393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:23 smithi044 ceph-mon[32404]: pgmap v4393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:25 smithi171 ceph-mon[37789]: pgmap v4394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:25 smithi044 ceph-mon[32404]: pgmap v4394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:25 smithi116 ceph-mon[37097]: pgmap v4394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:43:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7350.765349 secs 2024-07-26T10:43:26.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7350.765349 secs 2024-07-26T10:43:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7350.765349 secs 2024-07-26T10:43:27.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:27.713+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:43:27.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:27.713+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:43:27.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:27.713+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:27 smithi171 ceph-mon[37789]: pgmap v4395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:43:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:27 smithi044 ceph-mon[32404]: pgmap v4395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:43:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:27 smithi116 ceph-mon[37097]: pgmap v4395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:43:29.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:29 smithi044 ceph-mon[32404]: pgmap v4396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:43:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:29 smithi171 ceph-mon[37789]: pgmap v4396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:43:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:29 smithi116 ceph-mon[37097]: pgmap v4396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:43:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:31 smithi171 ceph-mon[37789]: pgmap v4397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7355.765493 secs 2024-07-26T10:43:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:31 smithi044 ceph-mon[32404]: pgmap v4397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7355.765493 secs 2024-07-26T10:43:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:31 smithi116 ceph-mon[37097]: pgmap v4397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7355.765493 secs 2024-07-26T10:43:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:33 smithi171 ceph-mon[37789]: pgmap v4398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:34.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:33 smithi044 ceph-mon[32404]: pgmap v4398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:33 smithi116 ceph-mon[37097]: pgmap v4398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:43:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:35 smithi171 ceph-mon[37789]: pgmap v4399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:35 smithi044 ceph-mon[32404]: pgmap v4399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:35 smithi116 ceph-mon[37097]: pgmap v4399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7360.765656 secs 2024-07-26T10:43:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7360.765656 secs 2024-07-26T10:43:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7360.765656 secs 2024-07-26T10:43:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:37 smithi171 ceph-mon[37789]: pgmap v4400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:37 smithi044 ceph-mon[32404]: pgmap v4400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:37 smithi116 ceph-mon[37097]: pgmap v4400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:39 smithi171 ceph-mon[37789]: pgmap v4401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:43:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:39 smithi044 ceph-mon[32404]: pgmap v4401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:43:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:39 smithi116 ceph-mon[37097]: pgmap v4401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:43:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7365.765810 secs 2024-07-26T10:43:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:41 smithi044 ceph-mon[32404]: pgmap v4402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7365.765810 secs 2024-07-26T10:43:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:41 smithi116 ceph-mon[37097]: pgmap v4402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7365.765810 secs 2024-07-26T10:43:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:41 smithi171 ceph-mon[37789]: pgmap v4402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:42.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:42.717+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:43:42.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:42.717+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:43:42.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:42.717+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:43 smithi044 ceph-mon[32404]: pgmap v4403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:43 smithi116 ceph-mon[37097]: pgmap v4403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:43 smithi171 ceph-mon[37789]: pgmap v4403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:43:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:45 smithi044 ceph-mon[32404]: pgmap v4404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:45 smithi116 ceph-mon[37097]: pgmap v4404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:45 smithi171 ceph-mon[37789]: pgmap v4404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:43:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7370.765952 secs 2024-07-26T10:43:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7370.765952 secs 2024-07-26T10:43:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7370.765952 secs 2024-07-26T10:43:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:47 smithi044 ceph-mon[32404]: pgmap v4405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:47 smithi116 ceph-mon[37097]: pgmap v4405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:47 smithi171 ceph-mon[37789]: pgmap v4405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:43:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:49 smithi044 ceph-mon[32404]: pgmap v4406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:43:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:49 smithi116 ceph-mon[37097]: pgmap v4406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:49 smithi171 ceph-mon[37789]: pgmap v4406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:43:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7375.766109 secs 2024-07-26T10:43:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:51 smithi044 ceph-mon[32404]: pgmap v4407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7375.766109 secs 2024-07-26T10:43:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:51 smithi116 ceph-mon[37097]: pgmap v4407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7375.766109 secs 2024-07-26T10:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:51 smithi171 ceph-mon[37789]: pgmap v4407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:43:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:53 smithi044 ceph-mon[32404]: pgmap v4408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:53 smithi116 ceph-mon[37097]: pgmap v4408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:53 smithi171 ceph-mon[37789]: pgmap v4408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:43:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:55 smithi044 ceph-mon[32404]: pgmap v4409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:43:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:55 smithi116 ceph-mon[37097]: pgmap v4409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:43:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:55 smithi171 ceph-mon[37789]: pgmap v4409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:43:56.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7380.766238 secs 2024-07-26T10:43:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7380.766238 secs 2024-07-26T10:43:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7380.766238 secs 2024-07-26T10:43:57.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:57.726+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:43:57.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:57.726+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:43:57.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:43:57.726+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:57 smithi044 ceph-mon[32404]: pgmap v4410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:43:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:57 smithi116 ceph-mon[37097]: pgmap v4410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:43:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:57 smithi171 ceph-mon[37789]: pgmap v4410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:44:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:43:59 smithi044 ceph-mon[32404]: pgmap v4411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:44:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:43:59 smithi116 ceph-mon[37097]: pgmap v4411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:44:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:43:59 smithi171 ceph-mon[37789]: pgmap v4411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:44:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7385.766404 secs 2024-07-26T10:44:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:01 smithi044 ceph-mon[32404]: pgmap v4412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7385.766404 secs 2024-07-26T10:44:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:01 smithi116 ceph-mon[37097]: pgmap v4412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7385.766404 secs 2024-07-26T10:44:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:01 smithi171 ceph-mon[37789]: pgmap v4412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:03 smithi044 ceph-mon[32404]: pgmap v4413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:03 smithi116 ceph-mon[37097]: pgmap v4413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:03 smithi171 ceph-mon[37789]: pgmap v4413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:05 smithi044 ceph-mon[32404]: pgmap v4414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:05 smithi116 ceph-mon[37097]: pgmap v4414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:05 smithi171 ceph-mon[37789]: pgmap v4414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7390.766549 secs 2024-07-26T10:44:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7390.766549 secs 2024-07-26T10:44:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7390.766549 secs 2024-07-26T10:44:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:07 smithi044 ceph-mon[32404]: pgmap v4415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:07 smithi116 ceph-mon[37097]: pgmap v4415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:07 smithi171 ceph-mon[37789]: pgmap v4415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:09 smithi044 ceph-mon[32404]: pgmap v4416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:44:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:09 smithi116 ceph-mon[37097]: pgmap v4416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:44:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:09 smithi171 ceph-mon[37789]: pgmap v4416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:44:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:10 smithi044 ceph-mon[32404]: pgmap v4417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:44:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:10 smithi044 ceph-mon[32404]: pgmap v4418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:44:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:10 smithi116 ceph-mon[37097]: pgmap v4417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:44:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:10 smithi116 ceph-mon[37097]: pgmap v4418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:44:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:10 smithi171 ceph-mon[37789]: pgmap v4417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:44:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:10 smithi171 ceph-mon[37789]: pgmap v4418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:44:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7395.766717 secs 2024-07-26T10:44:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7395.766717 secs 2024-07-26T10:44:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7395.766717 secs 2024-07-26T10:44:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:12.731+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:44:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:12.731+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:44:12.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:12.731+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:12 smithi044 ceph-mon[32404]: pgmap v4419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:44:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:12 smithi116 ceph-mon[37097]: pgmap v4419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:44:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:12 smithi171 ceph-mon[37789]: pgmap v4419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:44:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:14 smithi044 ceph-mon[32404]: pgmap v4420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:14 smithi116 ceph-mon[37097]: pgmap v4420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:14 smithi171 ceph-mon[37789]: pgmap v4420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:15.990+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:44:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:15.990+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:44:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:15.991+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:44:15.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:15.991+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:44:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7400.766878 secs 2024-07-26T10:44:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:16 smithi044 ceph-mon[32404]: pgmap v4421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7400.766878 secs 2024-07-26T10:44:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:16 smithi116 ceph-mon[37097]: pgmap v4421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7400.766878 secs 2024-07-26T10:44:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:16 smithi171 ceph-mon[37789]: pgmap v4421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:44:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:18 smithi044 ceph-mon[32404]: pgmap v4422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:44:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:18 smithi116 ceph-mon[37097]: pgmap v4422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:44:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:18 smithi171 ceph-mon[37789]: pgmap v4422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:44:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:20 smithi044 ceph-mon[32404]: pgmap v4423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:44:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7405.767029 secs 2024-07-26T10:44:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:20 smithi116 ceph-mon[37097]: pgmap v4423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:44:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7405.767029 secs 2024-07-26T10:44:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:20 smithi171 ceph-mon[37789]: pgmap v4423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:44:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7405.767029 secs 2024-07-26T10:44:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:22 smithi044 ceph-mon[32404]: pgmap v4424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:22 smithi116 ceph-mon[37097]: pgmap v4424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:22 smithi171 ceph-mon[37789]: pgmap v4424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:24 smithi044 ceph-mon[32404]: pgmap v4425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:24 smithi116 ceph-mon[37097]: pgmap v4425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:24 smithi171 ceph-mon[37789]: pgmap v4425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7410.767197 secs 2024-07-26T10:44:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7410.767197 secs 2024-07-26T10:44:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7410.767197 secs 2024-07-26T10:44:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:26 smithi116 ceph-mon[37097]: pgmap v4426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:26 smithi171 ceph-mon[37789]: pgmap v4426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:26 smithi044 ceph-mon[32404]: pgmap v4426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:27.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:27.739+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:44:27.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:27.739+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:44:27.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:27.739+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:28 smithi171 ceph-mon[37789]: pgmap v4427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:28 smithi044 ceph-mon[32404]: pgmap v4427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:28 smithi116 ceph-mon[37097]: pgmap v4427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:30 smithi171 ceph-mon[37789]: pgmap v4428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:31.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7415.767346 secs 2024-07-26T10:44:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:30 smithi044 ceph-mon[32404]: pgmap v4428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7415.767346 secs 2024-07-26T10:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:30 smithi116 ceph-mon[37097]: pgmap v4428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7415.767346 secs 2024-07-26T10:44:33.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:32 smithi044 ceph-mon[32404]: pgmap v4429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:32 smithi171 ceph-mon[37789]: pgmap v4429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:33.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:32 smithi116 ceph-mon[37097]: pgmap v4429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:34 smithi171 ceph-mon[37789]: pgmap v4430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:34 smithi044 ceph-mon[32404]: pgmap v4430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:34 smithi116 ceph-mon[37097]: pgmap v4430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7420.767497 secs 2024-07-26T10:44:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7420.767497 secs 2024-07-26T10:44:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7420.767497 secs 2024-07-26T10:44:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:36 smithi171 ceph-mon[37789]: pgmap v4431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:36 smithi044 ceph-mon[32404]: pgmap v4431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:36 smithi116 ceph-mon[37097]: pgmap v4431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:38 smithi171 ceph-mon[37789]: pgmap v4432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:38 smithi044 ceph-mon[32404]: pgmap v4432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:38 smithi116 ceph-mon[37097]: pgmap v4432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:40 smithi171 ceph-mon[37789]: pgmap v4433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7425.767685 secs 2024-07-26T10:44:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:40 smithi044 ceph-mon[32404]: pgmap v4433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7425.767685 secs 2024-07-26T10:44:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:40 smithi116 ceph-mon[37097]: pgmap v4433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7425.767685 secs 2024-07-26T10:44:42.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:42.743+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:44:42.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:42.743+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:44:42.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:42.744+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:42 smithi044 ceph-mon[32404]: pgmap v4434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:42 smithi171 ceph-mon[37789]: pgmap v4434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:42 smithi116 ceph-mon[37097]: pgmap v4434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:44:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:44 smithi171 ceph-mon[37789]: pgmap v4435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:44 smithi044 ceph-mon[32404]: pgmap v4435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:44 smithi116 ceph-mon[37097]: pgmap v4435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7430.767780 secs 2024-07-26T10:44:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7430.767780 secs 2024-07-26T10:44:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7430.767780 secs 2024-07-26T10:44:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:46 smithi044 ceph-mon[32404]: pgmap v4436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:46 smithi116 ceph-mon[37097]: pgmap v4436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:46 smithi171 ceph-mon[37789]: pgmap v4436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:48 smithi044 ceph-mon[32404]: pgmap v4437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:48 smithi116 ceph-mon[37097]: pgmap v4437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:48 smithi171 ceph-mon[37789]: pgmap v4437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:50 smithi044 ceph-mon[32404]: pgmap v4438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7435.767881 secs 2024-07-26T10:44:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:50 smithi116 ceph-mon[37097]: pgmap v4438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7435.767881 secs 2024-07-26T10:44:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:50 smithi171 ceph-mon[37789]: pgmap v4438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:44:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7435.767881 secs 2024-07-26T10:44:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:52 smithi044 ceph-mon[32404]: pgmap v4439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:52 smithi116 ceph-mon[37097]: pgmap v4439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:52 smithi171 ceph-mon[37789]: pgmap v4439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:54 smithi044 ceph-mon[32404]: pgmap v4440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:54 smithi116 ceph-mon[37097]: pgmap v4440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:54 smithi171 ceph-mon[37789]: pgmap v4440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:44:56.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7440.768030 secs 2024-07-26T10:44:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7440.768030 secs 2024-07-26T10:44:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7440.768030 secs 2024-07-26T10:44:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:56 smithi044 ceph-mon[32404]: pgmap v4441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:56 smithi116 ceph-mon[37097]: pgmap v4441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:56 smithi171 ceph-mon[37789]: pgmap v4441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:44:57.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:57.746+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:44:57.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:57.746+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:44:57.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:44:57.746+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:44:58 smithi044 ceph-mon[32404]: pgmap v4442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:44:58 smithi116 ceph-mon[37097]: pgmap v4442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:44:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:44:58 smithi171 ceph-mon[37789]: pgmap v4442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:01.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:00.999+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:45:01.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:00.999+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:45:01.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:00.999+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:45:01.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:01.000+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:45:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:00 smithi044 ceph-mon[32404]: pgmap v4443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7445.768186 secs 2024-07-26T10:45:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:00 smithi116 ceph-mon[37097]: pgmap v4443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7445.768186 secs 2024-07-26T10:45:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:00 smithi171 ceph-mon[37789]: pgmap v4443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7445.768186 secs 2024-07-26T10:45:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:02 smithi044 ceph-mon[32404]: pgmap v4444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:02 smithi116 ceph-mon[37097]: pgmap v4444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:02 smithi171 ceph-mon[37789]: pgmap v4444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:04 smithi044 ceph-mon[32404]: pgmap v4445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:04 smithi116 ceph-mon[37097]: pgmap v4445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:04 smithi171 ceph-mon[37789]: pgmap v4445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7450.768348 secs 2024-07-26T10:45:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7450.768348 secs 2024-07-26T10:45:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7450.768348 secs 2024-07-26T10:45:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:06 smithi044 ceph-mon[32404]: pgmap v4446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:06 smithi116 ceph-mon[37097]: pgmap v4446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:06 smithi171 ceph-mon[37789]: pgmap v4446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:08 smithi044 ceph-mon[32404]: pgmap v4447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:08 smithi116 ceph-mon[37097]: pgmap v4447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:08 smithi171 ceph-mon[37789]: pgmap v4447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: pgmap v4448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7455.768453 secs 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:11.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:45:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: pgmap v4448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7455.768453 secs 2024-07-26T10:45:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:11.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:45:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: pgmap v4448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:45:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7455.768453 secs 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:45:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:11 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:45:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:11 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:45:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:11 smithi044 ceph-mon[32404]: pgmap v4449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:45:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:11 smithi044 ceph-mon[32404]: pgmap v4450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:45:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:11 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:45:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:11 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:45:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:11 smithi116 ceph-mon[37097]: pgmap v4449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:45:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:11 smithi116 ceph-mon[37097]: pgmap v4450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:45:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:11 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:45:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:11 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:45:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:11 smithi171 ceph-mon[37789]: pgmap v4449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:45:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:11 smithi171 ceph-mon[37789]: pgmap v4450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:45:12.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:12.750+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:45:12.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:12.750+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:45:12.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:12.750+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:13 smithi044 ceph-mon[32404]: pgmap v4451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:13 smithi116 ceph-mon[37097]: pgmap v4451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:13 smithi171 ceph-mon[37789]: pgmap v4451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:16 smithi044 ceph-mon[32404]: pgmap v4452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7460.768607 secs 2024-07-26T10:45:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:16 smithi116 ceph-mon[37097]: pgmap v4452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7460.768607 secs 2024-07-26T10:45:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:16 smithi171 ceph-mon[37789]: pgmap v4452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7460.768607 secs 2024-07-26T10:45:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:18 smithi044 ceph-mon[32404]: pgmap v4453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:18 smithi116 ceph-mon[37097]: pgmap v4453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:18 smithi171 ceph-mon[37789]: pgmap v4453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:20 smithi044 ceph-mon[32404]: pgmap v4454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:20 smithi116 ceph-mon[37097]: pgmap v4454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:20 smithi171 ceph-mon[37789]: pgmap v4454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:45:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7465.768773 secs 2024-07-26T10:45:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7465.768773 secs 2024-07-26T10:45:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7465.768773 secs 2024-07-26T10:45:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:22 smithi171 ceph-mon[37789]: pgmap v4455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:45:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:22 smithi044 ceph-mon[32404]: pgmap v4455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:45:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:22 smithi116 ceph-mon[37097]: pgmap v4455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:45:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:24 smithi171 ceph-mon[37789]: pgmap v4456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:24 smithi044 ceph-mon[32404]: pgmap v4456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:24 smithi116 ceph-mon[37097]: pgmap v4456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:26.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:26 smithi116 ceph-mon[37097]: pgmap v4457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:26.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7470.768935 secs 2024-07-26T10:45:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:26 smithi171 ceph-mon[37789]: pgmap v4457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7470.768935 secs 2024-07-26T10:45:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:26 smithi044 ceph-mon[32404]: pgmap v4457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7470.768935 secs 2024-07-26T10:45:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:27.759+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:45:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:27.759+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:45:27.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:27.759+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:28 smithi171 ceph-mon[37789]: pgmap v4458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:28 smithi044 ceph-mon[32404]: pgmap v4458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:28 smithi116 ceph-mon[37097]: pgmap v4458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:30 smithi171 ceph-mon[37789]: pgmap v4459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:30 smithi044 ceph-mon[32404]: pgmap v4459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:30 smithi116 ceph-mon[37097]: pgmap v4459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7475.769085 secs 2024-07-26T10:45:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7475.769085 secs 2024-07-26T10:45:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7475.769085 secs 2024-07-26T10:45:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:32 smithi171 ceph-mon[37789]: pgmap v4460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:32 smithi044 ceph-mon[32404]: pgmap v4460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:32 smithi116 ceph-mon[37097]: pgmap v4460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:34.322 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:34 smithi044 ceph-mon[32404]: pgmap v4461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:34 smithi171 ceph-mon[37789]: pgmap v4461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:34 smithi116 ceph-mon[37097]: pgmap v4461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:36 smithi171 ceph-mon[37789]: pgmap v4462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7480.769250 secs 2024-07-26T10:45:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:36 smithi044 ceph-mon[32404]: pgmap v4462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7480.769250 secs 2024-07-26T10:45:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:36 smithi116 ceph-mon[37097]: pgmap v4462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7480.769250 secs 2024-07-26T10:45:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:38 smithi171 ceph-mon[37789]: pgmap v4463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:38 smithi044 ceph-mon[32404]: pgmap v4463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:38 smithi116 ceph-mon[37097]: pgmap v4463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:40 smithi044 ceph-mon[32404]: pgmap v4464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:40 smithi116 ceph-mon[37097]: pgmap v4464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:40 smithi171 ceph-mon[37789]: pgmap v4464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7485.769411 secs 2024-07-26T10:45:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7485.769411 secs 2024-07-26T10:45:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7485.769411 secs 2024-07-26T10:45:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:42 smithi044 ceph-mon[32404]: pgmap v4465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:42 smithi116 ceph-mon[37097]: pgmap v4465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:42 smithi171 ceph-mon[37789]: pgmap v4465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:42.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:42.763+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:45:42.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:42.763+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:42.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:42.763+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:45:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:44 smithi044 ceph-mon[32404]: pgmap v4466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:44 smithi116 ceph-mon[37097]: pgmap v4466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:44 smithi171 ceph-mon[37789]: pgmap v4466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:46 smithi044 ceph-mon[32404]: pgmap v4467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7490.769576 secs 2024-07-26T10:45:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:46 smithi116 ceph-mon[37097]: pgmap v4467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7490.769576 secs 2024-07-26T10:45:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:46 smithi171 ceph-mon[37789]: pgmap v4467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:45:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7490.769576 secs 2024-07-26T10:45:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:48 smithi044 ceph-mon[32404]: pgmap v4468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:48 smithi116 ceph-mon[37097]: pgmap v4468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:48 smithi171 ceph-mon[37789]: pgmap v4468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:49 smithi044 ceph-mon[32404]: pgmap v4469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:49 smithi116 ceph-mon[37097]: pgmap v4469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:49 smithi171 ceph-mon[37789]: pgmap v4469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7495.769746 secs 2024-07-26T10:45:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7495.769746 secs 2024-07-26T10:45:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7495.769746 secs 2024-07-26T10:45:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:51 smithi171 ceph-mon[37789]: pgmap v4470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:45:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:51 smithi044 ceph-mon[32404]: pgmap v4470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:45:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:51 smithi116 ceph-mon[37097]: pgmap v4470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:45:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:54 smithi171 ceph-mon[37789]: pgmap v4471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:54 smithi044 ceph-mon[32404]: pgmap v4471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:54 smithi116 ceph-mon[37097]: pgmap v4471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:45:56.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:56 smithi116 ceph-mon[37097]: pgmap v4472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:45:56.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7500.769916 secs 2024-07-26T10:45:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:56 smithi171 ceph-mon[37789]: pgmap v4472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:45:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7500.769916 secs 2024-07-26T10:45:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:56 smithi044 ceph-mon[32404]: pgmap v4472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:45:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7500.769916 secs 2024-07-26T10:45:57.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:57.772+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:45:57.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:57.772+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:45:57.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:45:57.772+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:45:58 smithi171 ceph-mon[37789]: pgmap v4473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:45:58 smithi044 ceph-mon[32404]: pgmap v4473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:45:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:45:58 smithi116 ceph-mon[37097]: pgmap v4473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:00 smithi171 ceph-mon[37789]: pgmap v4474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:00 smithi044 ceph-mon[32404]: pgmap v4474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:00 smithi116 ceph-mon[37097]: pgmap v4474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7505.770087 secs 2024-07-26T10:46:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7505.770087 secs 2024-07-26T10:46:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7505.770087 secs 2024-07-26T10:46:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:02 smithi171 ceph-mon[37789]: pgmap v4475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:46:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:02 smithi044 ceph-mon[32404]: pgmap v4475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:46:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:02 smithi116 ceph-mon[37097]: pgmap v4475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:46:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:04 smithi171 ceph-mon[37789]: pgmap v4476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:04 smithi044 ceph-mon[32404]: pgmap v4476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:04 smithi116 ceph-mon[37097]: pgmap v4476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:06 smithi171 ceph-mon[37789]: pgmap v4477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7510.770228 secs 2024-07-26T10:46:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:06 smithi044 ceph-mon[32404]: pgmap v4477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7510.770228 secs 2024-07-26T10:46:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:06 smithi116 ceph-mon[37097]: pgmap v4477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7510.770228 secs 2024-07-26T10:46:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:08 smithi044 ceph-mon[32404]: pgmap v4478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:08 smithi116 ceph-mon[37097]: pgmap v4478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:08 smithi171 ceph-mon[37789]: pgmap v4478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:10 smithi044 ceph-mon[32404]: pgmap v4479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:10 smithi116 ceph-mon[37097]: pgmap v4479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:10 smithi171 ceph-mon[37789]: pgmap v4479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7515.770400 secs 2024-07-26T10:46:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7515.770400 secs 2024-07-26T10:46:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7515.770400 secs 2024-07-26T10:46:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: pgmap v4480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:12.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: pgmap v4480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: pgmap v4480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:46:12.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:12.777+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:46:12.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:12.777+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:46:12.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:12.777+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: pgmap v4481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:46:13.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: pgmap v4482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:46:13.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: pgmap v4481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: pgmap v4482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:46:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:46:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:46:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:46:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:46:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:46:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:13.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:13.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: pgmap v4481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:46:13.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: pgmap v4482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:46:13.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:46:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:14 smithi171 ceph-mon[37789]: pgmap v4483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:14 smithi044 ceph-mon[32404]: pgmap v4483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:14 smithi116 ceph-mon[37097]: pgmap v4483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7520.770570 secs 2024-07-26T10:46:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7520.770570 secs 2024-07-26T10:46:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7520.770570 secs 2024-07-26T10:46:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:16 smithi171 ceph-mon[37789]: pgmap v4484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:46:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:16 smithi044 ceph-mon[32404]: pgmap v4484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:46:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:16 smithi116 ceph-mon[37097]: pgmap v4484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:46:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:18 smithi171 ceph-mon[37789]: pgmap v4485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:18 smithi044 ceph-mon[32404]: pgmap v4485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:18 smithi116 ceph-mon[37097]: pgmap v4485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:20 smithi171 ceph-mon[37789]: pgmap v4486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:20.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:20 smithi044 ceph-mon[32404]: pgmap v4486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:20 smithi116 ceph-mon[37097]: pgmap v4486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:46:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7525.770742 secs 2024-07-26T10:46:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7525.770742 secs 2024-07-26T10:46:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7525.770742 secs 2024-07-26T10:46:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:22 smithi171 ceph-mon[37789]: pgmap v4487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:46:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:22 smithi044 ceph-mon[32404]: pgmap v4487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:46:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:22 smithi116 ceph-mon[37097]: pgmap v4487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:46:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:24 smithi171 ceph-mon[37789]: pgmap v4488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:24 smithi044 ceph-mon[32404]: pgmap v4488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:24 smithi116 ceph-mon[37097]: pgmap v4488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7530.770909 secs 2024-07-26T10:46:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7530.770909 secs 2024-07-26T10:46:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7530.770909 secs 2024-07-26T10:46:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:26 smithi171 ceph-mon[37789]: pgmap v4489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:26 smithi044 ceph-mon[32404]: pgmap v4489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:26 smithi116 ceph-mon[37097]: pgmap v4489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:27.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:27.787+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:46:27.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:27.787+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:46:27.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:27.787+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:28 smithi171 ceph-mon[37789]: pgmap v4490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:28 smithi044 ceph-mon[32404]: pgmap v4490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:28 smithi116 ceph-mon[37097]: pgmap v4490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:30 smithi171 ceph-mon[37789]: pgmap v4491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:30 smithi044 ceph-mon[32404]: pgmap v4491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:30 smithi116 ceph-mon[37097]: pgmap v4491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7535.771054 secs 2024-07-26T10:46:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7535.771054 secs 2024-07-26T10:46:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7535.771054 secs 2024-07-26T10:46:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:32 smithi171 ceph-mon[37789]: pgmap v4492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:32 smithi044 ceph-mon[32404]: pgmap v4492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:32 smithi116 ceph-mon[37097]: pgmap v4492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:34.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:34 smithi044 ceph-mon[32404]: pgmap v4493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:34 smithi171 ceph-mon[37789]: pgmap v4493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:34 smithi116 ceph-mon[37097]: pgmap v4493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7540.771212 secs 2024-07-26T10:46:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7540.771212 secs 2024-07-26T10:46:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7540.771212 secs 2024-07-26T10:46:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:36 smithi044 ceph-mon[32404]: pgmap v4494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:36 smithi171 ceph-mon[37789]: pgmap v4494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:36 smithi116 ceph-mon[37097]: pgmap v4494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:38 smithi171 ceph-mon[37789]: pgmap v4495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:38 smithi044 ceph-mon[32404]: pgmap v4495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:38 smithi116 ceph-mon[37097]: pgmap v4495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:40 smithi171 ceph-mon[37789]: pgmap v4496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:40 smithi044 ceph-mon[32404]: pgmap v4496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:40 smithi116 ceph-mon[37097]: pgmap v4496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7545.771385 secs 2024-07-26T10:46:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7545.771385 secs 2024-07-26T10:46:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7545.771385 secs 2024-07-26T10:46:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:42 smithi171 ceph-mon[37789]: pgmap v4497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:42 smithi044 ceph-mon[32404]: pgmap v4497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:42 smithi116 ceph-mon[37097]: pgmap v4497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:42.791+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:46:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:42.791+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:42.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:42.791+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:46:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:44 smithi171 ceph-mon[37789]: pgmap v4498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:44 smithi044 ceph-mon[32404]: pgmap v4498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:44 smithi116 ceph-mon[37097]: pgmap v4498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7550.771535 secs 2024-07-26T10:46:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7550.771535 secs 2024-07-26T10:46:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7550.771535 secs 2024-07-26T10:46:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:46 smithi044 ceph-mon[32404]: pgmap v4499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:46 smithi116 ceph-mon[37097]: pgmap v4499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:46 smithi171 ceph-mon[37789]: pgmap v4499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:48 smithi171 ceph-mon[37789]: pgmap v4500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:48 smithi044 ceph-mon[32404]: pgmap v4500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:48 smithi116 ceph-mon[37097]: pgmap v4500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:50 smithi171 ceph-mon[37789]: pgmap v4501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:50 smithi044 ceph-mon[32404]: pgmap v4501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:50 smithi116 ceph-mon[37097]: pgmap v4501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7555.771705 secs 2024-07-26T10:46:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7555.771705 secs 2024-07-26T10:46:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7555.771705 secs 2024-07-26T10:46:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:52 smithi171 ceph-mon[37789]: pgmap v4502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:52 smithi044 ceph-mon[32404]: pgmap v4502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:52 smithi116 ceph-mon[37097]: pgmap v4502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:46:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:54 smithi044 ceph-mon[32404]: pgmap v4503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:54 smithi116 ceph-mon[37097]: pgmap v4503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:54 smithi171 ceph-mon[37789]: pgmap v4503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:46:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7560.771868 secs 2024-07-26T10:46:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7560.771868 secs 2024-07-26T10:46:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7560.771868 secs 2024-07-26T10:46:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:56 smithi044 ceph-mon[32404]: pgmap v4504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:56 smithi116 ceph-mon[37097]: pgmap v4504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:56 smithi171 ceph-mon[37789]: pgmap v4504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:46:57.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:57.792+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:46:57.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:57.792+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:46:57.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:46:57.792+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:46:58 smithi171 ceph-mon[37789]: pgmap v4505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:46:58 smithi044 ceph-mon[32404]: pgmap v4505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:46:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:46:58 smithi116 ceph-mon[37097]: pgmap v4505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:47:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:00 smithi171 ceph-mon[37789]: pgmap v4506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:00 smithi044 ceph-mon[32404]: pgmap v4506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:00 smithi116 ceph-mon[37097]: pgmap v4506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7565.772049 secs 2024-07-26T10:47:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7565.772049 secs 2024-07-26T10:47:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7565.772049 secs 2024-07-26T10:47:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:02 smithi044 ceph-mon[32404]: pgmap v4507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:02 smithi116 ceph-mon[37097]: pgmap v4507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:02 smithi171 ceph-mon[37789]: pgmap v4507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:04 smithi171 ceph-mon[37789]: pgmap v4508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:04 smithi044 ceph-mon[32404]: pgmap v4508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:04 smithi116 ceph-mon[37097]: pgmap v4508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7570.772220 secs 2024-07-26T10:47:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7570.772220 secs 2024-07-26T10:47:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7570.772220 secs 2024-07-26T10:47:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:06 smithi171 ceph-mon[37789]: pgmap v4509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:06 smithi044 ceph-mon[32404]: pgmap v4509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:06 smithi116 ceph-mon[37097]: pgmap v4509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:09 smithi171 ceph-mon[37789]: pgmap v4510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:47:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:09 smithi044 ceph-mon[32404]: pgmap v4510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:47:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:09 smithi116 ceph-mon[37097]: pgmap v4510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:47:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:11 smithi171 ceph-mon[37789]: pgmap v4511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7575.772381 secs 2024-07-26T10:47:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:11 smithi044 ceph-mon[32404]: pgmap v4511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7575.772381 secs 2024-07-26T10:47:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:11 smithi116 ceph-mon[37097]: pgmap v4511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7575.772381 secs 2024-07-26T10:47:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:12.796+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:47:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:12.796+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:12.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:12.796+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:47:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:13 smithi044 ceph-mon[32404]: pgmap v4512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:13 smithi116 ceph-mon[37097]: pgmap v4512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:13 smithi171 ceph-mon[37789]: pgmap v4512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:14 smithi044 ceph-mon[32404]: pgmap v4513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:47:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:14 smithi044 ceph-mon[32404]: pgmap v4514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:47:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:14 smithi116 ceph-mon[37097]: pgmap v4513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:47:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:14 smithi116 ceph-mon[37097]: pgmap v4514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:47:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:14 smithi171 ceph-mon[37789]: pgmap v4513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:47:14.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:14 smithi171 ceph-mon[37789]: pgmap v4514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:14.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:47:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:16.024+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:47:16.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:16.024+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:47:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:16.024+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:47:16.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:16.025+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:47:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7580.772524 secs 2024-07-26T10:47:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:16 smithi171 ceph-mon[37789]: pgmap v4515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7580.772524 secs 2024-07-26T10:47:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:16 smithi044 ceph-mon[32404]: pgmap v4515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7580.772524 secs 2024-07-26T10:47:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:16 smithi116 ceph-mon[37097]: pgmap v4515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:18 smithi171 ceph-mon[37789]: pgmap v4516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:47:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:18 smithi044 ceph-mon[32404]: pgmap v4516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:47:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:18 smithi116 ceph-mon[37097]: pgmap v4516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:47:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:20 smithi171 ceph-mon[37789]: pgmap v4517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:20 smithi044 ceph-mon[32404]: pgmap v4517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:20 smithi116 ceph-mon[37097]: pgmap v4517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:47:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7585.772665 secs 2024-07-26T10:47:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7585.772665 secs 2024-07-26T10:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7585.772665 secs 2024-07-26T10:47:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:22 smithi044 ceph-mon[32404]: pgmap v4518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:47:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:22 smithi116 ceph-mon[37097]: pgmap v4518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:47:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:22 smithi171 ceph-mon[37789]: pgmap v4518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:47:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:24 smithi044 ceph-mon[32404]: pgmap v4519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:47:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:24 smithi116 ceph-mon[37097]: pgmap v4519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:47:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:24 smithi171 ceph-mon[37789]: pgmap v4519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:47:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7590.772825 secs 2024-07-26T10:47:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:26 smithi044 ceph-mon[32404]: pgmap v4520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7590.772825 secs 2024-07-26T10:47:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:26 smithi116 ceph-mon[37097]: pgmap v4520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7590.772825 secs 2024-07-26T10:47:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:26 smithi171 ceph-mon[37789]: pgmap v4520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:27.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:27.805+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:47:27.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:27.805+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:47:27.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:27.805+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:28 smithi044 ceph-mon[32404]: pgmap v4521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:28 smithi116 ceph-mon[37097]: pgmap v4521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:28 smithi171 ceph-mon[37789]: pgmap v4521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:30 smithi044 ceph-mon[32404]: pgmap v4522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:30 smithi116 ceph-mon[37097]: pgmap v4522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:30 smithi171 ceph-mon[37789]: pgmap v4522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7595.772975 secs 2024-07-26T10:47:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7595.772975 secs 2024-07-26T10:47:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7595.772975 secs 2024-07-26T10:47:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:32 smithi044 ceph-mon[32404]: pgmap v4523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:32 smithi116 ceph-mon[37097]: pgmap v4523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:32 smithi171 ceph-mon[37789]: pgmap v4523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:34 smithi044 ceph-mon[32404]: pgmap v4524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:47:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:34 smithi116 ceph-mon[37097]: pgmap v4524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:47:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:34 smithi171 ceph-mon[37789]: pgmap v4524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:47:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7600.773091 secs 2024-07-26T10:47:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:36 smithi044 ceph-mon[32404]: pgmap v4525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:47:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7600.773091 secs 2024-07-26T10:47:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:36 smithi116 ceph-mon[37097]: pgmap v4525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:47:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7600.773091 secs 2024-07-26T10:47:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:36 smithi171 ceph-mon[37789]: pgmap v4525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:47:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:38 smithi044 ceph-mon[32404]: pgmap v4526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:47:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:38 smithi116 ceph-mon[37097]: pgmap v4526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:47:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:38 smithi171 ceph-mon[37789]: pgmap v4526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:47:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:40 smithi044 ceph-mon[32404]: pgmap v4527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:47:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:40 smithi116 ceph-mon[37097]: pgmap v4527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:47:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:40 smithi171 ceph-mon[37789]: pgmap v4527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:47:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7605.773235 secs 2024-07-26T10:47:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7605.773235 secs 2024-07-26T10:47:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7605.773235 secs 2024-07-26T10:47:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:42.809+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:47:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:42.809+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:47:42.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:42.809+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:42 smithi044 ceph-mon[32404]: pgmap v4528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:42 smithi116 ceph-mon[37097]: pgmap v4528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:42 smithi171 ceph-mon[37789]: pgmap v4528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:44 smithi044 ceph-mon[32404]: pgmap v4529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:47:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:44 smithi116 ceph-mon[37097]: pgmap v4529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:47:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:44 smithi171 ceph-mon[37789]: pgmap v4529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:47:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7610.773373 secs 2024-07-26T10:47:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:46 smithi044 ceph-mon[32404]: pgmap v4530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7610.773373 secs 2024-07-26T10:47:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:46 smithi116 ceph-mon[37097]: pgmap v4530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7610.773373 secs 2024-07-26T10:47:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:46 smithi171 ceph-mon[37789]: pgmap v4530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:48 smithi044 ceph-mon[32404]: pgmap v4531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:48 smithi116 ceph-mon[37097]: pgmap v4531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:48 smithi171 ceph-mon[37789]: pgmap v4531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:50 smithi044 ceph-mon[32404]: pgmap v4532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:50 smithi116 ceph-mon[37097]: pgmap v4532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:50 smithi171 ceph-mon[37789]: pgmap v4532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:47:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7615.773529 secs 2024-07-26T10:47:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7615.773529 secs 2024-07-26T10:47:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7615.773529 secs 2024-07-26T10:47:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:52 smithi044 ceph-mon[32404]: pgmap v4533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:52 smithi116 ceph-mon[37097]: pgmap v4533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:52 smithi171 ceph-mon[37789]: pgmap v4533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:47:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:54 smithi044 ceph-mon[32404]: pgmap v4534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:55.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:54 smithi116 ceph-mon[37097]: pgmap v4534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:54 smithi171 ceph-mon[37789]: pgmap v4534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:47:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7620.773672 secs 2024-07-26T10:47:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:56 smithi044 ceph-mon[32404]: pgmap v4535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7620.773672 secs 2024-07-26T10:47:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:56 smithi116 ceph-mon[37097]: pgmap v4535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7620.773672 secs 2024-07-26T10:47:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:56 smithi171 ceph-mon[37789]: pgmap v4535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:47:57.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:57.819+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:47:57.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:57.819+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:47:57.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:47:57.819+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:47:58 smithi044 ceph-mon[32404]: pgmap v4536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:47:58 smithi116 ceph-mon[37097]: pgmap v4536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:47:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:47:58 smithi171 ceph-mon[37789]: pgmap v4536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:00 smithi044 ceph-mon[32404]: pgmap v4537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:00 smithi116 ceph-mon[37097]: pgmap v4537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:00 smithi171 ceph-mon[37789]: pgmap v4537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7625.773841 secs 2024-07-26T10:48:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7625.773841 secs 2024-07-26T10:48:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7625.773841 secs 2024-07-26T10:48:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:02 smithi044 ceph-mon[32404]: pgmap v4538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:02 smithi116 ceph-mon[37097]: pgmap v4538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:02 smithi171 ceph-mon[37789]: pgmap v4538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:04 smithi044 ceph-mon[32404]: pgmap v4539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:04 smithi116 ceph-mon[37097]: pgmap v4539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:04 smithi171 ceph-mon[37789]: pgmap v4539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7630.773979 secs 2024-07-26T10:48:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:06 smithi044 ceph-mon[32404]: pgmap v4540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7630.773979 secs 2024-07-26T10:48:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:06 smithi116 ceph-mon[37097]: pgmap v4540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7630.773979 secs 2024-07-26T10:48:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:06 smithi171 ceph-mon[37789]: pgmap v4540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:08 smithi116 ceph-mon[37097]: pgmap v4541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:48:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:08 smithi171 ceph-mon[37789]: pgmap v4541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:48:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:08 smithi044 ceph-mon[32404]: pgmap v4541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:48:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:10 smithi171 ceph-mon[37789]: pgmap v4542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:10 smithi044 ceph-mon[32404]: pgmap v4542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:10 smithi116 ceph-mon[37097]: pgmap v4542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:48:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7635.774108 secs 2024-07-26T10:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7635.774108 secs 2024-07-26T10:48:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7635.774108 secs 2024-07-26T10:48:12.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:12.823+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:48:12.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:12.823+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:48:12.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:12.823+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:12 smithi171 ceph-mon[37789]: pgmap v4543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:12 smithi044 ceph-mon[32404]: pgmap v4543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:12 smithi116 ceph-mon[37097]: pgmap v4543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:14 smithi171 ceph-mon[37789]: pgmap v4544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:48:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:14 smithi044 ceph-mon[32404]: pgmap v4544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:48:15.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:15.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:14 smithi116 ceph-mon[37097]: pgmap v4544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:48:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7640.774252 secs 2024-07-26T10:48:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7640.774252 secs 2024-07-26T10:48:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:16.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7640.774252 secs 2024-07-26T10:48:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:48:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:16 smithi171 ceph-mon[37789]: pgmap v4545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:16 smithi171 ceph-mon[37789]: pgmap v4546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:48:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:16 smithi044 ceph-mon[32404]: pgmap v4545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:17.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:16 smithi044 ceph-mon[32404]: pgmap v4546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:48:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:16 smithi116 ceph-mon[37097]: pgmap v4545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:16 smithi116 ceph-mon[37097]: pgmap v4546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:48:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:18 smithi171 ceph-mon[37789]: pgmap v4547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:18 smithi044 ceph-mon[32404]: pgmap v4547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:18 smithi116 ceph-mon[37097]: pgmap v4547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:20 smithi171 ceph-mon[37789]: pgmap v4548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7645.774378 secs 2024-07-26T10:48:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:20 smithi044 ceph-mon[32404]: pgmap v4548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:21.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7645.774378 secs 2024-07-26T10:48:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:20 smithi116 ceph-mon[37097]: pgmap v4548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7645.774378 secs 2024-07-26T10:48:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:22 smithi171 ceph-mon[37789]: pgmap v4549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:22 smithi044 ceph-mon[32404]: pgmap v4549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:22 smithi116 ceph-mon[37097]: pgmap v4549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:24 smithi044 ceph-mon[32404]: pgmap v4550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:25.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:24 smithi116 ceph-mon[37097]: pgmap v4550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:24 smithi171 ceph-mon[37789]: pgmap v4550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:48:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7650.774526 secs 2024-07-26T10:48:26.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7650.774526 secs 2024-07-26T10:48:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7650.774526 secs 2024-07-26T10:48:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:26 smithi044 ceph-mon[32404]: pgmap v4551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:48:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:26 smithi116 ceph-mon[37097]: pgmap v4551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:48:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:26 smithi171 ceph-mon[37789]: pgmap v4551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:48:27.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:27.833+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:48:27.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:27.833+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:48:27.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:27.833+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:28 smithi044 ceph-mon[32404]: pgmap v4552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:28 smithi116 ceph-mon[37097]: pgmap v4552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:28 smithi171 ceph-mon[37789]: pgmap v4552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:30 smithi044 ceph-mon[32404]: pgmap v4553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7655.774692 secs 2024-07-26T10:48:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:30 smithi116 ceph-mon[37097]: pgmap v4553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7655.774692 secs 2024-07-26T10:48:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:30 smithi171 ceph-mon[37789]: pgmap v4553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:31.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7655.774692 secs 2024-07-26T10:48:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:32 smithi044 ceph-mon[32404]: pgmap v4554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:32 smithi116 ceph-mon[37097]: pgmap v4554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:32 smithi171 ceph-mon[37789]: pgmap v4554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:35.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:34 smithi171 ceph-mon[37789]: pgmap v4555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:34 smithi044 ceph-mon[32404]: pgmap v4555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:34 smithi116 ceph-mon[37097]: pgmap v4555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7660.774865 secs 2024-07-26T10:48:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7660.774865 secs 2024-07-26T10:48:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7660.774865 secs 2024-07-26T10:48:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:36 smithi044 ceph-mon[32404]: pgmap v4556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:36 smithi116 ceph-mon[37097]: pgmap v4556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:36 smithi171 ceph-mon[37789]: pgmap v4556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:38 smithi044 ceph-mon[32404]: pgmap v4557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:38 smithi116 ceph-mon[37097]: pgmap v4557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:38 smithi171 ceph-mon[37789]: pgmap v4557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:40 smithi044 ceph-mon[32404]: pgmap v4558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7665.775032 secs 2024-07-26T10:48:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:40 smithi116 ceph-mon[37097]: pgmap v4558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7665.775032 secs 2024-07-26T10:48:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:40 smithi171 ceph-mon[37789]: pgmap v4558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7665.775032 secs 2024-07-26T10:48:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:42.837+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:48:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:42.837+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:48:42.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:42.837+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:42 smithi044 ceph-mon[32404]: pgmap v4559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:42 smithi116 ceph-mon[37097]: pgmap v4559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:42 smithi171 ceph-mon[37789]: pgmap v4559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:44 smithi044 ceph-mon[32404]: pgmap v4560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:44 smithi116 ceph-mon[37097]: pgmap v4560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:44 smithi171 ceph-mon[37789]: pgmap v4560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7670.775167 secs 2024-07-26T10:48:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7670.775167 secs 2024-07-26T10:48:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7670.775167 secs 2024-07-26T10:48:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:46 smithi044 ceph-mon[32404]: pgmap v4561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:46 smithi116 ceph-mon[37097]: pgmap v4561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:46 smithi171 ceph-mon[37789]: pgmap v4561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:48 smithi044 ceph-mon[32404]: pgmap v4562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:48 smithi116 ceph-mon[37097]: pgmap v4562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:48 smithi171 ceph-mon[37789]: pgmap v4562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:50 smithi044 ceph-mon[32404]: pgmap v4563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7675.775311 secs 2024-07-26T10:48:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:50 smithi116 ceph-mon[37097]: pgmap v4563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7675.775311 secs 2024-07-26T10:48:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:50 smithi171 ceph-mon[37789]: pgmap v4563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:48:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7675.775311 secs 2024-07-26T10:48:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:52 smithi044 ceph-mon[32404]: pgmap v4564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:52 smithi116 ceph-mon[37097]: pgmap v4564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:52 smithi171 ceph-mon[37789]: pgmap v4564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:48:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:54 smithi044 ceph-mon[32404]: pgmap v4565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:54 smithi116 ceph-mon[37097]: pgmap v4565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:54 smithi171 ceph-mon[37789]: pgmap v4565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:48:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7680.775478 secs 2024-07-26T10:48:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7680.775478 secs 2024-07-26T10:48:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7680.775478 secs 2024-07-26T10:48:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:56 smithi044 ceph-mon[32404]: pgmap v4566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:56 smithi116 ceph-mon[37097]: pgmap v4566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:56 smithi171 ceph-mon[37789]: pgmap v4566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:48:57.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:57.839+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:48:57.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:57.839+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:48:57.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:48:57.839+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:48:58 smithi044 ceph-mon[32404]: pgmap v4567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:48:58 smithi116 ceph-mon[37097]: pgmap v4567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:48:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:48:58 smithi171 ceph-mon[37789]: pgmap v4567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:00 smithi044 ceph-mon[32404]: pgmap v4568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7685.775680 secs 2024-07-26T10:49:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:00 smithi116 ceph-mon[37097]: pgmap v4568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7685.775680 secs 2024-07-26T10:49:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:00 smithi171 ceph-mon[37789]: pgmap v4568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7685.775680 secs 2024-07-26T10:49:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:02 smithi044 ceph-mon[32404]: pgmap v4569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:02 smithi116 ceph-mon[37097]: pgmap v4569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:02 smithi171 ceph-mon[37789]: pgmap v4569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:04 smithi044 ceph-mon[32404]: pgmap v4570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:04 smithi116 ceph-mon[37097]: pgmap v4570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:04 smithi171 ceph-mon[37789]: pgmap v4570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7690.775826 secs 2024-07-26T10:49:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7690.775826 secs 2024-07-26T10:49:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7690.775826 secs 2024-07-26T10:49:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:07 smithi044 ceph-mon[32404]: pgmap v4571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:07 smithi116 ceph-mon[37097]: pgmap v4571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:07 smithi171 ceph-mon[37789]: pgmap v4571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:09.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:09 smithi044 ceph-mon[32404]: pgmap v4572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:09 smithi116 ceph-mon[37097]: pgmap v4572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:09 smithi171 ceph-mon[37789]: pgmap v4572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:11 smithi116 ceph-mon[37097]: pgmap v4573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7695.775915 secs 2024-07-26T10:49:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:11 smithi171 ceph-mon[37789]: pgmap v4573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7695.775915 secs 2024-07-26T10:49:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:11 smithi044 ceph-mon[32404]: pgmap v4573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7695.775915 secs 2024-07-26T10:49:12.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:12.843+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:49:12.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:12.843+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:12.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:12.843+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:49:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:13 smithi044 ceph-mon[32404]: pgmap v4574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:13 smithi171 ceph-mon[37789]: pgmap v4574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:13 smithi116 ceph-mon[37097]: pgmap v4574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:15 smithi171 ceph-mon[37789]: pgmap v4575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:15 smithi044 ceph-mon[32404]: pgmap v4575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:15 smithi116 ceph-mon[37097]: pgmap v4575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7700.776091 secs 2024-07-26T10:49:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7700.776091 secs 2024-07-26T10:49:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7700.776091 secs 2024-07-26T10:49:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:17 smithi171 ceph-mon[37789]: pgmap v4576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:17 smithi044 ceph-mon[32404]: pgmap v4576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:17 smithi116 ceph-mon[37097]: pgmap v4576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:18 smithi044 ceph-mon[32404]: pgmap v4577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:18 smithi116 ceph-mon[37097]: pgmap v4577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:18 smithi171 ceph-mon[37789]: pgmap v4577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:19.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:21 smithi171 ceph-mon[37789]: pgmap v4578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:21.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7705.776191 secs 2024-07-26T10:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:21 smithi044 ceph-mon[32404]: pgmap v4578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7705.776191 secs 2024-07-26T10:49:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:21 smithi116 ceph-mon[37097]: pgmap v4578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:49:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7705.776191 secs 2024-07-26T10:49:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:22 smithi171 ceph-mon[37789]: pgmap v4579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:22 smithi171 ceph-mon[37789]: pgmap v4580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:49:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:22 smithi044 ceph-mon[32404]: pgmap v4579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:22 smithi044 ceph-mon[32404]: pgmap v4580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:49:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:22 smithi116 ceph-mon[37097]: pgmap v4579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:22 smithi116 ceph-mon[37097]: pgmap v4580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:49:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:24 smithi171 ceph-mon[37789]: pgmap v4581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:49:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:24 smithi044 ceph-mon[32404]: pgmap v4581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:49:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:24 smithi116 ceph-mon[37097]: pgmap v4581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T10:49:26.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:26 smithi116 ceph-mon[37097]: pgmap v4582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:49:26.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7710.776335 secs 2024-07-26T10:49:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:26 smithi171 ceph-mon[37789]: pgmap v4582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:49:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7710.776335 secs 2024-07-26T10:49:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:26 smithi044 ceph-mon[32404]: pgmap v4582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:49:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7710.776335 secs 2024-07-26T10:49:27.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:27.853+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:49:27.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:27.853+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:49:27.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:27.853+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:28 smithi171 ceph-mon[37789]: pgmap v4583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:49:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:28 smithi044 ceph-mon[32404]: pgmap v4583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:49:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:28 smithi116 ceph-mon[37097]: pgmap v4583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:49:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:29 smithi044 ceph-mon[32404]: pgmap v4584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:49:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:29 smithi116 ceph-mon[37097]: pgmap v4584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:49:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:29 smithi171 ceph-mon[37789]: pgmap v4584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:49:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:31 smithi044 ceph-mon[32404]: pgmap v4585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:49:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7715.776475 secs 2024-07-26T10:49:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:31 smithi116 ceph-mon[37097]: pgmap v4585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:49:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7715.776475 secs 2024-07-26T10:49:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:31 smithi171 ceph-mon[37789]: pgmap v4585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:49:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7715.776475 secs 2024-07-26T10:49:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:33 smithi044 ceph-mon[32404]: pgmap v4586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:33 smithi116 ceph-mon[37097]: pgmap v4586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:33 smithi171 ceph-mon[37789]: pgmap v4586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:35.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:35 smithi044 ceph-mon[32404]: pgmap v4587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:35 smithi116 ceph-mon[37097]: pgmap v4587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:35 smithi171 ceph-mon[37789]: pgmap v4587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7720.776673 secs 2024-07-26T10:49:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7720.776673 secs 2024-07-26T10:49:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7720.776673 secs 2024-07-26T10:49:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:37 smithi044 ceph-mon[32404]: pgmap v4588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:37 smithi116 ceph-mon[37097]: pgmap v4588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:37 smithi171 ceph-mon[37789]: pgmap v4588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:39 smithi044 ceph-mon[32404]: pgmap v4589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:39 smithi116 ceph-mon[37097]: pgmap v4589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:39 smithi171 ceph-mon[37789]: pgmap v4589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:41 smithi044 ceph-mon[32404]: pgmap v4590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7725.776836 secs 2024-07-26T10:49:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:41 smithi116 ceph-mon[37097]: pgmap v4590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7725.776836 secs 2024-07-26T10:49:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:41 smithi171 ceph-mon[37789]: pgmap v4590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7725.776836 secs 2024-07-26T10:49:42.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:42.857+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:49:42.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:42.857+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:42.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:42.857+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:49:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:43 smithi044 ceph-mon[32404]: pgmap v4591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:43 smithi116 ceph-mon[37097]: pgmap v4591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:43 smithi171 ceph-mon[37789]: pgmap v4591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:45 smithi044 ceph-mon[32404]: pgmap v4592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:45 smithi116 ceph-mon[37097]: pgmap v4592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:45 smithi171 ceph-mon[37789]: pgmap v4592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7730.776979 secs 2024-07-26T10:49:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7730.776979 secs 2024-07-26T10:49:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7730.776979 secs 2024-07-26T10:49:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:47 smithi044 ceph-mon[32404]: pgmap v4593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:47 smithi116 ceph-mon[37097]: pgmap v4593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:47 smithi171 ceph-mon[37789]: pgmap v4593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:49 smithi044 ceph-mon[32404]: pgmap v4594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:49 smithi116 ceph-mon[37097]: pgmap v4594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:49 smithi171 ceph-mon[37789]: pgmap v4594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:49:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:51 smithi044 ceph-mon[32404]: pgmap v4595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7735.777122 secs 2024-07-26T10:49:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:51 smithi116 ceph-mon[37097]: pgmap v4595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7735.777122 secs 2024-07-26T10:49:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:51 smithi171 ceph-mon[37789]: pgmap v4595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:49:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7735.777122 secs 2024-07-26T10:49:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:53 smithi044 ceph-mon[32404]: pgmap v4596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:53 smithi116 ceph-mon[37097]: pgmap v4596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:53 smithi171 ceph-mon[37789]: pgmap v4596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:49:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:55 smithi044 ceph-mon[32404]: pgmap v4597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:55 smithi116 ceph-mon[37097]: pgmap v4597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:55 smithi171 ceph-mon[37789]: pgmap v4597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:49:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7740.777286 secs 2024-07-26T10:49:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7740.777286 secs 2024-07-26T10:49:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7740.777286 secs 2024-07-26T10:49:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:57 smithi171 ceph-mon[37789]: pgmap v4598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:57.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:57.866+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:49:57.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:57.866+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:57.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:49:57.866+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:49:57.869 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:57 smithi044 ceph-mon[32404]: pgmap v4598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:57 smithi116 ceph-mon[37097]: pgmap v4598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:49:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:49:59 smithi171 ceph-mon[37789]: pgmap v4599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:49:59 smithi044 ceph-mon[32404]: pgmap v4599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:49:59 smithi116 ceph-mon[37097]: pgmap v4599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T10:50:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:01 smithi171 ceph-mon[37789]: pgmap v4600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7745.777457 secs 2024-07-26T10:50:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:01 smithi044 ceph-mon[32404]: pgmap v4600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7745.777457 secs 2024-07-26T10:50:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:01 smithi116 ceph-mon[37097]: pgmap v4600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7745.777457 secs 2024-07-26T10:50:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:03 smithi171 ceph-mon[37789]: pgmap v4601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:03 smithi044 ceph-mon[32404]: pgmap v4601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:03 smithi116 ceph-mon[37097]: pgmap v4601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:05.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:05 smithi044 ceph-mon[32404]: pgmap v4602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:05 smithi171 ceph-mon[37789]: pgmap v4602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:05 smithi116 ceph-mon[37097]: pgmap v4602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7750.777656 secs 2024-07-26T10:50:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7750.777656 secs 2024-07-26T10:50:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7750.777656 secs 2024-07-26T10:50:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:07 smithi171 ceph-mon[37789]: pgmap v4603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:07 smithi044 ceph-mon[32404]: pgmap v4603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:07 smithi116 ceph-mon[37097]: pgmap v4603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:09 smithi171 ceph-mon[37789]: pgmap v4604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:09 smithi044 ceph-mon[32404]: pgmap v4604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:09 smithi116 ceph-mon[37097]: pgmap v4604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:11 smithi044 ceph-mon[32404]: pgmap v4605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7755.777800 secs 2024-07-26T10:50:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:11 smithi116 ceph-mon[37097]: pgmap v4605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7755.777800 secs 2024-07-26T10:50:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:11 smithi171 ceph-mon[37789]: pgmap v4605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7755.777800 secs 2024-07-26T10:50:12.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:12.871+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:50:12.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:12.871+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:12.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:12.871+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:50:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:13 smithi044 ceph-mon[32404]: pgmap v4606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:13 smithi116 ceph-mon[37097]: pgmap v4606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:13 smithi171 ceph-mon[37789]: pgmap v4606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:15 smithi044 ceph-mon[32404]: pgmap v4607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:15 smithi116 ceph-mon[37097]: pgmap v4607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:16.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:16.056+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:50:16.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:16.056+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:50:16.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:16.057+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:50:16.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:16.057+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:50:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:15 smithi171 ceph-mon[37789]: pgmap v4607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7760.777947 secs 2024-07-26T10:50:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7760.777947 secs 2024-07-26T10:50:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7760.777947 secs 2024-07-26T10:50:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:17 smithi044 ceph-mon[32404]: pgmap v4608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:17 smithi116 ceph-mon[37097]: pgmap v4608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:17 smithi171 ceph-mon[37789]: pgmap v4608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:19 smithi044 ceph-mon[32404]: pgmap v4609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:20.044 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:19 smithi116 ceph-mon[37097]: pgmap v4609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:19 smithi171 ceph-mon[37789]: pgmap v4609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:20 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:20 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:20 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:21 smithi044 ceph-mon[32404]: pgmap v4610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7765.778111 secs 2024-07-26T10:50:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:21 smithi116 ceph-mon[37097]: pgmap v4610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7765.778111 secs 2024-07-26T10:50:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:21 smithi171 ceph-mon[37789]: pgmap v4610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7765.778111 secs 2024-07-26T10:50:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:22 smithi044 ceph-mon[32404]: pgmap v4611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:50:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:22 smithi044 ceph-mon[32404]: pgmap v4612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:50:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:50:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:22 smithi116 ceph-mon[37097]: pgmap v4611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:50:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:22 smithi116 ceph-mon[37097]: pgmap v4612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:50:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:50:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:22 smithi171 ceph-mon[37789]: pgmap v4611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:50:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:22 smithi171 ceph-mon[37789]: pgmap v4612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:50:23.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:50:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:24 smithi116 ceph-mon[37097]: pgmap v4613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:50:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:24 smithi171 ceph-mon[37789]: pgmap v4613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:50:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:24 smithi044 ceph-mon[32404]: pgmap v4613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:50:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7770.778259 secs 2024-07-26T10:50:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:26 smithi171 ceph-mon[37789]: pgmap v4614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:50:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7770.778259 secs 2024-07-26T10:50:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:26 smithi044 ceph-mon[32404]: pgmap v4614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:50:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7770.778259 secs 2024-07-26T10:50:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:26 smithi116 ceph-mon[37097]: pgmap v4614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:50:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:27.880+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:50:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:27.880+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:27.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:27.880+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:50:29.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:28 smithi171 ceph-mon[37789]: pgmap v4615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:50:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:28 smithi044 ceph-mon[32404]: pgmap v4615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:50:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:28 smithi116 ceph-mon[37097]: pgmap v4615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:50:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:30 smithi044 ceph-mon[32404]: pgmap v4616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:50:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:30 smithi171 ceph-mon[37789]: pgmap v4616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:50:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:30 smithi116 ceph-mon[37097]: pgmap v4616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:50:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7775.778429 secs 2024-07-26T10:50:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7775.778429 secs 2024-07-26T10:50:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7775.778429 secs 2024-07-26T10:50:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:32 smithi171 ceph-mon[37789]: pgmap v4617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:50:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:32 smithi044 ceph-mon[32404]: pgmap v4617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:50:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:32 smithi116 ceph-mon[37097]: pgmap v4617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:50:35.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:34 smithi044 ceph-mon[32404]: pgmap v4618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:34 smithi171 ceph-mon[37789]: pgmap v4618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:34 smithi116 ceph-mon[37097]: pgmap v4618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7780.778577 secs 2024-07-26T10:50:37.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:36 smithi171 ceph-mon[37789]: pgmap v4619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:37.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7780.778577 secs 2024-07-26T10:50:37.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:36 smithi044 ceph-mon[32404]: pgmap v4619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7780.778577 secs 2024-07-26T10:50:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:36 smithi116 ceph-mon[37097]: pgmap v4619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:39.078 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:38 smithi116 ceph-mon[37097]: pgmap v4620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:38 smithi171 ceph-mon[37789]: pgmap v4620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:38 smithi044 ceph-mon[32404]: pgmap v4620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:40 smithi171 ceph-mon[37789]: pgmap v4621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:41.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7785.778728 secs 2024-07-26T10:50:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:40 smithi044 ceph-mon[32404]: pgmap v4621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7785.778728 secs 2024-07-26T10:50:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:40 smithi116 ceph-mon[37097]: pgmap v4621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7785.778728 secs 2024-07-26T10:50:42.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:42.882+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:42.882+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:50:42.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:42.882+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:42 smithi171 ceph-mon[37789]: pgmap v4622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:42 smithi044 ceph-mon[32404]: pgmap v4622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:42 smithi116 ceph-mon[37097]: pgmap v4622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:44 smithi044 ceph-mon[32404]: pgmap v4623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:44 smithi116 ceph-mon[37097]: pgmap v4623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:44 smithi171 ceph-mon[37789]: pgmap v4623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:46.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7790.778879 secs 2024-07-26T10:50:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7790.778879 secs 2024-07-26T10:50:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7790.778879 secs 2024-07-26T10:50:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:46 smithi044 ceph-mon[32404]: pgmap v4624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:46 smithi116 ceph-mon[37097]: pgmap v4624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:46 smithi171 ceph-mon[37789]: pgmap v4624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:49.162 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:48 smithi171 ceph-mon[37789]: pgmap v4625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:48 smithi044 ceph-mon[32404]: pgmap v4625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:48 smithi116 ceph-mon[37097]: pgmap v4625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:50 smithi044 ceph-mon[32404]: pgmap v4626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7795.778979 secs 2024-07-26T10:50:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:50 smithi116 ceph-mon[37097]: pgmap v4626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7795.778979 secs 2024-07-26T10:50:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:50 smithi171 ceph-mon[37789]: pgmap v4626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:50:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7795.778979 secs 2024-07-26T10:50:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:52 smithi044 ceph-mon[32404]: pgmap v4627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:52 smithi116 ceph-mon[37097]: pgmap v4627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:52 smithi171 ceph-mon[37789]: pgmap v4627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:50:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:54 smithi044 ceph-mon[32404]: pgmap v4628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:54 smithi116 ceph-mon[37097]: pgmap v4628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:54 smithi171 ceph-mon[37789]: pgmap v4628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:50:56.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7800.779136 secs 2024-07-26T10:50:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7800.779136 secs 2024-07-26T10:50:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7800.779136 secs 2024-07-26T10:50:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:56 smithi044 ceph-mon[32404]: pgmap v4629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:56 smithi116 ceph-mon[37097]: pgmap v4629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:56 smithi171 ceph-mon[37789]: pgmap v4629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:50:57.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:57.884+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:50:57.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:57.884+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:50:57.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:50:57.884+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:50:58 smithi044 ceph-mon[32404]: pgmap v4630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:50:58 smithi116 ceph-mon[37097]: pgmap v4630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:50:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:50:58 smithi171 ceph-mon[37789]: pgmap v4630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:00 smithi044 ceph-mon[32404]: pgmap v4631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7805.779301 secs 2024-07-26T10:51:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:00 smithi116 ceph-mon[37097]: pgmap v4631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7805.779301 secs 2024-07-26T10:51:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:00 smithi171 ceph-mon[37789]: pgmap v4631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7805.779301 secs 2024-07-26T10:51:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:02 smithi044 ceph-mon[32404]: pgmap v4632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:02 smithi116 ceph-mon[37097]: pgmap v4632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:02 smithi171 ceph-mon[37789]: pgmap v4632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:04 smithi044 ceph-mon[32404]: pgmap v4633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:04 smithi116 ceph-mon[37097]: pgmap v4633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:04 smithi171 ceph-mon[37789]: pgmap v4633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:06.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7810.779462 secs 2024-07-26T10:51:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7810.779462 secs 2024-07-26T10:51:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7810.779462 secs 2024-07-26T10:51:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:06 smithi044 ceph-mon[32404]: pgmap v4634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:06 smithi116 ceph-mon[37097]: pgmap v4634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:06 smithi171 ceph-mon[37789]: pgmap v4634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:08 smithi044 ceph-mon[32404]: pgmap v4635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:08 smithi116 ceph-mon[37097]: pgmap v4635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:08 smithi171 ceph-mon[37789]: pgmap v4635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:10 smithi044 ceph-mon[32404]: pgmap v4636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7815.779672 secs 2024-07-26T10:51:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:10 smithi116 ceph-mon[37097]: pgmap v4636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7815.779672 secs 2024-07-26T10:51:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:10 smithi171 ceph-mon[37789]: pgmap v4636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7815.779672 secs 2024-07-26T10:51:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:12.888+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:51:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:12.888+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:12.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:12.888+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:51:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:12 smithi044 ceph-mon[32404]: pgmap v4637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:12 smithi116 ceph-mon[37097]: pgmap v4637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:12 smithi171 ceph-mon[37789]: pgmap v4637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:14 smithi044 ceph-mon[32404]: pgmap v4638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:15.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:14 smithi116 ceph-mon[37097]: pgmap v4638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:14 smithi171 ceph-mon[37789]: pgmap v4638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7820.779810 secs 2024-07-26T10:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7820.779810 secs 2024-07-26T10:51:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7820.779810 secs 2024-07-26T10:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:16 smithi044 ceph-mon[32404]: pgmap v4639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:16 smithi116 ceph-mon[37097]: pgmap v4639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:16 smithi171 ceph-mon[37789]: pgmap v4639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:18 smithi044 ceph-mon[32404]: pgmap v4640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:18 smithi116 ceph-mon[37097]: pgmap v4640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:18 smithi171 ceph-mon[37789]: pgmap v4640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:20 smithi044 ceph-mon[32404]: pgmap v4641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7825.779971 secs 2024-07-26T10:51:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:20 smithi116 ceph-mon[37097]: pgmap v4641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7825.779971 secs 2024-07-26T10:51:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:20 smithi171 ceph-mon[37789]: pgmap v4641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:51:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7825.779971 secs 2024-07-26T10:51:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:22 smithi044 ceph-mon[32404]: pgmap v4642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:23.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:23.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:22 smithi116 ceph-mon[37097]: pgmap v4642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:22 smithi171 ceph-mon[37789]: pgmap v4642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:51:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:23 smithi044 ceph-mon[32404]: pgmap v4643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:51:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:23 smithi044 ceph-mon[32404]: pgmap v4644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:24.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:51:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:24 smithi116 ceph-mon[37097]: pgmap v4643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:51:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:24 smithi116 ceph-mon[37097]: pgmap v4644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:51:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:24 smithi171 ceph-mon[37789]: pgmap v4643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:51:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:24 smithi171 ceph-mon[37789]: pgmap v4644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:51:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:26 smithi044 ceph-mon[32404]: pgmap v4645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7830.780136 secs 2024-07-26T10:51:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:26 smithi116 ceph-mon[37097]: pgmap v4645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7830.780136 secs 2024-07-26T10:51:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:26 smithi171 ceph-mon[37789]: pgmap v4645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7830.780136 secs 2024-07-26T10:51:27.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:27.894+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:51:27.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:27.894+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:51:27.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:27.894+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:28 smithi044 ceph-mon[32404]: pgmap v4646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:28 smithi116 ceph-mon[37097]: pgmap v4646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:28 smithi171 ceph-mon[37789]: pgmap v4646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:30 smithi116 ceph-mon[37097]: pgmap v4647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:30 smithi171 ceph-mon[37789]: pgmap v4647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:30 smithi044 ceph-mon[32404]: pgmap v4647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:51:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7835.780272 secs 2024-07-26T10:51:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7835.780272 secs 2024-07-26T10:51:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7835.780272 secs 2024-07-26T10:51:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:32 smithi171 ceph-mon[37789]: pgmap v4648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:51:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:32 smithi044 ceph-mon[32404]: pgmap v4648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:51:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:32 smithi116 ceph-mon[37097]: pgmap v4648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:51:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:34 smithi171 ceph-mon[37789]: pgmap v4649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:51:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:34 smithi044 ceph-mon[32404]: pgmap v4649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:51:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:34 smithi116 ceph-mon[37097]: pgmap v4649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:51:36.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:36 smithi044 ceph-mon[32404]: pgmap v4650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:36.320 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7840.780379 secs 2024-07-26T10:51:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:36 smithi171 ceph-mon[37789]: pgmap v4650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7840.780379 secs 2024-07-26T10:51:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:36 smithi116 ceph-mon[37097]: pgmap v4650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7840.780379 secs 2024-07-26T10:51:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:38 smithi171 ceph-mon[37789]: pgmap v4651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:38 smithi044 ceph-mon[32404]: pgmap v4651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:38 smithi116 ceph-mon[37097]: pgmap v4651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:40 smithi171 ceph-mon[37789]: pgmap v4652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:40 smithi044 ceph-mon[32404]: pgmap v4652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:40 smithi116 ceph-mon[37097]: pgmap v4652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7845.780544 secs 2024-07-26T10:51:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7845.780544 secs 2024-07-26T10:51:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7845.780544 secs 2024-07-26T10:51:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:42 smithi171 ceph-mon[37789]: pgmap v4653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:42 smithi044 ceph-mon[32404]: pgmap v4653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:42 smithi116 ceph-mon[37097]: pgmap v4653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:42.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:42.898+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:51:42.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:42.898+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:51:42.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:42.898+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:44 smithi171 ceph-mon[37789]: pgmap v4654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:51:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:44 smithi044 ceph-mon[32404]: pgmap v4654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:51:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:44 smithi116 ceph-mon[37097]: pgmap v4654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:51:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:46 smithi171 ceph-mon[37789]: pgmap v4655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7850.780693 secs 2024-07-26T10:51:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:46 smithi044 ceph-mon[32404]: pgmap v4655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7850.780693 secs 2024-07-26T10:51:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:46 smithi116 ceph-mon[37097]: pgmap v4655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7850.780693 secs 2024-07-26T10:51:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:48 smithi044 ceph-mon[32404]: pgmap v4656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:48 smithi171 ceph-mon[37789]: pgmap v4656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:48.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:48 smithi116 ceph-mon[37097]: pgmap v4656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:50.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:50 smithi044 ceph-mon[32404]: pgmap v4657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:50 smithi116 ceph-mon[37097]: pgmap v4657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:50 smithi171 ceph-mon[37789]: pgmap v4657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7855.780860 secs 2024-07-26T10:51:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7855.780860 secs 2024-07-26T10:51:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7855.780860 secs 2024-07-26T10:51:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:52 smithi044 ceph-mon[32404]: pgmap v4658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:52 smithi116 ceph-mon[37097]: pgmap v4658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:52 smithi171 ceph-mon[37789]: pgmap v4658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:54 smithi044 ceph-mon[32404]: pgmap v4659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:51:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:54 smithi116 ceph-mon[37097]: pgmap v4659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:51:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:54 smithi171 ceph-mon[37789]: pgmap v4659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:51:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:56 smithi044 ceph-mon[32404]: pgmap v4660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7860.781023 secs 2024-07-26T10:51:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:56 smithi116 ceph-mon[37097]: pgmap v4660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7860.781023 secs 2024-07-26T10:51:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:56 smithi171 ceph-mon[37789]: pgmap v4660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:51:56.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7860.781023 secs 2024-07-26T10:51:57.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:57.906+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:51:57.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:57.906+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:57.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:51:57.906+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:51:58.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:51:58 smithi044 ceph-mon[32404]: pgmap v4661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:51:58 smithi116 ceph-mon[37097]: pgmap v4661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:51:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:51:58 smithi171 ceph-mon[37789]: pgmap v4661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:00 smithi044 ceph-mon[32404]: pgmap v4662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:00 smithi116 ceph-mon[37097]: pgmap v4662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:00 smithi171 ceph-mon[37789]: pgmap v4662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7865.781183 secs 2024-07-26T10:52:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7865.781183 secs 2024-07-26T10:52:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7865.781183 secs 2024-07-26T10:52:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:02 smithi044 ceph-mon[32404]: pgmap v4663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:02 smithi116 ceph-mon[37097]: pgmap v4663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:02 smithi171 ceph-mon[37789]: pgmap v4663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:04 smithi044 ceph-mon[32404]: pgmap v4664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:04 smithi116 ceph-mon[37097]: pgmap v4664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:04 smithi171 ceph-mon[37789]: pgmap v4664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:06 smithi044 ceph-mon[32404]: pgmap v4665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7870.781348 secs 2024-07-26T10:52:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:06 smithi116 ceph-mon[37097]: pgmap v4665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7870.781348 secs 2024-07-26T10:52:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:06 smithi171 ceph-mon[37789]: pgmap v4665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7870.781348 secs 2024-07-26T10:52:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:08 smithi044 ceph-mon[32404]: pgmap v4666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:08 smithi116 ceph-mon[37097]: pgmap v4666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:08 smithi171 ceph-mon[37789]: pgmap v4666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:10 smithi044 ceph-mon[32404]: pgmap v4667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:10 smithi116 ceph-mon[37097]: pgmap v4667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:10 smithi171 ceph-mon[37789]: pgmap v4667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7875.781492 secs 2024-07-26T10:52:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7875.781492 secs 2024-07-26T10:52:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7875.781492 secs 2024-07-26T10:52:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:12 smithi044 ceph-mon[32404]: pgmap v4668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:12 smithi116 ceph-mon[37097]: pgmap v4668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:12 smithi171 ceph-mon[37789]: pgmap v4668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:12.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:12.910+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:52:12.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:12.911+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:52:12.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:12.911+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:14 smithi044 ceph-mon[32404]: pgmap v4669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:14 smithi116 ceph-mon[37097]: pgmap v4669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:14.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:14 smithi171 ceph-mon[37789]: pgmap v4669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:16 smithi044 ceph-mon[32404]: pgmap v4670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7880.781675 secs 2024-07-26T10:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:16 smithi116 ceph-mon[37097]: pgmap v4670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7880.781675 secs 2024-07-26T10:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:16 smithi171 ceph-mon[37789]: pgmap v4670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7880.781675 secs 2024-07-26T10:52:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:18 smithi044 ceph-mon[32404]: pgmap v4671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:18 smithi116 ceph-mon[37097]: pgmap v4671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:18 smithi171 ceph-mon[37789]: pgmap v4671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:20 smithi044 ceph-mon[32404]: pgmap v4672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:20 smithi116 ceph-mon[37097]: pgmap v4672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:20 smithi171 ceph-mon[37789]: pgmap v4672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7885.781843 secs 2024-07-26T10:52:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7885.781843 secs 2024-07-26T10:52:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7885.781843 secs 2024-07-26T10:52:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:22 smithi044 ceph-mon[32404]: pgmap v4673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:22 smithi116 ceph-mon[37097]: pgmap v4673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:22 smithi171 ceph-mon[37789]: pgmap v4673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:24 smithi044 ceph-mon[32404]: pgmap v4674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:24 smithi116 ceph-mon[37097]: pgmap v4674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:24 smithi171 ceph-mon[37789]: pgmap v4674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:25 smithi044 ceph-mon[32404]: pgmap v4675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:52:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:25 smithi044 ceph-mon[32404]: pgmap v4676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:52:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:25 smithi116 ceph-mon[37097]: pgmap v4675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:52:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:25 smithi116 ceph-mon[37097]: pgmap v4676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:52:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:25.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:25.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:25 smithi171 ceph-mon[37789]: pgmap v4675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:52:25.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:25 smithi171 ceph-mon[37789]: pgmap v4676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:25.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:52:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7890.782011 secs 2024-07-26T10:52:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7890.782011 secs 2024-07-26T10:52:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7890.782011 secs 2024-07-26T10:52:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:27 smithi171 ceph-mon[37789]: pgmap v4677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:52:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:27 smithi044 ceph-mon[32404]: pgmap v4677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:52:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:27 smithi116 ceph-mon[37097]: pgmap v4677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:52:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:27.920+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:52:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:27.920+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:27.920+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:52:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:29 smithi171 ceph-mon[37789]: pgmap v4678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:29 smithi044 ceph-mon[32404]: pgmap v4678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:29 smithi116 ceph-mon[37097]: pgmap v4678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:31 smithi171 ceph-mon[37789]: pgmap v4679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7895.782167 secs 2024-07-26T10:52:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:31 smithi044 ceph-mon[32404]: pgmap v4679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7895.782167 secs 2024-07-26T10:52:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:31 smithi116 ceph-mon[37097]: pgmap v4679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:52:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7895.782167 secs 2024-07-26T10:52:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:33 smithi171 ceph-mon[37789]: pgmap v4680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:52:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:33 smithi044 ceph-mon[32404]: pgmap v4680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:52:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:33 smithi116 ceph-mon[37097]: pgmap v4680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:52:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:35 smithi171 ceph-mon[37789]: pgmap v4681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:52:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:35 smithi044 ceph-mon[32404]: pgmap v4681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:52:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:35 smithi116 ceph-mon[37097]: pgmap v4681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:52:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7900.782319 secs 2024-07-26T10:52:36.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7900.782319 secs 2024-07-26T10:52:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7900.782319 secs 2024-07-26T10:52:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:37 smithi171 ceph-mon[37789]: pgmap v4682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:37 smithi044 ceph-mon[32404]: pgmap v4682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:37 smithi116 ceph-mon[37097]: pgmap v4682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:39 smithi171 ceph-mon[37789]: pgmap v4683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:39 smithi044 ceph-mon[32404]: pgmap v4683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:39 smithi116 ceph-mon[37097]: pgmap v4683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:41 smithi171 ceph-mon[37789]: pgmap v4684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7905.782475 secs 2024-07-26T10:52:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:41 smithi044 ceph-mon[32404]: pgmap v4684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7905.782475 secs 2024-07-26T10:52:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:41 smithi116 ceph-mon[37097]: pgmap v4684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7905.782475 secs 2024-07-26T10:52:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:42.925+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:52:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:42.925+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:52:42.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:42.925+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:43 smithi171 ceph-mon[37789]: pgmap v4685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:43.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:43 smithi044 ceph-mon[32404]: pgmap v4685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:43 smithi116 ceph-mon[37097]: pgmap v4685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:45 smithi171 ceph-mon[37789]: pgmap v4686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:45 smithi044 ceph-mon[32404]: pgmap v4686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:45 smithi116 ceph-mon[37097]: pgmap v4686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7910.782682 secs 2024-07-26T10:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7910.782682 secs 2024-07-26T10:52:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7910.782682 secs 2024-07-26T10:52:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:47 smithi044 ceph-mon[32404]: pgmap v4687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:47 smithi116 ceph-mon[37097]: pgmap v4687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:47 smithi171 ceph-mon[37789]: pgmap v4687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:48 smithi044 ceph-mon[32404]: pgmap v4688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:48.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:48 smithi116 ceph-mon[37097]: pgmap v4688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:48 smithi171 ceph-mon[37789]: pgmap v4688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:51 smithi044 ceph-mon[32404]: pgmap v4689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7915.782839 secs 2024-07-26T10:52:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:51 smithi116 ceph-mon[37097]: pgmap v4689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7915.782839 secs 2024-07-26T10:52:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:51 smithi171 ceph-mon[37789]: pgmap v4689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:52:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7915.782839 secs 2024-07-26T10:52:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:53 smithi044 ceph-mon[32404]: pgmap v4690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:53 smithi116 ceph-mon[37097]: pgmap v4690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:53 smithi171 ceph-mon[37789]: pgmap v4690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:52:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:54 smithi044 ceph-mon[32404]: pgmap v4691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:54 smithi116 ceph-mon[37097]: pgmap v4691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:54 smithi171 ceph-mon[37789]: pgmap v4691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:52:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7920.782999 secs 2024-07-26T10:52:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7920.782999 secs 2024-07-26T10:52:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7920.782999 secs 2024-07-26T10:52:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:56 smithi044 ceph-mon[32404]: pgmap v4692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:52:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:56 smithi116 ceph-mon[37097]: pgmap v4692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:52:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:56 smithi171 ceph-mon[37789]: pgmap v4692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:52:57.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:57.928+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:52:57.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:57.928+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:52:57.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:52:57.928+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:52:59 smithi044 ceph-mon[32404]: pgmap v4693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:52:59 smithi116 ceph-mon[37097]: pgmap v4693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:52:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:52:59 smithi171 ceph-mon[37789]: pgmap v4693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:00 smithi044 ceph-mon[32404]: pgmap v4694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:00 smithi116 ceph-mon[37097]: pgmap v4694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:00 smithi171 ceph-mon[37789]: pgmap v4694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7925.783171 secs 2024-07-26T10:53:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7925.783171 secs 2024-07-26T10:53:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7925.783171 secs 2024-07-26T10:53:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:02 smithi044 ceph-mon[32404]: pgmap v4695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:02 smithi116 ceph-mon[37097]: pgmap v4695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:02 smithi171 ceph-mon[37789]: pgmap v4695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:05 smithi044 ceph-mon[32404]: pgmap v4696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:05 smithi116 ceph-mon[37097]: pgmap v4696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:05 smithi171 ceph-mon[37789]: pgmap v4696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:06.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7930.783343 secs 2024-07-26T10:53:06.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:06 smithi171 ceph-mon[37789]: pgmap v4697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7930.783343 secs 2024-07-26T10:53:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:06 smithi044 ceph-mon[32404]: pgmap v4697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7930.783343 secs 2024-07-26T10:53:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:06 smithi116 ceph-mon[37097]: pgmap v4697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:09 smithi044 ceph-mon[32404]: pgmap v4698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:09 smithi116 ceph-mon[37097]: pgmap v4698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:09 smithi171 ceph-mon[37789]: pgmap v4698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:10 smithi044 ceph-mon[32404]: pgmap v4699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:10 smithi116 ceph-mon[37097]: pgmap v4699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:10 smithi171 ceph-mon[37789]: pgmap v4699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7935.783461 secs 2024-07-26T10:53:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7935.783461 secs 2024-07-26T10:53:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7935.783461 secs 2024-07-26T10:53:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:12 smithi044 ceph-mon[32404]: pgmap v4700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:12 smithi116 ceph-mon[37097]: pgmap v4700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:12 smithi171 ceph-mon[37789]: pgmap v4700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:12.933+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:53:12.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:12.933+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:12.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:12.933+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:53:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:14 smithi044 ceph-mon[32404]: pgmap v4701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:14 smithi116 ceph-mon[37097]: pgmap v4701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:14 smithi171 ceph-mon[37789]: pgmap v4701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.088+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:53:16.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.088+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:53:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.089+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T10:53:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.089+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T10:53:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.090+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:53:16.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.090+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:53:16.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.091+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:53:16.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:16.091+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:53:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7940.783606 secs 2024-07-26T10:53:16.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7940.783606 secs 2024-07-26T10:53:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7940.783606 secs 2024-07-26T10:53:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:16 smithi044 ceph-mon[32404]: pgmap v4702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:16 smithi116 ceph-mon[37097]: pgmap v4702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:16 smithi171 ceph-mon[37789]: pgmap v4702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:18 smithi044 ceph-mon[32404]: pgmap v4703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:18 smithi116 ceph-mon[37097]: pgmap v4703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:18 smithi171 ceph-mon[37789]: pgmap v4703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:53:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:20 smithi044 ceph-mon[32404]: pgmap v4704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:20 smithi116 ceph-mon[37097]: pgmap v4704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:20 smithi171 ceph-mon[37789]: pgmap v4704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:21.698 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7945.783773 secs 2024-07-26T10:53:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7945.783773 secs 2024-07-26T10:53:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7945.783773 secs 2024-07-26T10:53:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:22 smithi044 ceph-mon[32404]: pgmap v4705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:22 smithi116 ceph-mon[37097]: pgmap v4705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:22 smithi171 ceph-mon[37789]: pgmap v4705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:24 smithi044 ceph-mon[32404]: pgmap v4706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:53:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:24 smithi116 ceph-mon[37097]: pgmap v4706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:53:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:24 smithi171 ceph-mon[37789]: pgmap v4706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:53:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7950.783862 secs 2024-07-26T10:53:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:26.675 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:26.675 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:26 smithi116 ceph-mon[37097]: pgmap v4707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:53:26.675 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:26 smithi116 ceph-mon[37097]: pgmap v4708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:26.675 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:53:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7950.783862 secs 2024-07-26T10:53:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:26 smithi044 ceph-mon[32404]: pgmap v4707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:53:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:26 smithi044 ceph-mon[32404]: pgmap v4708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:53:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7950.783862 secs 2024-07-26T10:53:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:26 smithi171 ceph-mon[37789]: pgmap v4707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T10:53:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:26 smithi171 ceph-mon[37789]: pgmap v4708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:26.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:53:27.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:27.943+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:53:27.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:27.943+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:53:27.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:27.943+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:28 smithi044 ceph-mon[32404]: pgmap v4709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T10:53:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:28 smithi116 ceph-mon[37097]: pgmap v4709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T10:53:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:28 smithi171 ceph-mon[37789]: pgmap v4709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T10:53:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:30 smithi044 ceph-mon[32404]: pgmap v4710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:53:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:30 smithi116 ceph-mon[37097]: pgmap v4710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:53:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:30 smithi171 ceph-mon[37789]: pgmap v4710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:53:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7955.783959 secs 2024-07-26T10:53:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7955.783959 secs 2024-07-26T10:53:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7955.783959 secs 2024-07-26T10:53:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:32 smithi044 ceph-mon[32404]: pgmap v4711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:53:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:32 smithi116 ceph-mon[37097]: pgmap v4711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:53:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:32 smithi171 ceph-mon[37789]: pgmap v4711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:53:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:34 smithi044 ceph-mon[32404]: pgmap v4712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:34 smithi116 ceph-mon[37097]: pgmap v4712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:34 smithi171 ceph-mon[37789]: pgmap v4712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:53:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7960.784103 secs 2024-07-26T10:53:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:36 smithi044 ceph-mon[32404]: pgmap v4713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:53:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7960.784103 secs 2024-07-26T10:53:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:36 smithi116 ceph-mon[37097]: pgmap v4713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:53:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7960.784103 secs 2024-07-26T10:53:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:36 smithi171 ceph-mon[37789]: pgmap v4713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:53:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:38 smithi044 ceph-mon[32404]: pgmap v4714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:53:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:38 smithi116 ceph-mon[37097]: pgmap v4714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:53:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:38 smithi171 ceph-mon[37789]: pgmap v4714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:53:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:40 smithi044 ceph-mon[32404]: pgmap v4715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:53:41.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7965.784271 secs 2024-07-26T10:53:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:40 smithi116 ceph-mon[37097]: pgmap v4715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:53:41.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7965.784271 secs 2024-07-26T10:53:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:40 smithi171 ceph-mon[37789]: pgmap v4715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:53:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7965.784271 secs 2024-07-26T10:53:42.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:42.948+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:53:42.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:42.948+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:53:42.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:42.948+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:42 smithi044 ceph-mon[32404]: pgmap v4716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:43.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:42 smithi116 ceph-mon[37097]: pgmap v4716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:42 smithi171 ceph-mon[37789]: pgmap v4716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:44 smithi044 ceph-mon[32404]: pgmap v4717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:53:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:44 smithi116 ceph-mon[37097]: pgmap v4717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:53:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:44 smithi171 ceph-mon[37789]: pgmap v4717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:53:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7970.784421 secs 2024-07-26T10:53:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7970.784421 secs 2024-07-26T10:53:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7970.784421 secs 2024-07-26T10:53:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:46 smithi044 ceph-mon[32404]: pgmap v4718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:46 smithi116 ceph-mon[37097]: pgmap v4718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:46 smithi171 ceph-mon[37789]: pgmap v4718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:49.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:48 smithi044 ceph-mon[32404]: pgmap v4719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:48 smithi116 ceph-mon[37097]: pgmap v4719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:48 smithi171 ceph-mon[37789]: pgmap v4719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:50 smithi044 ceph-mon[32404]: pgmap v4720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7975.784588 secs 2024-07-26T10:53:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:50 smithi116 ceph-mon[37097]: pgmap v4720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7975.784588 secs 2024-07-26T10:53:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:50 smithi171 ceph-mon[37789]: pgmap v4720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:53:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7975.784588 secs 2024-07-26T10:53:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:52 smithi044 ceph-mon[32404]: pgmap v4721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:52 smithi116 ceph-mon[37097]: pgmap v4721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:52 smithi171 ceph-mon[37789]: pgmap v4721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:53:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:54 smithi044 ceph-mon[32404]: pgmap v4722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:53:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:54 smithi116 ceph-mon[37097]: pgmap v4722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:53:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:54 smithi171 ceph-mon[37789]: pgmap v4722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:53:56.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7980.784749 secs 2024-07-26T10:53:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7980.784749 secs 2024-07-26T10:53:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7980.784749 secs 2024-07-26T10:53:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:56 smithi044 ceph-mon[32404]: pgmap v4723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:56 smithi116 ceph-mon[37097]: pgmap v4723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:56 smithi171 ceph-mon[37789]: pgmap v4723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:53:57.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:57.958+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:53:57.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:57.958+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:53:57.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:53:57.958+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:53:58 smithi044 ceph-mon[32404]: pgmap v4724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:53:58 smithi116 ceph-mon[37097]: pgmap v4724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:53:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:53:58 smithi171 ceph-mon[37789]: pgmap v4724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:00 smithi044 ceph-mon[32404]: pgmap v4725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7985.784913 secs 2024-07-26T10:54:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:00 smithi116 ceph-mon[37097]: pgmap v4725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7985.784913 secs 2024-07-26T10:54:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:00 smithi171 ceph-mon[37789]: pgmap v4725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7985.784913 secs 2024-07-26T10:54:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:02 smithi044 ceph-mon[32404]: pgmap v4726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:03.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:02 smithi116 ceph-mon[37097]: pgmap v4726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:02 smithi171 ceph-mon[37789]: pgmap v4726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:04 smithi044 ceph-mon[32404]: pgmap v4727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:04 smithi116 ceph-mon[37097]: pgmap v4727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:04 smithi171 ceph-mon[37789]: pgmap v4727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:06.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7990.785077 secs 2024-07-26T10:54:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7990.785077 secs 2024-07-26T10:54:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7990.785077 secs 2024-07-26T10:54:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:06 smithi044 ceph-mon[32404]: pgmap v4728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:06 smithi116 ceph-mon[37097]: pgmap v4728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:06 smithi171 ceph-mon[37789]: pgmap v4728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:08 smithi044 ceph-mon[32404]: pgmap v4729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:08 smithi116 ceph-mon[37097]: pgmap v4729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:08 smithi171 ceph-mon[37789]: pgmap v4729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:10 smithi044 ceph-mon[32404]: pgmap v4730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 7995.785225 secs 2024-07-26T10:54:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:10 smithi116 ceph-mon[37097]: pgmap v4730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 7995.785225 secs 2024-07-26T10:54:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:10 smithi171 ceph-mon[37789]: pgmap v4730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 7995.785225 secs 2024-07-26T10:54:12.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:12.958+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:54:12.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:12.958+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:54:12.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:12.958+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:12 smithi044 ceph-mon[32404]: pgmap v4731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:12 smithi116 ceph-mon[37097]: pgmap v4731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:12 smithi171 ceph-mon[37789]: pgmap v4731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:14 smithi044 ceph-mon[32404]: pgmap v4732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:14 smithi116 ceph-mon[37097]: pgmap v4732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:14 smithi171 ceph-mon[37789]: pgmap v4732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8000.785368 secs 2024-07-26T10:54:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8000.785368 secs 2024-07-26T10:54:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8000.785368 secs 2024-07-26T10:54:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:17 smithi044 ceph-mon[32404]: pgmap v4733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:17 smithi116 ceph-mon[37097]: pgmap v4733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:17 smithi171 ceph-mon[37789]: pgmap v4733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:19 smithi044 ceph-mon[32404]: pgmap v4734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:19 smithi116 ceph-mon[37097]: pgmap v4734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:19 smithi171 ceph-mon[37789]: pgmap v4734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:21 smithi116 ceph-mon[37097]: pgmap v4735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8005.785512 secs 2024-07-26T10:54:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:21 smithi171 ceph-mon[37789]: pgmap v4735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8005.785512 secs 2024-07-26T10:54:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:21 smithi044 ceph-mon[32404]: pgmap v4735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8005.785512 secs 2024-07-26T10:54:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:23 smithi171 ceph-mon[37789]: pgmap v4736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:23 smithi044 ceph-mon[32404]: pgmap v4736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:23 smithi116 ceph-mon[37097]: pgmap v4736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:25 smithi171 ceph-mon[37789]: pgmap v4737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:25 smithi044 ceph-mon[32404]: pgmap v4737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:25 smithi116 ceph-mon[37097]: pgmap v4737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8010.785680 secs 2024-07-26T10:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:26.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8010.785680 secs 2024-07-26T10:54:26.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8010.785680 secs 2024-07-26T10:54:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:27 smithi171 ceph-mon[37789]: pgmap v4738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:27.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:54:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:27 smithi044 ceph-mon[32404]: pgmap v4738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:27.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:27.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:54:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:27 smithi116 ceph-mon[37097]: pgmap v4738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:54:27.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:27.967+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:54:27.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:27.967+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:54:27.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:27.967+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:28 smithi044 ceph-mon[32404]: pgmap v4739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:28 smithi171 ceph-mon[37789]: pgmap v4739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:28.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:28 smithi116 ceph-mon[37097]: pgmap v4739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:30 smithi171 ceph-mon[37789]: pgmap v4740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:54:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:30 smithi044 ceph-mon[32404]: pgmap v4740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:54:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:30 smithi116 ceph-mon[37097]: pgmap v4740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:54:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8015.785848 secs 2024-07-26T10:54:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8015.785848 secs 2024-07-26T10:54:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8015.785848 secs 2024-07-26T10:54:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:32 smithi171 ceph-mon[37789]: pgmap v4741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:54:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:32 smithi044 ceph-mon[32404]: pgmap v4741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:54:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:32 smithi116 ceph-mon[37097]: pgmap v4741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:54:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:34 smithi171 ceph-mon[37789]: pgmap v4742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:34 smithi044 ceph-mon[32404]: pgmap v4742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:34 smithi116 ceph-mon[37097]: pgmap v4742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:36 smithi044 ceph-mon[32404]: pgmap v4743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:54:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8020.785993 secs 2024-07-26T10:54:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:36 smithi116 ceph-mon[37097]: pgmap v4743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:54:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8020.785993 secs 2024-07-26T10:54:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:36 smithi171 ceph-mon[37789]: pgmap v4743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:54:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8020.785993 secs 2024-07-26T10:54:38.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:38 smithi044 ceph-mon[32404]: pgmap v4744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:38 smithi116 ceph-mon[37097]: pgmap v4744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:38 smithi171 ceph-mon[37789]: pgmap v4744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:54:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:40 smithi044 ceph-mon[32404]: pgmap v4745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:40 smithi116 ceph-mon[37097]: pgmap v4745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:40 smithi171 ceph-mon[37789]: pgmap v4745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8025.786140 secs 2024-07-26T10:54:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8025.786140 secs 2024-07-26T10:54:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8025.786140 secs 2024-07-26T10:54:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:42 smithi044 ceph-mon[32404]: pgmap v4746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:42 smithi116 ceph-mon[37097]: pgmap v4746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:42 smithi171 ceph-mon[37789]: pgmap v4746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:42.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:42.971+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:54:42.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:42.971+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:54:42.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:42.971+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:44 smithi044 ceph-mon[32404]: pgmap v4747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:44 smithi116 ceph-mon[37097]: pgmap v4747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:44 smithi171 ceph-mon[37789]: pgmap v4747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:46 smithi044 ceph-mon[32404]: pgmap v4748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8030.786296 secs 2024-07-26T10:54:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:46 smithi116 ceph-mon[37097]: pgmap v4748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8030.786296 secs 2024-07-26T10:54:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:46 smithi171 ceph-mon[37789]: pgmap v4748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8030.786296 secs 2024-07-26T10:54:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:48 smithi044 ceph-mon[32404]: pgmap v4749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:48 smithi116 ceph-mon[37097]: pgmap v4749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:48 smithi171 ceph-mon[37789]: pgmap v4749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:50 smithi044 ceph-mon[32404]: pgmap v4750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:50.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:50 smithi116 ceph-mon[37097]: pgmap v4750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:50 smithi171 ceph-mon[37789]: pgmap v4750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:54:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8035.786461 secs 2024-07-26T10:54:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8035.786461 secs 2024-07-26T10:54:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8035.786461 secs 2024-07-26T10:54:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:52 smithi044 ceph-mon[32404]: pgmap v4751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:52 smithi116 ceph-mon[37097]: pgmap v4751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:52 smithi171 ceph-mon[37789]: pgmap v4751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:54:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:54 smithi044 ceph-mon[32404]: pgmap v4752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:54 smithi116 ceph-mon[37097]: pgmap v4752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:54 smithi171 ceph-mon[37789]: pgmap v4752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:54:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:56 smithi044 ceph-mon[32404]: pgmap v4753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8040.786568 secs 2024-07-26T10:54:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:56 smithi116 ceph-mon[37097]: pgmap v4753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8040.786568 secs 2024-07-26T10:54:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:56 smithi171 ceph-mon[37789]: pgmap v4753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:54:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8040.786568 secs 2024-07-26T10:54:57.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:57.981+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:54:57.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:57.981+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:54:57.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:54:57.981+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:54:58 smithi044 ceph-mon[32404]: pgmap v4754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:54:58 smithi116 ceph-mon[37097]: pgmap v4754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:54:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:54:58 smithi171 ceph-mon[37789]: pgmap v4754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:00 smithi044 ceph-mon[32404]: pgmap v4755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:00 smithi116 ceph-mon[37097]: pgmap v4755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:00 smithi171 ceph-mon[37789]: pgmap v4755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8045.786722 secs 2024-07-26T10:55:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8045.786722 secs 2024-07-26T10:55:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8045.786722 secs 2024-07-26T10:55:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:02 smithi044 ceph-mon[32404]: pgmap v4756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:02 smithi116 ceph-mon[37097]: pgmap v4756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:02 smithi171 ceph-mon[37789]: pgmap v4756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:04 smithi044 ceph-mon[32404]: pgmap v4757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:04 smithi116 ceph-mon[37097]: pgmap v4757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:04 smithi171 ceph-mon[37789]: pgmap v4757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:06.391 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:06 smithi171 ceph-mon[37789]: pgmap v4758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:06.391 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8050.786883 secs 2024-07-26T10:55:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:06 smithi044 ceph-mon[32404]: pgmap v4758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:06.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8050.786883 secs 2024-07-26T10:55:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:06 smithi116 ceph-mon[37097]: pgmap v4758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8050.786883 secs 2024-07-26T10:55:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:08 smithi044 ceph-mon[32404]: pgmap v4759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:08 smithi116 ceph-mon[37097]: pgmap v4759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:08 smithi171 ceph-mon[37789]: pgmap v4759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:10 smithi044 ceph-mon[32404]: pgmap v4760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:10 smithi116 ceph-mon[37097]: pgmap v4760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:10 smithi171 ceph-mon[37789]: pgmap v4760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8055.787017 secs 2024-07-26T10:55:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8055.787017 secs 2024-07-26T10:55:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8055.787017 secs 2024-07-26T10:55:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:12 smithi044 ceph-mon[32404]: pgmap v4761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:12 smithi116 ceph-mon[37097]: pgmap v4761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:12 smithi171 ceph-mon[37789]: pgmap v4761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:12.985+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:55:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:12.985+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:55:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:12.985+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:14 smithi044 ceph-mon[32404]: pgmap v4762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:14 smithi116 ceph-mon[37097]: pgmap v4762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:14 smithi171 ceph-mon[37789]: pgmap v4762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:16 smithi044 ceph-mon[32404]: pgmap v4763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8060.787182 secs 2024-07-26T10:55:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:16 smithi116 ceph-mon[37097]: pgmap v4763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8060.787182 secs 2024-07-26T10:55:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:16 smithi171 ceph-mon[37789]: pgmap v4763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8060.787182 secs 2024-07-26T10:55:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:18 smithi044 ceph-mon[32404]: pgmap v4764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:18 smithi116 ceph-mon[37097]: pgmap v4764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:18 smithi171 ceph-mon[37789]: pgmap v4764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:20 smithi044 ceph-mon[32404]: pgmap v4765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:20.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:20 smithi116 ceph-mon[37097]: pgmap v4765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:20 smithi171 ceph-mon[37789]: pgmap v4765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:55:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8065.787329 secs 2024-07-26T10:55:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8065.787329 secs 2024-07-26T10:55:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8065.787329 secs 2024-07-26T10:55:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:22 smithi116 ceph-mon[37097]: pgmap v4766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:22 smithi171 ceph-mon[37789]: pgmap v4766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:22 smithi044 ceph-mon[32404]: pgmap v4766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:24 smithi171 ceph-mon[37789]: pgmap v4767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:24 smithi044 ceph-mon[32404]: pgmap v4767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:24 smithi116 ceph-mon[37097]: pgmap v4767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:26.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:26 smithi116 ceph-mon[37097]: pgmap v4768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:26.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8070.787499 secs 2024-07-26T10:55:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:26 smithi171 ceph-mon[37789]: pgmap v4768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8070.787499 secs 2024-07-26T10:55:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:26 smithi044 ceph-mon[32404]: pgmap v4768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:55:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8070.787499 secs 2024-07-26T10:55:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:27.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:27.989+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:55:27.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:27.989+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:55:27.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:27.989+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: pgmap v4769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:28.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: pgmap v4769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:28.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:28.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: pgmap v4769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:55:28.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:28.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:28.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:55:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:29 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:55:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:29 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:55:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:29 smithi171 ceph-mon[37789]: pgmap v4770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:55:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:29 smithi171 ceph-mon[37789]: pgmap v4771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:29 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:29 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:29 smithi044 ceph-mon[32404]: pgmap v4770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:55:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:29 smithi044 ceph-mon[32404]: pgmap v4771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:29 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T10:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:29 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T10:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:29 smithi116 ceph-mon[37097]: pgmap v4770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T10:55:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:29 smithi116 ceph-mon[37097]: pgmap v4771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:31 smithi171 ceph-mon[37789]: pgmap v4772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8075.787712 secs 2024-07-26T10:55:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:31 smithi044 ceph-mon[32404]: pgmap v4772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8075.787712 secs 2024-07-26T10:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:31 smithi116 ceph-mon[37097]: pgmap v4772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8075.787712 secs 2024-07-26T10:55:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:33 smithi171 ceph-mon[37789]: pgmap v4773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:33 smithi044 ceph-mon[32404]: pgmap v4773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:33 smithi116 ceph-mon[37097]: pgmap v4773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:35 smithi171 ceph-mon[37789]: pgmap v4774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:35 smithi044 ceph-mon[32404]: pgmap v4774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:35 smithi116 ceph-mon[37097]: pgmap v4774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:55:36.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8080.787878 secs 2024-07-26T10:55:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8080.787878 secs 2024-07-26T10:55:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8080.787878 secs 2024-07-26T10:55:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:37 smithi044 ceph-mon[32404]: pgmap v4775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:55:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:37 smithi116 ceph-mon[37097]: pgmap v4775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:55:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:37 smithi171 ceph-mon[37789]: pgmap v4775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T10:55:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:39 smithi044 ceph-mon[32404]: pgmap v4776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:55:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:39 smithi116 ceph-mon[37097]: pgmap v4776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:55:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:39 smithi171 ceph-mon[37789]: pgmap v4776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T10:55:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:41 smithi044 ceph-mon[32404]: pgmap v4777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8085.787996 secs 2024-07-26T10:55:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:41 smithi116 ceph-mon[37097]: pgmap v4777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8085.787996 secs 2024-07-26T10:55:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:41 smithi171 ceph-mon[37789]: pgmap v4777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8085.787996 secs 2024-07-26T10:55:42.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:42.993+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:55:42.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:42.993+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:42.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:42.993+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:55:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:43 smithi044 ceph-mon[32404]: pgmap v4778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:55:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:43 smithi116 ceph-mon[37097]: pgmap v4778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:55:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:43 smithi171 ceph-mon[37789]: pgmap v4778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:55:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:45 smithi044 ceph-mon[32404]: pgmap v4779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:55:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:45 smithi116 ceph-mon[37097]: pgmap v4779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:55:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:45 smithi171 ceph-mon[37789]: pgmap v4779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:55:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8090.788181 secs 2024-07-26T10:55:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8090.788181 secs 2024-07-26T10:55:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8090.788181 secs 2024-07-26T10:55:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:47 smithi044 ceph-mon[32404]: pgmap v4780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:47 smithi116 ceph-mon[37097]: pgmap v4780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:47 smithi171 ceph-mon[37789]: pgmap v4780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:55:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:49 smithi044 ceph-mon[32404]: pgmap v4781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:55:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:49 smithi116 ceph-mon[37097]: pgmap v4781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:55:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:49 smithi171 ceph-mon[37789]: pgmap v4781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T10:55:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:50 smithi044 ceph-mon[32404]: pgmap v4782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:50 smithi116 ceph-mon[37097]: pgmap v4782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:50 smithi171 ceph-mon[37789]: pgmap v4782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8095.788309 secs 2024-07-26T10:55:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8095.788309 secs 2024-07-26T10:55:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8095.788309 secs 2024-07-26T10:55:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:52 smithi044 ceph-mon[32404]: pgmap v4783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:52 smithi116 ceph-mon[37097]: pgmap v4783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:52 smithi171 ceph-mon[37789]: pgmap v4783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:54 smithi044 ceph-mon[32404]: pgmap v4784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:54 smithi116 ceph-mon[37097]: pgmap v4784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:54 smithi171 ceph-mon[37789]: pgmap v4784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:55:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8100.788489 secs 2024-07-26T10:55:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8100.788489 secs 2024-07-26T10:55:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8100.788489 secs 2024-07-26T10:55:57.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:56 smithi044 ceph-mon[32404]: pgmap v4785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:56 smithi116 ceph-mon[37097]: pgmap v4785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:56 smithi171 ceph-mon[37789]: pgmap v4785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:55:58.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:58.002+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:55:58.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:58.002+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:58.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:55:58.002+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:55:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:55:59 smithi044 ceph-mon[32404]: pgmap v4786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:55:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:55:59 smithi116 ceph-mon[37097]: pgmap v4786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:55:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:55:59 smithi171 ceph-mon[37789]: pgmap v4786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:01 smithi044 ceph-mon[32404]: pgmap v4787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8105.788694 secs 2024-07-26T10:56:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:01 smithi116 ceph-mon[37097]: pgmap v4787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8105.788694 secs 2024-07-26T10:56:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:01 smithi171 ceph-mon[37789]: pgmap v4787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8105.788694 secs 2024-07-26T10:56:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:03 smithi044 ceph-mon[32404]: pgmap v4788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:03 smithi116 ceph-mon[37097]: pgmap v4788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:03 smithi171 ceph-mon[37789]: pgmap v4788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:05 smithi044 ceph-mon[32404]: pgmap v4789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:05 smithi116 ceph-mon[37097]: pgmap v4789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:05 smithi171 ceph-mon[37789]: pgmap v4789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8110.788866 secs 2024-07-26T10:56:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8110.788866 secs 2024-07-26T10:56:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8110.788866 secs 2024-07-26T10:56:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:07 smithi044 ceph-mon[32404]: pgmap v4790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:07 smithi116 ceph-mon[37097]: pgmap v4790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:07 smithi171 ceph-mon[37789]: pgmap v4790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:09 smithi044 ceph-mon[32404]: pgmap v4791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:09 smithi116 ceph-mon[37097]: pgmap v4791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:09 smithi171 ceph-mon[37789]: pgmap v4791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:11 smithi044 ceph-mon[32404]: pgmap v4792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8115.789049 secs 2024-07-26T10:56:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:11 smithi116 ceph-mon[37097]: pgmap v4792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8115.789049 secs 2024-07-26T10:56:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:11 smithi171 ceph-mon[37789]: pgmap v4792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8115.789049 secs 2024-07-26T10:56:13.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:13.006+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:56:13.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:13.006+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:56:13.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:13.006+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:13 smithi044 ceph-mon[32404]: pgmap v4793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:13 smithi116 ceph-mon[37097]: pgmap v4793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:13 smithi171 ceph-mon[37789]: pgmap v4793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:14.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 msgr2=0x7fcf540c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T10:56:14.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 msgr2=0x7fcf540c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:56:14.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 0x7fcf540c3ec0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf5806d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:56:14.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 0x7fcf540c3ec0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf5806d000 comp rx=0 tx=0).stop 2024-07-26T10:56:14.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 msgr2=0x7fcf540c3ec0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T10:56:14.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 msgr2=0x7fcf540c3ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:56:14.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5300 0x7fcf540c3ec0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:56:14.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.710+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:56:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.711+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:56:14.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:14.713+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58072000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:56:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:15 smithi044 ceph-mon[32404]: pgmap v4794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:15 smithi116 ceph-mon[37097]: pgmap v4794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:15 smithi171 ceph-mon[37789]: pgmap v4794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:16.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:16.121+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:56:16.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:16.121+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:56:16.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:16.122+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:56:16.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:16.122+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:56:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8120.789232 secs 2024-07-26T10:56:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8120.789232 secs 2024-07-26T10:56:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8120.789232 secs 2024-07-26T10:56:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:17 smithi044 ceph-mon[32404]: pgmap v4795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:17 smithi116 ceph-mon[37097]: pgmap v4795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:17 smithi171 ceph-mon[37789]: pgmap v4795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:19 smithi116 ceph-mon[37097]: pgmap v4796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:19 smithi171 ceph-mon[37789]: pgmap v4796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:19 smithi044 ceph-mon[32404]: pgmap v4796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:21 smithi171 ceph-mon[37789]: pgmap v4797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8125.789413 secs 2024-07-26T10:56:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:21 smithi044 ceph-mon[32404]: pgmap v4797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8125.789413 secs 2024-07-26T10:56:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:21 smithi116 ceph-mon[37097]: pgmap v4797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8125.789413 secs 2024-07-26T10:56:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:23 smithi171 ceph-mon[37789]: pgmap v4798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:23 smithi044 ceph-mon[32404]: pgmap v4798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:23 smithi116 ceph-mon[37097]: pgmap v4798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:25 smithi171 ceph-mon[37789]: pgmap v4799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:25 smithi044 ceph-mon[32404]: pgmap v4799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:25 smithi116 ceph-mon[37097]: pgmap v4799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8130.789572 secs 2024-07-26T10:56:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8130.789572 secs 2024-07-26T10:56:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8130.789572 secs 2024-07-26T10:56:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:27 smithi171 ceph-mon[37789]: pgmap v4800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:27 smithi044 ceph-mon[32404]: pgmap v4800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:27 smithi116 ceph-mon[37097]: pgmap v4800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:28.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:28.016+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:56:28.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:28.016+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:56:28.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:28.016+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: pgmap v4801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: pgmap v4801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: pgmap v4801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:29.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:56:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:56:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:56:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:56:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:56:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: pgmap v4802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:56:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: pgmap v4803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:56:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: pgmap v4802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: pgmap v4803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:56:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: pgmap v4802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: pgmap v4803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:56:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8135.789724 secs 2024-07-26T10:56:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8135.789724 secs 2024-07-26T10:56:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8135.789724 secs 2024-07-26T10:56:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:32 smithi171 ceph-mon[37789]: pgmap v4804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:56:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:32 smithi044 ceph-mon[32404]: pgmap v4804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:56:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:32 smithi116 ceph-mon[37097]: pgmap v4804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:56:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:33 smithi044 ceph-mon[32404]: pgmap v4805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:33 smithi116 ceph-mon[37097]: pgmap v4805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:33 smithi171 ceph-mon[37789]: pgmap v4805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:35 smithi044 ceph-mon[32404]: pgmap v4806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:35 smithi116 ceph-mon[37097]: pgmap v4806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:35 smithi171 ceph-mon[37789]: pgmap v4806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T10:56:36.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8140.789819 secs 2024-07-26T10:56:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8140.789819 secs 2024-07-26T10:56:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8140.789819 secs 2024-07-26T10:56:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:37 smithi044 ceph-mon[32404]: pgmap v4807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:56:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:37 smithi116 ceph-mon[37097]: pgmap v4807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:56:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:37 smithi171 ceph-mon[37789]: pgmap v4807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T10:56:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:39 smithi044 ceph-mon[32404]: pgmap v4808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:39 smithi116 ceph-mon[37097]: pgmap v4808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:39 smithi171 ceph-mon[37789]: pgmap v4808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:56:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8145.789973 secs 2024-07-26T10:56:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8145.789973 secs 2024-07-26T10:56:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8145.789973 secs 2024-07-26T10:56:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:41 smithi044 ceph-mon[32404]: pgmap v4809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:41 smithi116 ceph-mon[37097]: pgmap v4809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:41 smithi171 ceph-mon[37789]: pgmap v4809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:43.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:43.020+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:56:43.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:43.021+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:56:43.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:43.021+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:43 smithi044 ceph-mon[32404]: pgmap v4810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:43 smithi116 ceph-mon[37097]: pgmap v4810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:43 smithi171 ceph-mon[37789]: pgmap v4810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:45 smithi044 ceph-mon[32404]: pgmap v4811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:45 smithi116 ceph-mon[37097]: pgmap v4811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:45 smithi171 ceph-mon[37789]: pgmap v4811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8150.790135 secs 2024-07-26T10:56:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8150.790135 secs 2024-07-26T10:56:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8150.790135 secs 2024-07-26T10:56:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:48 smithi044 ceph-mon[32404]: pgmap v4812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:48 smithi116 ceph-mon[37097]: pgmap v4812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:48 smithi171 ceph-mon[37789]: pgmap v4812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:50 smithi044 ceph-mon[32404]: pgmap v4813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:51.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8155.790294 secs 2024-07-26T10:56:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:50 smithi116 ceph-mon[37097]: pgmap v4813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:51.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8155.790294 secs 2024-07-26T10:56:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:50 smithi171 ceph-mon[37789]: pgmap v4813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:56:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8155.790294 secs 2024-07-26T10:56:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:52 smithi044 ceph-mon[32404]: pgmap v4814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:52 smithi116 ceph-mon[37097]: pgmap v4814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:52 smithi171 ceph-mon[37789]: pgmap v4814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T10:56:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:54 smithi044 ceph-mon[32404]: pgmap v4815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:54 smithi116 ceph-mon[37097]: pgmap v4815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:54 smithi171 ceph-mon[37789]: pgmap v4815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:56:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8160.790459 secs 2024-07-26T10:56:56.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8160.790459 secs 2024-07-26T10:56:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8160.790459 secs 2024-07-26T10:56:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:56 smithi044 ceph-mon[32404]: pgmap v4816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:56 smithi116 ceph-mon[37097]: pgmap v4816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:56 smithi171 ceph-mon[37789]: pgmap v4816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:56:58.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:58.030+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:56:58.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:58.030+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:56:58.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:56:58.030+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:56:58 smithi044 ceph-mon[32404]: pgmap v4817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:56:58 smithi116 ceph-mon[37097]: pgmap v4817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:56:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:56:58 smithi171 ceph-mon[37789]: pgmap v4817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:00 smithi044 ceph-mon[32404]: pgmap v4818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:01.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8165.790612 secs 2024-07-26T10:57:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:00 smithi116 ceph-mon[37097]: pgmap v4818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:01.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8165.790612 secs 2024-07-26T10:57:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:00 smithi171 ceph-mon[37789]: pgmap v4818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8165.790612 secs 2024-07-26T10:57:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:02 smithi044 ceph-mon[32404]: pgmap v4819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:02 smithi116 ceph-mon[37097]: pgmap v4819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:02 smithi171 ceph-mon[37789]: pgmap v4819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:05 smithi044 ceph-mon[32404]: pgmap v4820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:05 smithi116 ceph-mon[37097]: pgmap v4820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:05 smithi171 ceph-mon[37789]: pgmap v4820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8170.790772 secs 2024-07-26T10:57:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8170.790772 secs 2024-07-26T10:57:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8170.790772 secs 2024-07-26T10:57:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:07 smithi116 ceph-mon[37097]: pgmap v4821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:07 smithi171 ceph-mon[37789]: pgmap v4821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:07 smithi044 ceph-mon[32404]: pgmap v4821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:09 smithi171 ceph-mon[37789]: pgmap v4822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:09 smithi044 ceph-mon[32404]: pgmap v4822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:09 smithi116 ceph-mon[37097]: pgmap v4822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:11 smithi171 ceph-mon[37789]: pgmap v4823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8175.790921 secs 2024-07-26T10:57:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:11 smithi044 ceph-mon[32404]: pgmap v4823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8175.790921 secs 2024-07-26T10:57:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:11 smithi116 ceph-mon[37097]: pgmap v4823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8175.790921 secs 2024-07-26T10:57:13.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:13.034+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:57:13.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:13.034+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:57:13.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:13.034+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:13 smithi171 ceph-mon[37789]: pgmap v4824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:13 smithi044 ceph-mon[32404]: pgmap v4824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:13 smithi116 ceph-mon[37097]: pgmap v4824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:15 smithi171 ceph-mon[37789]: pgmap v4825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:15 smithi044 ceph-mon[32404]: pgmap v4825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:15 smithi116 ceph-mon[37097]: pgmap v4825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8180.791084 secs 2024-07-26T10:57:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8180.791084 secs 2024-07-26T10:57:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8180.791084 secs 2024-07-26T10:57:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:17 smithi171 ceph-mon[37789]: pgmap v4826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:17 smithi044 ceph-mon[32404]: pgmap v4826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:17 smithi116 ceph-mon[37097]: pgmap v4826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:19 smithi171 ceph-mon[37789]: pgmap v4827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:19 smithi044 ceph-mon[32404]: pgmap v4827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:19 smithi116 ceph-mon[37097]: pgmap v4827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:21 smithi171 ceph-mon[37789]: pgmap v4828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8185.791237 secs 2024-07-26T10:57:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:21 smithi044 ceph-mon[32404]: pgmap v4828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8185.791237 secs 2024-07-26T10:57:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:21 smithi116 ceph-mon[37097]: pgmap v4828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8185.791237 secs 2024-07-26T10:57:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:23 smithi171 ceph-mon[37789]: pgmap v4829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:23 smithi044 ceph-mon[32404]: pgmap v4829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:23 smithi116 ceph-mon[37097]: pgmap v4829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:25 smithi171 ceph-mon[37789]: pgmap v4830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:25 smithi044 ceph-mon[32404]: pgmap v4830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:25 smithi116 ceph-mon[37097]: pgmap v4830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:57:26.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8190.791401 secs 2024-07-26T10:57:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8190.791401 secs 2024-07-26T10:57:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8190.791401 secs 2024-07-26T10:57:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:27 smithi044 ceph-mon[32404]: pgmap v4831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:27 smithi116 ceph-mon[37097]: pgmap v4831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:27 smithi171 ceph-mon[37789]: pgmap v4831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:28.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:28.038+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:57:28.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:28.038+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:57:28.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:28.038+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:29 smithi044 ceph-mon[32404]: pgmap v4832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:29 smithi116 ceph-mon[37097]: pgmap v4832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:29 smithi171 ceph-mon[37789]: pgmap v4832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:30.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:31 smithi044 ceph-mon[32404]: pgmap v4833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8195.791506 secs 2024-07-26T10:57:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:57:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:31 smithi116 ceph-mon[37097]: pgmap v4833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8195.791506 secs 2024-07-26T10:57:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:57:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:31 smithi171 ceph-mon[37789]: pgmap v4833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8195.791506 secs 2024-07-26T10:57:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:57:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:32 smithi044 ceph-mon[32404]: pgmap v4834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:57:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:32 smithi044 ceph-mon[32404]: pgmap v4835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:32 smithi116 ceph-mon[37097]: pgmap v4834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:57:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:32 smithi116 ceph-mon[37097]: pgmap v4835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:32 smithi171 ceph-mon[37789]: pgmap v4834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T10:57:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:32 smithi171 ceph-mon[37789]: pgmap v4835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:34 smithi044 ceph-mon[32404]: pgmap v4836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:34 smithi116 ceph-mon[37097]: pgmap v4836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:34 smithi171 ceph-mon[37789]: pgmap v4836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:36 smithi044 ceph-mon[32404]: pgmap v4837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8200.791676 secs 2024-07-26T10:57:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:36 smithi116 ceph-mon[37097]: pgmap v4837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8200.791676 secs 2024-07-26T10:57:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:36 smithi171 ceph-mon[37789]: pgmap v4837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8200.791676 secs 2024-07-26T10:57:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:38 smithi044 ceph-mon[32404]: pgmap v4838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:57:38.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:38 smithi116 ceph-mon[37097]: pgmap v4838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:57:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:38 smithi171 ceph-mon[37789]: pgmap v4838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T10:57:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:40 smithi044 ceph-mon[32404]: pgmap v4839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:40 smithi116 ceph-mon[37097]: pgmap v4839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:40 smithi171 ceph-mon[37789]: pgmap v4839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T10:57:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8205.791781 secs 2024-07-26T10:57:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8205.791781 secs 2024-07-26T10:57:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8205.791781 secs 2024-07-26T10:57:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:42 smithi044 ceph-mon[32404]: pgmap v4840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:57:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:42 smithi116 ceph-mon[37097]: pgmap v4840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:57:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:42 smithi171 ceph-mon[37789]: pgmap v4840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:57:43.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:43.042+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:57:43.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:43.043+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:43.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:43.043+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:57:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:44 smithi044 ceph-mon[32404]: pgmap v4841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:44 smithi116 ceph-mon[37097]: pgmap v4841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:44 smithi171 ceph-mon[37789]: pgmap v4841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:46 smithi044 ceph-mon[32404]: pgmap v4842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8210.791886 secs 2024-07-26T10:57:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:46 smithi116 ceph-mon[37097]: pgmap v4842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8210.791886 secs 2024-07-26T10:57:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:46 smithi171 ceph-mon[37789]: pgmap v4842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8210.791886 secs 2024-07-26T10:57:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:48 smithi044 ceph-mon[32404]: pgmap v4843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:48 smithi116 ceph-mon[37097]: pgmap v4843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:48 smithi171 ceph-mon[37789]: pgmap v4843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:50 smithi044 ceph-mon[32404]: pgmap v4844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:57:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:50 smithi116 ceph-mon[37097]: pgmap v4844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:57:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:50 smithi171 ceph-mon[37789]: pgmap v4844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:57:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8215.792022 secs 2024-07-26T10:57:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8215.792022 secs 2024-07-26T10:57:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8215.792022 secs 2024-07-26T10:57:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:52 smithi044 ceph-mon[32404]: pgmap v4845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:52 smithi116 ceph-mon[37097]: pgmap v4845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:52 smithi171 ceph-mon[37789]: pgmap v4845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:57:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:54 smithi044 ceph-mon[32404]: pgmap v4846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:54 smithi116 ceph-mon[37097]: pgmap v4846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:54 smithi171 ceph-mon[37789]: pgmap v4846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:57:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:56 smithi044 ceph-mon[32404]: pgmap v4847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8220.792183 secs 2024-07-26T10:57:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:56 smithi116 ceph-mon[37097]: pgmap v4847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8220.792183 secs 2024-07-26T10:57:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:56 smithi171 ceph-mon[37789]: pgmap v4847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:57:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8220.792183 secs 2024-07-26T10:57:58.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:58.045+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:57:58.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:58.045+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:57:58.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:57:58.045+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:57:58 smithi044 ceph-mon[32404]: pgmap v4848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:57:58 smithi116 ceph-mon[37097]: pgmap v4848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:57:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:57:58 smithi171 ceph-mon[37789]: pgmap v4848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:00 smithi044 ceph-mon[32404]: pgmap v4849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:00 smithi116 ceph-mon[37097]: pgmap v4849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:00 smithi171 ceph-mon[37789]: pgmap v4849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8225.792357 secs 2024-07-26T10:58:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8225.792357 secs 2024-07-26T10:58:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8225.792357 secs 2024-07-26T10:58:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:02 smithi044 ceph-mon[32404]: pgmap v4850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:02 smithi116 ceph-mon[37097]: pgmap v4850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:02 smithi171 ceph-mon[37789]: pgmap v4850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:04 smithi044 ceph-mon[32404]: pgmap v4851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:04 smithi116 ceph-mon[37097]: pgmap v4851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:04 smithi171 ceph-mon[37789]: pgmap v4851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:06 smithi044 ceph-mon[32404]: pgmap v4852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8230.792528 secs 2024-07-26T10:58:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:06 smithi116 ceph-mon[37097]: pgmap v4852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:06.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8230.792528 secs 2024-07-26T10:58:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:06 smithi171 ceph-mon[37789]: pgmap v4852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8230.792528 secs 2024-07-26T10:58:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:08 smithi044 ceph-mon[32404]: pgmap v4853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:08.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:08 smithi116 ceph-mon[37097]: pgmap v4853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:08 smithi171 ceph-mon[37789]: pgmap v4853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:10.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:10 smithi116 ceph-mon[37097]: pgmap v4854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:10 smithi171 ceph-mon[37789]: pgmap v4854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:10 smithi044 ceph-mon[32404]: pgmap v4854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8235.792706 secs 2024-07-26T10:58:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8235.792706 secs 2024-07-26T10:58:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8235.792706 secs 2024-07-26T10:58:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:12 smithi171 ceph-mon[37789]: pgmap v4855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:12 smithi044 ceph-mon[32404]: pgmap v4855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:12 smithi116 ceph-mon[37097]: pgmap v4855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:13.049+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:58:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:13.049+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:13.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:13.049+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:58:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:14 smithi171 ceph-mon[37789]: pgmap v4856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:14 smithi044 ceph-mon[32404]: pgmap v4856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:14 smithi116 ceph-mon[37097]: pgmap v4856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:16 smithi171 ceph-mon[37789]: pgmap v4857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8240.792876 secs 2024-07-26T10:58:16.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:16 smithi044 ceph-mon[32404]: pgmap v4857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:16.610 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8240.792876 secs 2024-07-26T10:58:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:16 smithi116 ceph-mon[37097]: pgmap v4857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8240.792876 secs 2024-07-26T10:58:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:18 smithi171 ceph-mon[37789]: pgmap v4858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:18 smithi044 ceph-mon[32404]: pgmap v4858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:18 smithi116 ceph-mon[37097]: pgmap v4858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:20 smithi171 ceph-mon[37789]: pgmap v4859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:20 smithi044 ceph-mon[32404]: pgmap v4859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:20 smithi116 ceph-mon[37097]: pgmap v4859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8245.793057 secs 2024-07-26T10:58:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8245.793057 secs 2024-07-26T10:58:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8245.793057 secs 2024-07-26T10:58:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:22 smithi171 ceph-mon[37789]: pgmap v4860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:22 smithi044 ceph-mon[32404]: pgmap v4860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:22 smithi116 ceph-mon[37097]: pgmap v4860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:24 smithi171 ceph-mon[37789]: pgmap v4861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:24 smithi044 ceph-mon[32404]: pgmap v4861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:24 smithi116 ceph-mon[37097]: pgmap v4861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:26 smithi171 ceph-mon[37789]: pgmap v4862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8250.793227 secs 2024-07-26T10:58:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:26 smithi044 ceph-mon[32404]: pgmap v4862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8250.793227 secs 2024-07-26T10:58:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:26 smithi116 ceph-mon[37097]: pgmap v4862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8250.793227 secs 2024-07-26T10:58:28.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:28.058+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:58:28.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:28.058+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:28.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:28.058+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:58:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:28 smithi044 ceph-mon[32404]: pgmap v4863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:28 smithi116 ceph-mon[37097]: pgmap v4863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:28 smithi171 ceph-mon[37789]: pgmap v4863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:30 smithi044 ceph-mon[32404]: pgmap v4864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:30 smithi116 ceph-mon[37097]: pgmap v4864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:30 smithi171 ceph-mon[37789]: pgmap v4864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8255.793399 secs 2024-07-26T10:58:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8255.793399 secs 2024-07-26T10:58:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8255.793399 secs 2024-07-26T10:58:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:32 smithi044 ceph-mon[32404]: pgmap v4865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:32 smithi116 ceph-mon[37097]: pgmap v4865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:32 smithi171 ceph-mon[37789]: pgmap v4865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:33.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:58:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:34 smithi044 ceph-mon[32404]: pgmap v4866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:34.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:34 smithi044 ceph-mon[32404]: pgmap v4867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:58:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:34 smithi116 ceph-mon[37097]: pgmap v4866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:34 smithi116 ceph-mon[37097]: pgmap v4867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:58:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:34 smithi171 ceph-mon[37789]: pgmap v4866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:34 smithi171 ceph-mon[37789]: pgmap v4867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:58:36.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:36 smithi171 ceph-mon[37789]: pgmap v4868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:36.640 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8260.793568 secs 2024-07-26T10:58:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:36 smithi044 ceph-mon[32404]: pgmap v4868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8260.793568 secs 2024-07-26T10:58:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:36 smithi116 ceph-mon[37097]: pgmap v4868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8260.793568 secs 2024-07-26T10:58:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:38 smithi044 ceph-mon[32404]: pgmap v4869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:38 smithi116 ceph-mon[37097]: pgmap v4869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:38 smithi171 ceph-mon[37789]: pgmap v4869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:40 smithi044 ceph-mon[32404]: pgmap v4870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:40 smithi116 ceph-mon[37097]: pgmap v4870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:40 smithi171 ceph-mon[37789]: pgmap v4870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8265.793728 secs 2024-07-26T10:58:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8265.793728 secs 2024-07-26T10:58:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8265.793728 secs 2024-07-26T10:58:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:42 smithi044 ceph-mon[32404]: pgmap v4871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:42 smithi116 ceph-mon[37097]: pgmap v4871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:42 smithi171 ceph-mon[37789]: pgmap v4871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:43.062+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:43.062+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:43.062+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:44 smithi044 ceph-mon[32404]: pgmap v4872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:58:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:44 smithi116 ceph-mon[37097]: pgmap v4872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:58:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:44 smithi171 ceph-mon[37789]: pgmap v4872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:58:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:46 smithi044 ceph-mon[32404]: pgmap v4873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8270.793876 secs 2024-07-26T10:58:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:46 smithi116 ceph-mon[37097]: pgmap v4873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8270.793876 secs 2024-07-26T10:58:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:46 smithi171 ceph-mon[37789]: pgmap v4873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8270.793876 secs 2024-07-26T10:58:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:48 smithi044 ceph-mon[32404]: pgmap v4874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:48 smithi116 ceph-mon[37097]: pgmap v4874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:48 smithi171 ceph-mon[37789]: pgmap v4874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:50 smithi044 ceph-mon[32404]: pgmap v4875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:50 smithi116 ceph-mon[37097]: pgmap v4875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:50 smithi171 ceph-mon[37789]: pgmap v4875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:58:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8275.794024 secs 2024-07-26T10:58:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:51 smithi044 ceph-mon[32404]: pgmap v4876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8275.794024 secs 2024-07-26T10:58:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:51 smithi116 ceph-mon[37097]: pgmap v4876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8275.794024 secs 2024-07-26T10:58:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:51 smithi171 ceph-mon[37789]: pgmap v4876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:58:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:53 smithi044 ceph-mon[32404]: pgmap v4877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:53 smithi116 ceph-mon[37097]: pgmap v4877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:53 smithi171 ceph-mon[37789]: pgmap v4877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:58:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:55 smithi044 ceph-mon[32404]: pgmap v4878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:55 smithi116 ceph-mon[37097]: pgmap v4878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:55 smithi171 ceph-mon[37789]: pgmap v4878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:58:56.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8280.794160 secs 2024-07-26T10:58:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8280.794160 secs 2024-07-26T10:58:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8280.794160 secs 2024-07-26T10:58:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:57 smithi044 ceph-mon[32404]: pgmap v4879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:57 smithi116 ceph-mon[37097]: pgmap v4879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:58:58.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:58.071+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:58:58.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:58.072+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:58:58.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:58:58.072+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:57 smithi171 ceph-mon[37789]: pgmap v4879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:58:59 smithi044 ceph-mon[32404]: pgmap v4880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:58:59 smithi116 ceph-mon[37097]: pgmap v4880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:58:59 smithi171 ceph-mon[37789]: pgmap v4880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8285.794329 secs 2024-07-26T10:59:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:01 smithi044 ceph-mon[32404]: pgmap v4881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8285.794329 secs 2024-07-26T10:59:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:01 smithi116 ceph-mon[37097]: pgmap v4881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8285.794329 secs 2024-07-26T10:59:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:01 smithi171 ceph-mon[37789]: pgmap v4881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:03 smithi044 ceph-mon[32404]: pgmap v4882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:03 smithi116 ceph-mon[37097]: pgmap v4882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:03 smithi171 ceph-mon[37789]: pgmap v4882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:05 smithi044 ceph-mon[32404]: pgmap v4883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:59:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:05 smithi116 ceph-mon[37097]: pgmap v4883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:59:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:05 smithi171 ceph-mon[37789]: pgmap v4883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T10:59:06.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8290.794480 secs 2024-07-26T10:59:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8290.794480 secs 2024-07-26T10:59:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8290.794480 secs 2024-07-26T10:59:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:07 smithi044 ceph-mon[32404]: pgmap v4884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:07 smithi116 ceph-mon[37097]: pgmap v4884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:07 smithi171 ceph-mon[37789]: pgmap v4884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:09 smithi044 ceph-mon[32404]: pgmap v4885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:59:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:09 smithi116 ceph-mon[37097]: pgmap v4885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:59:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:09 smithi171 ceph-mon[37789]: pgmap v4885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T10:59:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8295.794670 secs 2024-07-26T10:59:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:11 smithi044 ceph-mon[32404]: pgmap v4886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:59:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8295.794670 secs 2024-07-26T10:59:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:11 smithi116 ceph-mon[37097]: pgmap v4886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:59:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8295.794670 secs 2024-07-26T10:59:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:11 smithi171 ceph-mon[37789]: pgmap v4886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:59:13.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:13.077+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:59:13.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:13.077+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:59:13.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:13.077+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:13 smithi044 ceph-mon[32404]: pgmap v4887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:13 smithi116 ceph-mon[37097]: pgmap v4887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:13 smithi171 ceph-mon[37789]: pgmap v4887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T10:59:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:15 smithi044 ceph-mon[32404]: pgmap v4888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:15 smithi116 ceph-mon[37097]: pgmap v4888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:15 smithi171 ceph-mon[37789]: pgmap v4888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:16.155+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:59:16.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:16.155+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:59:16.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:16.156+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T10:59:16.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:16.156+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T10:59:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8300.794822 secs 2024-07-26T10:59:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8300.794822 secs 2024-07-26T10:59:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8300.794822 secs 2024-07-26T10:59:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:17 smithi044 ceph-mon[32404]: pgmap v4889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:17 smithi116 ceph-mon[37097]: pgmap v4889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:17 smithi171 ceph-mon[37789]: pgmap v4889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:19 smithi044 ceph-mon[32404]: pgmap v4890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:19 smithi116 ceph-mon[37097]: pgmap v4890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:19 smithi171 ceph-mon[37789]: pgmap v4890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8305.794969 secs 2024-07-26T10:59:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:21 smithi044 ceph-mon[32404]: pgmap v4891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8305.794969 secs 2024-07-26T10:59:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:21 smithi116 ceph-mon[37097]: pgmap v4891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8305.794969 secs 2024-07-26T10:59:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:21 smithi171 ceph-mon[37789]: pgmap v4891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:23 smithi044 ceph-mon[32404]: pgmap v4892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:23 smithi116 ceph-mon[37097]: pgmap v4892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:23 smithi171 ceph-mon[37789]: pgmap v4892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:25 smithi044 ceph-mon[32404]: pgmap v4893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:26.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:25 smithi116 ceph-mon[37097]: pgmap v4893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:25 smithi171 ceph-mon[37789]: pgmap v4893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8310.795131 secs 2024-07-26T10:59:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8310.795131 secs 2024-07-26T10:59:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8310.795131 secs 2024-07-26T10:59:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:27 smithi171 ceph-mon[37789]: pgmap v4894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:28.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:28.086+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:59:28.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:28.086+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:59:28.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:28.086+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:28.090 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:27 smithi044 ceph-mon[32404]: pgmap v4894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:27 smithi116 ceph-mon[37097]: pgmap v4894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T10:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:29 smithi171 ceph-mon[37789]: pgmap v4895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:29 smithi044 ceph-mon[32404]: pgmap v4895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:29 smithi116 ceph-mon[37097]: pgmap v4895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T10:59:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8315.795263 secs 2024-07-26T10:59:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:31 smithi171 ceph-mon[37789]: pgmap v4896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8315.795263 secs 2024-07-26T10:59:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:31 smithi044 ceph-mon[32404]: pgmap v4896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8315.795263 secs 2024-07-26T10:59:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:31 smithi116 ceph-mon[37097]: pgmap v4896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:34.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:33 smithi116 ceph-mon[37097]: pgmap v4897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:34.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:34.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:33 smithi171 ceph-mon[37789]: pgmap v4897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:33 smithi044 ceph-mon[32404]: pgmap v4897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:34.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:34.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:34.160 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: pgmap v4898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: pgmap v4898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: pgmap v4898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:37.154 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8320.795373 secs 2024-07-26T10:59:37.154 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:37.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8320.795373 secs 2024-07-26T10:59:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8320.795373 secs 2024-07-26T10:59:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:37 smithi044 ceph-mon[32404]: pgmap v4899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:37 smithi116 ceph-mon[37097]: pgmap v4899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:37 smithi171 ceph-mon[37789]: pgmap v4899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: pgmap v4900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: pgmap v4901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: pgmap v4900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:59:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: pgmap v4901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:39.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:39.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: pgmap v4900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T10:59:39.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: pgmap v4901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:39.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T10:59:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:41 smithi044 ceph-mon[32404]: pgmap v4902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8325.795531 secs 2024-07-26T10:59:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:41 smithi116 ceph-mon[37097]: pgmap v4902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8325.795531 secs 2024-07-26T10:59:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:41 smithi171 ceph-mon[37789]: pgmap v4902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8325.795531 secs 2024-07-26T10:59:43.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:43.091+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:59:43.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:43.091+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:43.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:43.091+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:59:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:43 smithi044 ceph-mon[32404]: pgmap v4903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:43 smithi116 ceph-mon[37097]: pgmap v4903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:43 smithi171 ceph-mon[37789]: pgmap v4903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T10:59:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:45 smithi044 ceph-mon[32404]: pgmap v4904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:45 smithi116 ceph-mon[37097]: pgmap v4904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:45 smithi171 ceph-mon[37789]: pgmap v4904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8330.795701 secs 2024-07-26T10:59:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8330.795701 secs 2024-07-26T10:59:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8330.795701 secs 2024-07-26T10:59:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:47 smithi044 ceph-mon[32404]: pgmap v4905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:59:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:47 smithi116 ceph-mon[37097]: pgmap v4905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:59:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:47 smithi171 ceph-mon[37789]: pgmap v4905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T10:59:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:49 smithi044 ceph-mon[32404]: pgmap v4906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:59:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:49 smithi116 ceph-mon[37097]: pgmap v4906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:59:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:49 smithi171 ceph-mon[37789]: pgmap v4906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T10:59:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:51 smithi044 ceph-mon[32404]: pgmap v4907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8335.795869 secs 2024-07-26T10:59:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:51 smithi116 ceph-mon[37097]: pgmap v4907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8335.795869 secs 2024-07-26T10:59:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:51 smithi171 ceph-mon[37789]: pgmap v4907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8335.795869 secs 2024-07-26T10:59:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:53 smithi044 ceph-mon[32404]: pgmap v4908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:53 smithi116 ceph-mon[37097]: pgmap v4908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:53 smithi171 ceph-mon[37789]: pgmap v4908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:55 smithi044 ceph-mon[32404]: pgmap v4909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:55.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:55 smithi116 ceph-mon[37097]: pgmap v4909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:55 smithi171 ceph-mon[37789]: pgmap v4909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T10:59:56.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8340.796033 secs 2024-07-26T10:59:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8340.796033 secs 2024-07-26T10:59:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8340.796033 secs 2024-07-26T10:59:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:57 smithi044 ceph-mon[32404]: pgmap v4910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:57 smithi116 ceph-mon[37097]: pgmap v4910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:57 smithi171 ceph-mon[37789]: pgmap v4910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T10:59:58.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:58.095+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T10:59:58.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:58.095+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T10:59:58.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T10:59:58.095+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 10:59:59 smithi044 ceph-mon[32404]: pgmap v4911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:59:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 10:59:59 smithi116 ceph-mon[37097]: pgmap v4911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T10:59:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 10:59:59 smithi171 ceph-mon[37789]: pgmap v4911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:00:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:00:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:00:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:01 smithi044 ceph-mon[32404]: pgmap v4912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8345.796180 secs 2024-07-26T11:00:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:01 smithi116 ceph-mon[37097]: pgmap v4912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8345.796180 secs 2024-07-26T11:00:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:01 smithi171 ceph-mon[37789]: pgmap v4912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8345.796180 secs 2024-07-26T11:00:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:03 smithi044 ceph-mon[32404]: pgmap v4913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:03.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:03 smithi116 ceph-mon[37097]: pgmap v4913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:03 smithi171 ceph-mon[37789]: pgmap v4913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:05 smithi044 ceph-mon[32404]: pgmap v4914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:05 smithi116 ceph-mon[37097]: pgmap v4914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:05 smithi171 ceph-mon[37789]: pgmap v4914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8350.796328 secs 2024-07-26T11:00:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8350.796328 secs 2024-07-26T11:00:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8350.796328 secs 2024-07-26T11:00:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:07 smithi044 ceph-mon[32404]: pgmap v4915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:07 smithi116 ceph-mon[37097]: pgmap v4915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:07 smithi171 ceph-mon[37789]: pgmap v4915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:09 smithi044 ceph-mon[32404]: pgmap v4916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:09 smithi116 ceph-mon[37097]: pgmap v4916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:09 smithi171 ceph-mon[37789]: pgmap v4916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:11 smithi044 ceph-mon[32404]: pgmap v4917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8355.796495 secs 2024-07-26T11:00:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:11 smithi116 ceph-mon[37097]: pgmap v4917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8355.796495 secs 2024-07-26T11:00:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:11 smithi171 ceph-mon[37789]: pgmap v4917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8355.796495 secs 2024-07-26T11:00:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:13.098+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:00:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:13.098+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:00:13.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:13.098+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:13 smithi044 ceph-mon[32404]: pgmap v4918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:13 smithi116 ceph-mon[37097]: pgmap v4918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:13 smithi171 ceph-mon[37789]: pgmap v4918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:15 smithi044 ceph-mon[32404]: pgmap v4919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:15 smithi116 ceph-mon[37097]: pgmap v4919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:15 smithi171 ceph-mon[37789]: pgmap v4919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8360.796671 secs 2024-07-26T11:00:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8360.796671 secs 2024-07-26T11:00:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8360.796671 secs 2024-07-26T11:00:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:17 smithi044 ceph-mon[32404]: pgmap v4920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:17 smithi116 ceph-mon[37097]: pgmap v4920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:17 smithi171 ceph-mon[37789]: pgmap v4920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:19 smithi044 ceph-mon[32404]: pgmap v4921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:19 smithi116 ceph-mon[37097]: pgmap v4921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:19 smithi171 ceph-mon[37789]: pgmap v4921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:21 smithi116 ceph-mon[37097]: pgmap v4922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8365.796809 secs 2024-07-26T11:00:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:21 smithi171 ceph-mon[37789]: pgmap v4922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8365.796809 secs 2024-07-26T11:00:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:21 smithi044 ceph-mon[32404]: pgmap v4922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8365.796809 secs 2024-07-26T11:00:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:23 smithi171 ceph-mon[37789]: pgmap v4923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:23 smithi044 ceph-mon[32404]: pgmap v4923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:23 smithi116 ceph-mon[37097]: pgmap v4923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:25 smithi171 ceph-mon[37789]: pgmap v4924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:25 smithi044 ceph-mon[32404]: pgmap v4924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:25 smithi116 ceph-mon[37097]: pgmap v4924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8370.796946 secs 2024-07-26T11:00:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8370.796946 secs 2024-07-26T11:00:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8370.796946 secs 2024-07-26T11:00:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:27 smithi171 ceph-mon[37789]: pgmap v4925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:27 smithi044 ceph-mon[32404]: pgmap v4925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:27 smithi116 ceph-mon[37097]: pgmap v4925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:28.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:28.106+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:00:28.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:28.106+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:28.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:28.106+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:00:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:29 smithi171 ceph-mon[37789]: pgmap v4926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:29 smithi044 ceph-mon[32404]: pgmap v4926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:29 smithi116 ceph-mon[37097]: pgmap v4926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:31 smithi171 ceph-mon[37789]: pgmap v4927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8375.797082 secs 2024-07-26T11:00:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:31 smithi044 ceph-mon[32404]: pgmap v4927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8375.797082 secs 2024-07-26T11:00:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:31 smithi116 ceph-mon[37097]: pgmap v4927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8375.797082 secs 2024-07-26T11:00:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:33 smithi171 ceph-mon[37789]: pgmap v4928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:33 smithi044 ceph-mon[32404]: pgmap v4928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:33 smithi116 ceph-mon[37097]: pgmap v4928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:35 smithi171 ceph-mon[37789]: pgmap v4929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:35 smithi044 ceph-mon[32404]: pgmap v4929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:35 smithi116 ceph-mon[37097]: pgmap v4929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:00:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8380.797222 secs 2024-07-26T11:00:36.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8380.797222 secs 2024-07-26T11:00:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8380.797222 secs 2024-07-26T11:00:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:37 smithi171 ceph-mon[37789]: pgmap v4930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:37 smithi044 ceph-mon[32404]: pgmap v4930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:37 smithi116 ceph-mon[37097]: pgmap v4930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:38.726 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:39 smithi044 ceph-mon[32404]: pgmap v4931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:00:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:39 smithi116 ceph-mon[37097]: pgmap v4931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:00:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:39 smithi171 ceph-mon[37789]: pgmap v4931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:39.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:39.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:00:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:40 smithi044 ceph-mon[32404]: pgmap v4932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:00:40.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:40 smithi044 ceph-mon[32404]: pgmap v4933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:40 smithi116 ceph-mon[37097]: pgmap v4932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:00:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:40 smithi116 ceph-mon[37097]: pgmap v4933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:40 smithi171 ceph-mon[37789]: pgmap v4932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:00:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:40 smithi171 ceph-mon[37789]: pgmap v4933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8385.797362 secs 2024-07-26T11:00:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8385.797362 secs 2024-07-26T11:00:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8385.797362 secs 2024-07-26T11:00:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:42 smithi044 ceph-mon[32404]: pgmap v4934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:42 smithi116 ceph-mon[37097]: pgmap v4934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:42 smithi171 ceph-mon[37789]: pgmap v4934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:43.112+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:00:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:43.112+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:00:43.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:43.112+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:44 smithi044 ceph-mon[32404]: pgmap v4935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:44 smithi116 ceph-mon[37097]: pgmap v4935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:44 smithi171 ceph-mon[37789]: pgmap v4935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:46 smithi044 ceph-mon[32404]: pgmap v4936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8390.797498 secs 2024-07-26T11:00:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:46 smithi116 ceph-mon[37097]: pgmap v4936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8390.797498 secs 2024-07-26T11:00:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:46 smithi171 ceph-mon[37789]: pgmap v4936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:00:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8390.797498 secs 2024-07-26T11:00:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:48 smithi171 ceph-mon[37789]: pgmap v4937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:48 smithi044 ceph-mon[32404]: pgmap v4937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:48 smithi116 ceph-mon[37097]: pgmap v4937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:50 smithi044 ceph-mon[32404]: pgmap v4938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:50 smithi116 ceph-mon[37097]: pgmap v4938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:50 smithi171 ceph-mon[37789]: pgmap v4938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:00:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8395.797606 secs 2024-07-26T11:00:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8395.797606 secs 2024-07-26T11:00:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8395.797606 secs 2024-07-26T11:00:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:52 smithi044 ceph-mon[32404]: pgmap v4939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:52 smithi116 ceph-mon[37097]: pgmap v4939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:52 smithi171 ceph-mon[37789]: pgmap v4939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:54 smithi044 ceph-mon[32404]: pgmap v4940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:54 smithi116 ceph-mon[37097]: pgmap v4940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:54 smithi171 ceph-mon[37789]: pgmap v4940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:00:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:56 smithi044 ceph-mon[32404]: pgmap v4941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8400.797731 secs 2024-07-26T11:00:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:56 smithi116 ceph-mon[37097]: pgmap v4941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8400.797731 secs 2024-07-26T11:00:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:56 smithi171 ceph-mon[37789]: pgmap v4941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:00:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8400.797731 secs 2024-07-26T11:00:58.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:58.114+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:00:58.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:58.114+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:58.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:00:58.114+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:00:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:00:58 smithi044 ceph-mon[32404]: pgmap v4942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:00:58 smithi116 ceph-mon[37097]: pgmap v4942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:00:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:00:58 smithi171 ceph-mon[37789]: pgmap v4942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:00 smithi044 ceph-mon[32404]: pgmap v4943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:00 smithi116 ceph-mon[37097]: pgmap v4943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:00 smithi171 ceph-mon[37789]: pgmap v4943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8405.797827 secs 2024-07-26T11:01:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8405.797827 secs 2024-07-26T11:01:01.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8405.797827 secs 2024-07-26T11:01:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:02 smithi044 ceph-mon[32404]: pgmap v4944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:02 smithi116 ceph-mon[37097]: pgmap v4944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:02 smithi171 ceph-mon[37789]: pgmap v4944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:03 smithi044 ceph-mon[32404]: pgmap v4945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:03 smithi116 ceph-mon[37097]: pgmap v4945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:03 smithi171 ceph-mon[37789]: pgmap v4945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:06 smithi044 ceph-mon[32404]: pgmap v4946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8410.797972 secs 2024-07-26T11:01:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:06 smithi116 ceph-mon[37097]: pgmap v4946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8410.797972 secs 2024-07-26T11:01:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:06 smithi171 ceph-mon[37789]: pgmap v4946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:06.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8410.797972 secs 2024-07-26T11:01:08.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:08 smithi044 ceph-mon[32404]: pgmap v4947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:08.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:08 smithi116 ceph-mon[37097]: pgmap v4947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:08 smithi171 ceph-mon[37789]: pgmap v4947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:10 smithi044 ceph-mon[32404]: pgmap v4948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:10 smithi116 ceph-mon[37097]: pgmap v4948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:10 smithi171 ceph-mon[37789]: pgmap v4948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8415.798114 secs 2024-07-26T11:01:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8415.798114 secs 2024-07-26T11:01:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8415.798114 secs 2024-07-26T11:01:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:12 smithi044 ceph-mon[32404]: pgmap v4949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:12 smithi116 ceph-mon[37097]: pgmap v4949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:12 smithi171 ceph-mon[37789]: pgmap v4949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:13.118+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:01:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:13.118+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:01:13.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:13.118+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:14 smithi044 ceph-mon[32404]: pgmap v4950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:14 smithi116 ceph-mon[37097]: pgmap v4950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:14 smithi171 ceph-mon[37789]: pgmap v4950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:16 smithi044 ceph-mon[32404]: pgmap v4951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8420.798278 secs 2024-07-26T11:01:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:16 smithi116 ceph-mon[37097]: pgmap v4951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8420.798278 secs 2024-07-26T11:01:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:16 smithi171 ceph-mon[37789]: pgmap v4951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8420.798278 secs 2024-07-26T11:01:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:18 smithi044 ceph-mon[32404]: pgmap v4952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:18 smithi116 ceph-mon[37097]: pgmap v4952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:18 smithi171 ceph-mon[37789]: pgmap v4952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:20 smithi044 ceph-mon[32404]: pgmap v4953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:20 smithi116 ceph-mon[37097]: pgmap v4953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:20 smithi171 ceph-mon[37789]: pgmap v4953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8425.798440 secs 2024-07-26T11:01:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8425.798440 secs 2024-07-26T11:01:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8425.798440 secs 2024-07-26T11:01:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:22 smithi044 ceph-mon[32404]: pgmap v4954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:22 smithi116 ceph-mon[37097]: pgmap v4954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:22 smithi171 ceph-mon[37789]: pgmap v4954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:24 smithi044 ceph-mon[32404]: pgmap v4955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:24 smithi116 ceph-mon[37097]: pgmap v4955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:24.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:24 smithi171 ceph-mon[37789]: pgmap v4955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:26 smithi044 ceph-mon[32404]: pgmap v4956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8430.798573 secs 2024-07-26T11:01:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:26 smithi116 ceph-mon[37097]: pgmap v4956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8430.798573 secs 2024-07-26T11:01:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:26 smithi171 ceph-mon[37789]: pgmap v4956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8430.798573 secs 2024-07-26T11:01:28.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:28.127+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:01:28.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:28.127+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:01:28.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:28.127+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:28 smithi044 ceph-mon[32404]: pgmap v4957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:28 smithi116 ceph-mon[37097]: pgmap v4957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:28 smithi171 ceph-mon[37789]: pgmap v4957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:30 smithi044 ceph-mon[32404]: pgmap v4958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:30 smithi116 ceph-mon[37097]: pgmap v4958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:30 smithi171 ceph-mon[37789]: pgmap v4958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:31.181+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:01:31.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:31.181+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:01:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:31.181+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:01:31.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:31.182+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:01:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8435.798713 secs 2024-07-26T11:01:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8435.798713 secs 2024-07-26T11:01:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8435.798713 secs 2024-07-26T11:01:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:32 smithi044 ceph-mon[32404]: pgmap v4959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:32 smithi116 ceph-mon[37097]: pgmap v4959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:32 smithi171 ceph-mon[37789]: pgmap v4959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:33 smithi044 ceph-mon[32404]: pgmap v4960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:33 smithi116 ceph-mon[37097]: pgmap v4960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:33 smithi171 ceph-mon[37789]: pgmap v4960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:01:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:36 smithi044 ceph-mon[32404]: pgmap v4961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8440.798847 secs 2024-07-26T11:01:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:36 smithi116 ceph-mon[37097]: pgmap v4961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8440.798847 secs 2024-07-26T11:01:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:36 smithi171 ceph-mon[37789]: pgmap v4961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:01:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8440.798847 secs 2024-07-26T11:01:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:38 smithi044 ceph-mon[32404]: pgmap v4962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:38 smithi116 ceph-mon[37097]: pgmap v4962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:38 smithi171 ceph-mon[37789]: pgmap v4962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:40 smithi044 ceph-mon[32404]: pgmap v4963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:40 smithi116 ceph-mon[37097]: pgmap v4963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:40 smithi171 ceph-mon[37789]: pgmap v4963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8445.798949 secs 2024-07-26T11:01:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:01:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8445.798949 secs 2024-07-26T11:01:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:01:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8445.798949 secs 2024-07-26T11:01:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:01:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:42 smithi044 ceph-mon[32404]: pgmap v4964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:01:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:42 smithi044 ceph-mon[32404]: pgmap v4965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:42 smithi116 ceph-mon[37097]: pgmap v4964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:01:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:42 smithi116 ceph-mon[37097]: pgmap v4965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:42 smithi171 ceph-mon[37789]: pgmap v4964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:01:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:42 smithi171 ceph-mon[37789]: pgmap v4965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:43.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:43.130+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:01:43.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:43.130+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:01:43.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:43.130+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:44 smithi044 ceph-mon[32404]: pgmap v4966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:44 smithi116 ceph-mon[37097]: pgmap v4966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:44 smithi171 ceph-mon[37789]: pgmap v4966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:01:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:46 smithi044 ceph-mon[32404]: pgmap v4967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8450.799116 secs 2024-07-26T11:01:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:46 smithi116 ceph-mon[37097]: pgmap v4967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8450.799116 secs 2024-07-26T11:01:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:46 smithi171 ceph-mon[37789]: pgmap v4967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8450.799116 secs 2024-07-26T11:01:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:48 smithi044 ceph-mon[32404]: pgmap v4968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:48 smithi116 ceph-mon[37097]: pgmap v4968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:48 smithi171 ceph-mon[37789]: pgmap v4968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:50 smithi044 ceph-mon[32404]: pgmap v4969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:50 smithi116 ceph-mon[37097]: pgmap v4969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:50 smithi171 ceph-mon[37789]: pgmap v4969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:01:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8455.799290 secs 2024-07-26T11:01:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8455.799290 secs 2024-07-26T11:01:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8455.799290 secs 2024-07-26T11:01:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:52 smithi044 ceph-mon[32404]: pgmap v4970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:01:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:52 smithi116 ceph-mon[37097]: pgmap v4970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:01:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:52 smithi171 ceph-mon[37789]: pgmap v4970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:01:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:54 smithi044 ceph-mon[32404]: pgmap v4971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:54 smithi116 ceph-mon[37097]: pgmap v4971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:54 smithi171 ceph-mon[37789]: pgmap v4971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:56 smithi044 ceph-mon[32404]: pgmap v4972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8460.799449 secs 2024-07-26T11:01:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:56 smithi116 ceph-mon[37097]: pgmap v4972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8460.799449 secs 2024-07-26T11:01:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:56 smithi171 ceph-mon[37789]: pgmap v4972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:01:56.874 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8460.799449 secs 2024-07-26T11:01:58.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:58.139+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:01:58.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:58.139+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:01:58.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:01:58.139+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:58 smithi044 ceph-mon[32404]: pgmap v4973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:58 smithi116 ceph-mon[37097]: pgmap v4973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:58 smithi171 ceph-mon[37789]: pgmap v4973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:01:59 smithi044 ceph-mon[32404]: pgmap v4974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:01:59 smithi116 ceph-mon[37097]: pgmap v4974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:01:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:01:59 smithi171 ceph-mon[37789]: pgmap v4974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8465.799598 secs 2024-07-26T11:02:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:01 smithi044 ceph-mon[32404]: pgmap v4975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8465.799598 secs 2024-07-26T11:02:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:01 smithi116 ceph-mon[37097]: pgmap v4975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8465.799598 secs 2024-07-26T11:02:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:01 smithi171 ceph-mon[37789]: pgmap v4975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:03 smithi044 ceph-mon[32404]: pgmap v4976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:03 smithi116 ceph-mon[37097]: pgmap v4976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:03 smithi171 ceph-mon[37789]: pgmap v4976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:05 smithi044 ceph-mon[32404]: pgmap v4977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:05 smithi116 ceph-mon[37097]: pgmap v4977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:05 smithi171 ceph-mon[37789]: pgmap v4977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:06.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8470.799747 secs 2024-07-26T11:02:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8470.799747 secs 2024-07-26T11:02:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8470.799747 secs 2024-07-26T11:02:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:07 smithi044 ceph-mon[32404]: pgmap v4978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:07 smithi116 ceph-mon[37097]: pgmap v4978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:07 smithi171 ceph-mon[37789]: pgmap v4978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:09 smithi044 ceph-mon[32404]: pgmap v4979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:09 smithi116 ceph-mon[37097]: pgmap v4979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:09 smithi171 ceph-mon[37789]: pgmap v4979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8475.799912 secs 2024-07-26T11:02:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:11 smithi044 ceph-mon[32404]: pgmap v4980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8475.799912 secs 2024-07-26T11:02:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:11 smithi116 ceph-mon[37097]: pgmap v4980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8475.799912 secs 2024-07-26T11:02:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:11 smithi171 ceph-mon[37789]: pgmap v4980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:13.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:13.143+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:02:13.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:13.144+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:02:13.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:13.144+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:13 smithi044 ceph-mon[32404]: pgmap v4981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:13 smithi116 ceph-mon[37097]: pgmap v4981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:13 smithi171 ceph-mon[37789]: pgmap v4981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:15 smithi044 ceph-mon[32404]: pgmap v4982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:15 smithi116 ceph-mon[37097]: pgmap v4982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:15 smithi171 ceph-mon[37789]: pgmap v4982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:16.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:16.189+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:02:16.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:16.189+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:02:16.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:16.190+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:02:16.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:16.190+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:02:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8480.800077 secs 2024-07-26T11:02:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8480.800077 secs 2024-07-26T11:02:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8480.800077 secs 2024-07-26T11:02:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:17 smithi044 ceph-mon[32404]: pgmap v4983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:18.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:17 smithi116 ceph-mon[37097]: pgmap v4983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:17 smithi171 ceph-mon[37789]: pgmap v4983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:19 smithi044 ceph-mon[32404]: pgmap v4984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:19 smithi116 ceph-mon[37097]: pgmap v4984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:19 smithi171 ceph-mon[37789]: pgmap v4984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8485.800228 secs 2024-07-26T11:02:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:21 smithi044 ceph-mon[32404]: pgmap v4985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8485.800228 secs 2024-07-26T11:02:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:21 smithi116 ceph-mon[37097]: pgmap v4985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8485.800228 secs 2024-07-26T11:02:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:21 smithi171 ceph-mon[37789]: pgmap v4985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:23 smithi044 ceph-mon[32404]: pgmap v4986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:23 smithi116 ceph-mon[37097]: pgmap v4986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:23 smithi171 ceph-mon[37789]: pgmap v4986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:25 smithi044 ceph-mon[32404]: pgmap v4987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:26.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:25 smithi116 ceph-mon[37097]: pgmap v4987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:25 smithi171 ceph-mon[37789]: pgmap v4987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8490.800370 secs 2024-07-26T11:02:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8490.800370 secs 2024-07-26T11:02:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8490.800370 secs 2024-07-26T11:02:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:27 smithi116 ceph-mon[37097]: pgmap v4988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:27 smithi171 ceph-mon[37789]: pgmap v4988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:28.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:28.153+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:02:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:28.153+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:02:28.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:28.153+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:28.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:27 smithi044 ceph-mon[32404]: pgmap v4988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:29 smithi171 ceph-mon[37789]: pgmap v4989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:29 smithi044 ceph-mon[32404]: pgmap v4989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:29 smithi116 ceph-mon[37097]: pgmap v4989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8495.800524 secs 2024-07-26T11:02:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:31 smithi171 ceph-mon[37789]: pgmap v4990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8495.800524 secs 2024-07-26T11:02:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:31 smithi044 ceph-mon[32404]: pgmap v4990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8495.800524 secs 2024-07-26T11:02:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:31 smithi116 ceph-mon[37097]: pgmap v4990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:33 smithi171 ceph-mon[37789]: pgmap v4991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:33 smithi044 ceph-mon[32404]: pgmap v4991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:33 smithi116 ceph-mon[37097]: pgmap v4991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:36.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:35 smithi044 ceph-mon[32404]: pgmap v4992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:35 smithi171 ceph-mon[37789]: pgmap v4992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:35 smithi116 ceph-mon[37097]: pgmap v4992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:02:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8500.800679 secs 2024-07-26T11:02:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8500.800679 secs 2024-07-26T11:02:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8500.800679 secs 2024-07-26T11:02:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:37 smithi171 ceph-mon[37789]: pgmap v4993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:37 smithi044 ceph-mon[32404]: pgmap v4993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:37 smithi116 ceph-mon[37097]: pgmap v4993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:40 smithi044 ceph-mon[32404]: pgmap v4994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:40 smithi116 ceph-mon[37097]: pgmap v4994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:40 smithi171 ceph-mon[37789]: pgmap v4994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:02:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8505.800831 secs 2024-07-26T11:02:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8505.800831 secs 2024-07-26T11:02:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8505.800831 secs 2024-07-26T11:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:42 smithi044 ceph-mon[32404]: pgmap v4995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:02:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:42 smithi116 ceph-mon[37097]: pgmap v4995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:02:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:42 smithi171 ceph-mon[37789]: pgmap v4995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:02:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:02:43.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:43.156+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:02:43.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:43.156+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:02:43.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:43.156+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:43 smithi044 ceph-mon[32404]: pgmap v4996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:02:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:43 smithi044 ceph-mon[32404]: pgmap v4997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:43 smithi116 ceph-mon[37097]: pgmap v4996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:02:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:43 smithi116 ceph-mon[37097]: pgmap v4997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:43 smithi171 ceph-mon[37789]: pgmap v4996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:02:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:43 smithi171 ceph-mon[37789]: pgmap v4997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:45 smithi044 ceph-mon[32404]: pgmap v4998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:45 smithi116 ceph-mon[37097]: pgmap v4998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:45 smithi171 ceph-mon[37789]: pgmap v4998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8510.800990 secs 2024-07-26T11:02:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8510.800990 secs 2024-07-26T11:02:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8510.800990 secs 2024-07-26T11:02:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:47 smithi044 ceph-mon[32404]: pgmap v4999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:47 smithi116 ceph-mon[37097]: pgmap v4999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:47 smithi171 ceph-mon[37789]: pgmap v4999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:02:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:49 smithi044 ceph-mon[32404]: pgmap v5000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:49 smithi116 ceph-mon[37097]: pgmap v5000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:49 smithi171 ceph-mon[37789]: pgmap v5000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:51 smithi044 ceph-mon[32404]: pgmap v5001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8515.801151 secs 2024-07-26T11:02:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:51 smithi116 ceph-mon[37097]: pgmap v5001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8515.801151 secs 2024-07-26T11:02:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:51 smithi171 ceph-mon[37789]: pgmap v5001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:02:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8515.801151 secs 2024-07-26T11:02:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:53 smithi044 ceph-mon[32404]: pgmap v5002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:02:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:53 smithi116 ceph-mon[37097]: pgmap v5002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:02:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:53 smithi171 ceph-mon[37789]: pgmap v5002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:02:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:55 smithi044 ceph-mon[32404]: pgmap v5003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:55 smithi116 ceph-mon[37097]: pgmap v5003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:55 smithi171 ceph-mon[37789]: pgmap v5003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:02:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8520.801300 secs 2024-07-26T11:02:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8520.801300 secs 2024-07-26T11:02:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8520.801300 secs 2024-07-26T11:02:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:57 smithi044 ceph-mon[32404]: pgmap v5004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:02:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:57 smithi116 ceph-mon[37097]: pgmap v5004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:02:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:57 smithi171 ceph-mon[37789]: pgmap v5004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:02:58.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:58.165+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:02:58.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:58.165+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:02:58.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:02:58.165+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:02:59 smithi044 ceph-mon[32404]: pgmap v5005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:02:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:02:59 smithi116 ceph-mon[37097]: pgmap v5005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:02:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:02:59 smithi171 ceph-mon[37789]: pgmap v5005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:03:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:01 smithi044 ceph-mon[32404]: pgmap v5006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8525.801453 secs 2024-07-26T11:03:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:01 smithi116 ceph-mon[37097]: pgmap v5006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8525.801453 secs 2024-07-26T11:03:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:01 smithi171 ceph-mon[37789]: pgmap v5006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8525.801453 secs 2024-07-26T11:03:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:03 smithi044 ceph-mon[32404]: pgmap v5007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:03:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:03 smithi116 ceph-mon[37097]: pgmap v5007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:03:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:03 smithi171 ceph-mon[37789]: pgmap v5007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:03:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:05 smithi044 ceph-mon[32404]: pgmap v5008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:05.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:05 smithi116 ceph-mon[37097]: pgmap v5008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:05 smithi171 ceph-mon[37789]: pgmap v5008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8530.801590 secs 2024-07-26T11:03:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8530.801590 secs 2024-07-26T11:03:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8530.801590 secs 2024-07-26T11:03:07.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:07 smithi044 ceph-mon[32404]: pgmap v5009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:07 smithi116 ceph-mon[37097]: pgmap v5009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:07 smithi171 ceph-mon[37789]: pgmap v5009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:09 smithi044 ceph-mon[32404]: pgmap v5010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:03:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:09 smithi116 ceph-mon[37097]: pgmap v5010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:03:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:09 smithi171 ceph-mon[37789]: pgmap v5010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:03:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:11 smithi044 ceph-mon[32404]: pgmap v5011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8535.801724 secs 2024-07-26T11:03:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:11 smithi116 ceph-mon[37097]: pgmap v5011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8535.801724 secs 2024-07-26T11:03:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:11 smithi171 ceph-mon[37789]: pgmap v5011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8535.801724 secs 2024-07-26T11:03:13.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:13.170+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:03:13.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:13.170+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:13.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:13.170+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:03:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:13 smithi044 ceph-mon[32404]: pgmap v5012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:03:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:13 smithi116 ceph-mon[37097]: pgmap v5012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:03:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:13 smithi171 ceph-mon[37789]: pgmap v5012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:03:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:15 smithi044 ceph-mon[32404]: pgmap v5013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:03:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:15 smithi116 ceph-mon[37097]: pgmap v5013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:03:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:15 smithi171 ceph-mon[37789]: pgmap v5013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:03:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8540.801883 secs 2024-07-26T11:03:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8540.801883 secs 2024-07-26T11:03:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8540.801883 secs 2024-07-26T11:03:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:17 smithi171 ceph-mon[37789]: pgmap v5014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:03:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:17 smithi044 ceph-mon[32404]: pgmap v5014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:03:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:17 smithi116 ceph-mon[37097]: pgmap v5014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:03:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:19 smithi171 ceph-mon[37789]: pgmap v5015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:03:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:19 smithi044 ceph-mon[32404]: pgmap v5015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:03:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:19 smithi116 ceph-mon[37097]: pgmap v5015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:03:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:21 smithi171 ceph-mon[37789]: pgmap v5016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8545.802022 secs 2024-07-26T11:03:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:21 smithi044 ceph-mon[32404]: pgmap v5016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8545.802022 secs 2024-07-26T11:03:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:21 smithi116 ceph-mon[37097]: pgmap v5016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:03:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8545.802022 secs 2024-07-26T11:03:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:23 smithi171 ceph-mon[37789]: pgmap v5017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:03:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:23 smithi044 ceph-mon[32404]: pgmap v5017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:03:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:23 smithi116 ceph-mon[37097]: pgmap v5017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:03:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:25 smithi171 ceph-mon[37789]: pgmap v5018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:25 smithi044 ceph-mon[32404]: pgmap v5018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:25 smithi116 ceph-mon[37097]: pgmap v5018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8550.802186 secs 2024-07-26T11:03:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8550.802186 secs 2024-07-26T11:03:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8550.802186 secs 2024-07-26T11:03:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:27 smithi171 ceph-mon[37789]: pgmap v5019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:27 smithi044 ceph-mon[32404]: pgmap v5019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:27 smithi116 ceph-mon[37097]: pgmap v5019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:28.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:28.179+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:03:28.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:28.180+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:03:28.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:28.180+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:29 smithi171 ceph-mon[37789]: pgmap v5020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:03:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:29 smithi044 ceph-mon[32404]: pgmap v5020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:03:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:29 smithi116 ceph-mon[37097]: pgmap v5020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:03:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:31 smithi171 ceph-mon[37789]: pgmap v5021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8555.802346 secs 2024-07-26T11:03:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:31 smithi044 ceph-mon[32404]: pgmap v5021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8555.802346 secs 2024-07-26T11:03:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:31 smithi116 ceph-mon[37097]: pgmap v5021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8555.802346 secs 2024-07-26T11:03:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:33 smithi171 ceph-mon[37789]: pgmap v5022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:33 smithi044 ceph-mon[32404]: pgmap v5022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:33 smithi116 ceph-mon[37097]: pgmap v5022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:35 smithi044 ceph-mon[32404]: pgmap v5023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:35 smithi116 ceph-mon[37097]: pgmap v5023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:35 smithi171 ceph-mon[37789]: pgmap v5023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:36.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8560.802508 secs 2024-07-26T11:03:36.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8560.802508 secs 2024-07-26T11:03:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8560.802508 secs 2024-07-26T11:03:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:37 smithi044 ceph-mon[32404]: pgmap v5024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:37 smithi116 ceph-mon[37097]: pgmap v5024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:37 smithi171 ceph-mon[37789]: pgmap v5024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:39 smithi044 ceph-mon[32404]: pgmap v5025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:03:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:39 smithi116 ceph-mon[37097]: pgmap v5025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:03:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:39 smithi171 ceph-mon[37789]: pgmap v5025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:03:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:41 smithi044 ceph-mon[32404]: pgmap v5026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8565.802679 secs 2024-07-26T11:03:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:41 smithi116 ceph-mon[37097]: pgmap v5026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8565.802679 secs 2024-07-26T11:03:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:41 smithi171 ceph-mon[37789]: pgmap v5026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8565.802679 secs 2024-07-26T11:03:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:43.181+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:03:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:43.181+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:03:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:43.181+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:43 smithi044 ceph-mon[32404]: pgmap v5027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:43.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:43.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:03:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:43 smithi116 ceph-mon[37097]: pgmap v5027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:03:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:43 smithi171 ceph-mon[37789]: pgmap v5027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:43.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:03:44.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:44 smithi044 ceph-mon[32404]: pgmap v5028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:03:44.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:44 smithi044 ceph-mon[32404]: pgmap v5029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:44 smithi116 ceph-mon[37097]: pgmap v5028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:03:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:44 smithi116 ceph-mon[37097]: pgmap v5029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:44 smithi171 ceph-mon[37789]: pgmap v5028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:03:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:44 smithi171 ceph-mon[37789]: pgmap v5029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:46 smithi044 ceph-mon[32404]: pgmap v5030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8570.802829 secs 2024-07-26T11:03:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:46 smithi116 ceph-mon[37097]: pgmap v5030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8570.802829 secs 2024-07-26T11:03:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:46 smithi171 ceph-mon[37789]: pgmap v5030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8570.802829 secs 2024-07-26T11:03:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:48 smithi044 ceph-mon[32404]: pgmap v5031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:48 smithi116 ceph-mon[37097]: pgmap v5031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:48 smithi171 ceph-mon[37789]: pgmap v5031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:50 smithi044 ceph-mon[32404]: pgmap v5032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:50 smithi116 ceph-mon[37097]: pgmap v5032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:50 smithi171 ceph-mon[37789]: pgmap v5032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:03:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8575.802994 secs 2024-07-26T11:03:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8575.802994 secs 2024-07-26T11:03:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8575.802994 secs 2024-07-26T11:03:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:52 smithi044 ceph-mon[32404]: pgmap v5033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:52 smithi116 ceph-mon[37097]: pgmap v5033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:52 smithi171 ceph-mon[37789]: pgmap v5033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:54 smithi044 ceph-mon[32404]: pgmap v5034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:03:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:54 smithi116 ceph-mon[37097]: pgmap v5034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:03:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:54 smithi171 ceph-mon[37789]: pgmap v5034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:03:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:56 smithi044 ceph-mon[32404]: pgmap v5035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8580.803150 secs 2024-07-26T11:03:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:56 smithi116 ceph-mon[37097]: pgmap v5035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8580.803150 secs 2024-07-26T11:03:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:56 smithi171 ceph-mon[37789]: pgmap v5035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:03:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8580.803150 secs 2024-07-26T11:03:58.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:58.191+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:03:58.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:58.191+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:03:58.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:03:58.191+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:03:58 smithi044 ceph-mon[32404]: pgmap v5036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:03:58 smithi116 ceph-mon[37097]: pgmap v5036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:03:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:03:58 smithi171 ceph-mon[37789]: pgmap v5036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:00 smithi044 ceph-mon[32404]: pgmap v5037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:00 smithi116 ceph-mon[37097]: pgmap v5037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:00 smithi171 ceph-mon[37789]: pgmap v5037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8585.803290 secs 2024-07-26T11:04:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8585.803290 secs 2024-07-26T11:04:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8585.803290 secs 2024-07-26T11:04:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:02 smithi044 ceph-mon[32404]: pgmap v5038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:02 smithi116 ceph-mon[37097]: pgmap v5038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:02 smithi171 ceph-mon[37789]: pgmap v5038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:04 smithi044 ceph-mon[32404]: pgmap v5039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:04 smithi116 ceph-mon[37097]: pgmap v5039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:04 smithi171 ceph-mon[37789]: pgmap v5039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:06 smithi044 ceph-mon[32404]: pgmap v5040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8590.803434 secs 2024-07-26T11:04:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:06 smithi116 ceph-mon[37097]: pgmap v5040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8590.803434 secs 2024-07-26T11:04:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:06 smithi171 ceph-mon[37789]: pgmap v5040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8590.803434 secs 2024-07-26T11:04:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:08 smithi044 ceph-mon[32404]: pgmap v5041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:08 smithi116 ceph-mon[37097]: pgmap v5041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:08 smithi171 ceph-mon[37789]: pgmap v5041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:10 smithi044 ceph-mon[32404]: pgmap v5042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:10 smithi116 ceph-mon[37097]: pgmap v5042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:10 smithi171 ceph-mon[37789]: pgmap v5042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8595.803595 secs 2024-07-26T11:04:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8595.803595 secs 2024-07-26T11:04:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8595.803595 secs 2024-07-26T11:04:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:12 smithi044 ceph-mon[32404]: pgmap v5043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:12 smithi116 ceph-mon[37097]: pgmap v5043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:12 smithi171 ceph-mon[37789]: pgmap v5043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:13.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:13.195+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:04:13.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:13.195+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:04:13.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:13.195+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:13 smithi044 ceph-mon[32404]: pgmap v5044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:13 smithi116 ceph-mon[37097]: pgmap v5044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:13 smithi171 ceph-mon[37789]: pgmap v5044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:16 smithi044 ceph-mon[32404]: pgmap v5045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8600.803756 secs 2024-07-26T11:04:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:16 smithi116 ceph-mon[37097]: pgmap v5045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8600.803756 secs 2024-07-26T11:04:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:16 smithi171 ceph-mon[37789]: pgmap v5045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8600.803756 secs 2024-07-26T11:04:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:18 smithi044 ceph-mon[32404]: pgmap v5046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:18 smithi116 ceph-mon[37097]: pgmap v5046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:18 smithi171 ceph-mon[37789]: pgmap v5046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:20 smithi044 ceph-mon[32404]: pgmap v5047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:20 smithi116 ceph-mon[37097]: pgmap v5047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:20 smithi171 ceph-mon[37789]: pgmap v5047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8605.803917 secs 2024-07-26T11:04:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8605.803917 secs 2024-07-26T11:04:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8605.803917 secs 2024-07-26T11:04:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:22 smithi044 ceph-mon[32404]: pgmap v5048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:22 smithi116 ceph-mon[37097]: pgmap v5048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:22 smithi171 ceph-mon[37789]: pgmap v5048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:24 smithi044 ceph-mon[32404]: pgmap v5049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:24 smithi116 ceph-mon[37097]: pgmap v5049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:24 smithi171 ceph-mon[37789]: pgmap v5049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:26 smithi044 ceph-mon[32404]: pgmap v5050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8610.804059 secs 2024-07-26T11:04:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:26 smithi116 ceph-mon[37097]: pgmap v5050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8610.804059 secs 2024-07-26T11:04:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:26 smithi171 ceph-mon[37789]: pgmap v5050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8610.804059 secs 2024-07-26T11:04:28.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:28.205+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:04:28.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:28.205+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:04:28.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:28.205+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:28 smithi044 ceph-mon[32404]: pgmap v5051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:28 smithi116 ceph-mon[37097]: pgmap v5051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:28 smithi171 ceph-mon[37789]: pgmap v5051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:30 smithi044 ceph-mon[32404]: pgmap v5052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:30 smithi116 ceph-mon[37097]: pgmap v5052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:30 smithi171 ceph-mon[37789]: pgmap v5052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8615.804230 secs 2024-07-26T11:04:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8615.804230 secs 2024-07-26T11:04:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8615.804230 secs 2024-07-26T11:04:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:32 smithi044 ceph-mon[32404]: pgmap v5053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:32 smithi116 ceph-mon[37097]: pgmap v5053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:32 smithi171 ceph-mon[37789]: pgmap v5053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:34.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:34 smithi044 ceph-mon[32404]: pgmap v5054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:34 smithi116 ceph-mon[37097]: pgmap v5054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:34 smithi171 ceph-mon[37789]: pgmap v5054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:36 smithi044 ceph-mon[32404]: pgmap v5055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8620.804397 secs 2024-07-26T11:04:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:36 smithi116 ceph-mon[37097]: pgmap v5055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8620.804397 secs 2024-07-26T11:04:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:36 smithi171 ceph-mon[37789]: pgmap v5055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8620.804397 secs 2024-07-26T11:04:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:38 smithi044 ceph-mon[32404]: pgmap v5056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:38 smithi116 ceph-mon[37097]: pgmap v5056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:38.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:38 smithi171 ceph-mon[37789]: pgmap v5056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:40.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:40 smithi116 ceph-mon[37097]: pgmap v5057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:40 smithi171 ceph-mon[37789]: pgmap v5057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:40 smithi044 ceph-mon[32404]: pgmap v5057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:04:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8625.804554 secs 2024-07-26T11:04:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8625.804554 secs 2024-07-26T11:04:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8625.804554 secs 2024-07-26T11:04:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:42 smithi171 ceph-mon[37789]: pgmap v5058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:42 smithi044 ceph-mon[32404]: pgmap v5058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:42 smithi116 ceph-mon[37097]: pgmap v5058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:43.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:43.210+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:04:43.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:43.210+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:04:43.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:43.210+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:44 smithi171 ceph-mon[37789]: pgmap v5059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:44 smithi044 ceph-mon[32404]: pgmap v5059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:44.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:44 smithi116 ceph-mon[37097]: pgmap v5059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:45 smithi171 ceph-mon[37789]: pgmap v5060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:04:45.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:45 smithi171 ceph-mon[37789]: pgmap v5061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:45.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:04:45.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:45.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:45.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:45 smithi044 ceph-mon[32404]: pgmap v5060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:04:45.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:45 smithi044 ceph-mon[32404]: pgmap v5061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:45.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:04:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:45 smithi116 ceph-mon[37097]: pgmap v5060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:04:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:45 smithi116 ceph-mon[37097]: pgmap v5061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:04:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8630.804718 secs 2024-07-26T11:04:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8630.804718 secs 2024-07-26T11:04:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8630.804718 secs 2024-07-26T11:04:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:47 smithi171 ceph-mon[37789]: pgmap v5062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:47 smithi044 ceph-mon[32404]: pgmap v5062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:47 smithi116 ceph-mon[37097]: pgmap v5062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:49 smithi044 ceph-mon[32404]: pgmap v5063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:49 smithi116 ceph-mon[37097]: pgmap v5063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:49 smithi171 ceph-mon[37789]: pgmap v5063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:51 smithi044 ceph-mon[32404]: pgmap v5064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8635.804819 secs 2024-07-26T11:04:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:51 smithi116 ceph-mon[37097]: pgmap v5064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8635.804819 secs 2024-07-26T11:04:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:51 smithi171 ceph-mon[37789]: pgmap v5064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:04:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8635.804819 secs 2024-07-26T11:04:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:53 smithi044 ceph-mon[32404]: pgmap v5065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:53 smithi116 ceph-mon[37097]: pgmap v5065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:53 smithi171 ceph-mon[37789]: pgmap v5065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:04:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:55 smithi044 ceph-mon[32404]: pgmap v5066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:55 smithi116 ceph-mon[37097]: pgmap v5066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:55 smithi171 ceph-mon[37789]: pgmap v5066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:04:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8640.804979 secs 2024-07-26T11:04:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8640.804979 secs 2024-07-26T11:04:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8640.804979 secs 2024-07-26T11:04:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:57 smithi044 ceph-mon[32404]: pgmap v5067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:57 smithi116 ceph-mon[37097]: pgmap v5067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:57 smithi171 ceph-mon[37789]: pgmap v5067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:04:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:58.219+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:04:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:58.220+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:04:58.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:04:58.220+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:04:59 smithi044 ceph-mon[32404]: pgmap v5068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:04:59 smithi116 ceph-mon[37097]: pgmap v5068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:04:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:04:59 smithi171 ceph-mon[37789]: pgmap v5068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:01 smithi044 ceph-mon[32404]: pgmap v5069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8645.805149 secs 2024-07-26T11:05:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:01 smithi116 ceph-mon[37097]: pgmap v5069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8645.805149 secs 2024-07-26T11:05:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:01 smithi171 ceph-mon[37789]: pgmap v5069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8645.805149 secs 2024-07-26T11:05:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:03 smithi044 ceph-mon[32404]: pgmap v5070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:03 smithi116 ceph-mon[37097]: pgmap v5070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:03 smithi171 ceph-mon[37789]: pgmap v5070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:05 smithi044 ceph-mon[32404]: pgmap v5071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:05 smithi116 ceph-mon[37097]: pgmap v5071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:05 smithi171 ceph-mon[37789]: pgmap v5071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8650.805322 secs 2024-07-26T11:05:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:06 smithi116 ceph-mon[37097]: pgmap v5072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8650.805322 secs 2024-07-26T11:05:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:06 smithi171 ceph-mon[37789]: pgmap v5072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8650.805322 secs 2024-07-26T11:05:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:06 smithi044 ceph-mon[32404]: pgmap v5072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:09 smithi044 ceph-mon[32404]: pgmap v5073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:09.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:09 smithi116 ceph-mon[37097]: pgmap v5073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:09 smithi171 ceph-mon[37789]: pgmap v5073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:11 smithi044 ceph-mon[32404]: pgmap v5074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8655.805484 secs 2024-07-26T11:05:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:11 smithi116 ceph-mon[37097]: pgmap v5074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8655.805484 secs 2024-07-26T11:05:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:11 smithi171 ceph-mon[37789]: pgmap v5074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8655.805484 secs 2024-07-26T11:05:13.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:13.225+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:05:13.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:13.225+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:05:13.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:13.225+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:13 smithi044 ceph-mon[32404]: pgmap v5075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:13 smithi116 ceph-mon[37097]: pgmap v5075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:13 smithi171 ceph-mon[37789]: pgmap v5075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:15 smithi044 ceph-mon[32404]: pgmap v5076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:15 smithi116 ceph-mon[37097]: pgmap v5076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:15 smithi171 ceph-mon[37789]: pgmap v5076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:16.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:16.221+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:05:16.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:16.221+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:05:16.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:16.222+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:05:16.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:16.223+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:05:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8660.805678 secs 2024-07-26T11:05:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8660.805678 secs 2024-07-26T11:05:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8660.805678 secs 2024-07-26T11:05:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:17 smithi044 ceph-mon[32404]: pgmap v5077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:17 smithi116 ceph-mon[37097]: pgmap v5077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:17 smithi171 ceph-mon[37789]: pgmap v5077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:19 smithi044 ceph-mon[32404]: pgmap v5078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:19 smithi116 ceph-mon[37097]: pgmap v5078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:19 smithi171 ceph-mon[37789]: pgmap v5078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:21 smithi044 ceph-mon[32404]: pgmap v5079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8665.805825 secs 2024-07-26T11:05:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:21 smithi116 ceph-mon[37097]: pgmap v5079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8665.805825 secs 2024-07-26T11:05:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:21 smithi171 ceph-mon[37789]: pgmap v5079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8665.805825 secs 2024-07-26T11:05:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:23 smithi044 ceph-mon[32404]: pgmap v5080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:23 smithi116 ceph-mon[37097]: pgmap v5080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:23 smithi171 ceph-mon[37789]: pgmap v5080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:24 smithi171 ceph-mon[37789]: pgmap v5081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:24 smithi044 ceph-mon[32404]: pgmap v5081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:24 smithi116 ceph-mon[37097]: pgmap v5081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8670.805961 secs 2024-07-26T11:05:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8670.805961 secs 2024-07-26T11:05:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8670.805961 secs 2024-07-26T11:05:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:27 smithi171 ceph-mon[37789]: pgmap v5082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:27 smithi044 ceph-mon[32404]: pgmap v5082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:27 smithi116 ceph-mon[37097]: pgmap v5082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:28.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:28.235+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:05:28.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:28.235+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:05:28.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:28.235+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:29 smithi044 ceph-mon[32404]: pgmap v5083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:29 smithi116 ceph-mon[37097]: pgmap v5083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:29 smithi171 ceph-mon[37789]: pgmap v5083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:31 smithi044 ceph-mon[32404]: pgmap v5084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8675.806106 secs 2024-07-26T11:05:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:31 smithi116 ceph-mon[37097]: pgmap v5084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8675.806106 secs 2024-07-26T11:05:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:31 smithi171 ceph-mon[37789]: pgmap v5084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8675.806106 secs 2024-07-26T11:05:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:33 smithi044 ceph-mon[32404]: pgmap v5085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:33 smithi116 ceph-mon[37097]: pgmap v5085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:33 smithi171 ceph-mon[37789]: pgmap v5085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:35 smithi044 ceph-mon[32404]: pgmap v5086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:35.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:35 smithi116 ceph-mon[37097]: pgmap v5086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:35 smithi171 ceph-mon[37789]: pgmap v5086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8680.806281 secs 2024-07-26T11:05:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8680.806281 secs 2024-07-26T11:05:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8680.806281 secs 2024-07-26T11:05:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:37 smithi044 ceph-mon[32404]: pgmap v5087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:37 smithi116 ceph-mon[37097]: pgmap v5087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:37 smithi171 ceph-mon[37789]: pgmap v5087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:38 smithi171 ceph-mon[37789]: pgmap v5088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:38 smithi044 ceph-mon[32404]: pgmap v5088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:38 smithi116 ceph-mon[37097]: pgmap v5088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:41 smithi044 ceph-mon[32404]: pgmap v5089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8685.806415 secs 2024-07-26T11:05:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:41 smithi116 ceph-mon[37097]: pgmap v5089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8685.806415 secs 2024-07-26T11:05:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:41 smithi171 ceph-mon[37789]: pgmap v5089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8685.806415 secs 2024-07-26T11:05:43.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:43.239+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:05:43.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:43.239+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:05:43.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:43.239+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:43 smithi044 ceph-mon[32404]: pgmap v5090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:43 smithi116 ceph-mon[37097]: pgmap v5090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:43 smithi171 ceph-mon[37789]: pgmap v5090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:05:45.752 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:45 smithi116 ceph-mon[37097]: pgmap v5091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:45.752 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:45 smithi044 ceph-mon[32404]: pgmap v5091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:45 smithi171 ceph-mon[37789]: pgmap v5091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:05:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:46.762 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8690.806550 secs 2024-07-26T11:05:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8690.806550 secs 2024-07-26T11:05:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8690.806550 secs 2024-07-26T11:05:47.597 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:47 smithi116 ceph-mon[37097]: pgmap v5092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:47 smithi044 ceph-mon[32404]: pgmap v5092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:47 smithi171 ceph-mon[37789]: pgmap v5092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:49 smithi116 ceph-mon[37097]: pgmap v5093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:49 smithi171 ceph-mon[37789]: pgmap v5093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:49.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:49 smithi044 ceph-mon[32404]: pgmap v5093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:05:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: pgmap v5094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8695.806670 secs 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: pgmap v5094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8695.806670 secs 2024-07-26T11:05:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:51.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: pgmap v5094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8695.806670 secs 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:51.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:05:52.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:52 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:05:52.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:52 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:05:52.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:52 smithi044 ceph-mon[32404]: pgmap v5095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:05:52.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:52 smithi044 ceph-mon[32404]: pgmap v5096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:52 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:05:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:52 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:05:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:52 smithi116 ceph-mon[37097]: pgmap v5095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:05:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:52 smithi116 ceph-mon[37097]: pgmap v5096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:52 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:05:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:52 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:05:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:52 smithi171 ceph-mon[37789]: pgmap v5095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:05:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:52 smithi171 ceph-mon[37789]: pgmap v5096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:54.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:54 smithi044 ceph-mon[32404]: pgmap v5097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:05:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:54 smithi116 ceph-mon[37097]: pgmap v5097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:05:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:54 smithi171 ceph-mon[37789]: pgmap v5097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:05:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:56 smithi044 ceph-mon[32404]: pgmap v5098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:05:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8700.806811 secs 2024-07-26T11:05:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:56 smithi116 ceph-mon[37097]: pgmap v5098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:05:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8700.806811 secs 2024-07-26T11:05:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:56 smithi171 ceph-mon[37789]: pgmap v5098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:05:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8700.806811 secs 2024-07-26T11:05:58.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:58.248+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:05:58.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:58.248+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:58.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:05:58.248+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:05:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:05:58 smithi044 ceph-mon[32404]: pgmap v5099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:05:58 smithi116 ceph-mon[37097]: pgmap v5099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:05:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:05:58 smithi171 ceph-mon[37789]: pgmap v5099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:00 smithi044 ceph-mon[32404]: pgmap v5100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:06:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:00 smithi116 ceph-mon[37097]: pgmap v5100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:06:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:00 smithi171 ceph-mon[37789]: pgmap v5100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:06:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8705.806943 secs 2024-07-26T11:06:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8705.806943 secs 2024-07-26T11:06:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8705.806943 secs 2024-07-26T11:06:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:02 smithi171 ceph-mon[37789]: pgmap v5101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:06:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:02 smithi044 ceph-mon[32404]: pgmap v5101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:06:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:02 smithi116 ceph-mon[37097]: pgmap v5101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:06:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:03 smithi171 ceph-mon[37789]: pgmap v5102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:03 smithi044 ceph-mon[32404]: pgmap v5102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:03 smithi116 ceph-mon[37097]: pgmap v5102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:06 smithi044 ceph-mon[32404]: pgmap v5103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8710.807106 secs 2024-07-26T11:06:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:06 smithi116 ceph-mon[37097]: pgmap v5103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8710.807106 secs 2024-07-26T11:06:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:06 smithi171 ceph-mon[37789]: pgmap v5103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8710.807106 secs 2024-07-26T11:06:08.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:08 smithi044 ceph-mon[32404]: pgmap v5104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:08 smithi116 ceph-mon[37097]: pgmap v5104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:08 smithi171 ceph-mon[37789]: pgmap v5104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:10 smithi044 ceph-mon[32404]: pgmap v5105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:10 smithi116 ceph-mon[37097]: pgmap v5105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:10 smithi171 ceph-mon[37789]: pgmap v5105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8715.807271 secs 2024-07-26T11:06:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8715.807271 secs 2024-07-26T11:06:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8715.807271 secs 2024-07-26T11:06:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:12 smithi044 ceph-mon[32404]: pgmap v5106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:12 smithi116 ceph-mon[37097]: pgmap v5106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:12 smithi171 ceph-mon[37789]: pgmap v5106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:13.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:13.253+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:06:13.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:13.253+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:06:13.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:13.253+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:14 smithi044 ceph-mon[32404]: pgmap v5107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:14 smithi116 ceph-mon[37097]: pgmap v5107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:14 smithi171 ceph-mon[37789]: pgmap v5107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:16 smithi044 ceph-mon[32404]: pgmap v5108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8720.807444 secs 2024-07-26T11:06:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:16 smithi116 ceph-mon[37097]: pgmap v5108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8720.807444 secs 2024-07-26T11:06:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:16 smithi171 ceph-mon[37789]: pgmap v5108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8720.807444 secs 2024-07-26T11:06:18.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:18 smithi116 ceph-mon[37097]: pgmap v5109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:18 smithi171 ceph-mon[37789]: pgmap v5109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:18.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:18 smithi044 ceph-mon[32404]: pgmap v5109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:20 smithi171 ceph-mon[37789]: pgmap v5110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:20 smithi044 ceph-mon[32404]: pgmap v5110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:20 smithi116 ceph-mon[37097]: pgmap v5110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8725.807608 secs 2024-07-26T11:06:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:21 smithi171 ceph-mon[37789]: pgmap v5111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8725.807608 secs 2024-07-26T11:06:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:21 smithi044 ceph-mon[32404]: pgmap v5111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8725.807608 secs 2024-07-26T11:06:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:21 smithi116 ceph-mon[37097]: pgmap v5111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:24 smithi044 ceph-mon[32404]: pgmap v5112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:24 smithi116 ceph-mon[37097]: pgmap v5112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:24 smithi171 ceph-mon[37789]: pgmap v5112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:26 smithi044 ceph-mon[32404]: pgmap v5113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8730.807776 secs 2024-07-26T11:06:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:26 smithi116 ceph-mon[37097]: pgmap v5113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8730.807776 secs 2024-07-26T11:06:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:26 smithi171 ceph-mon[37789]: pgmap v5113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8730.807776 secs 2024-07-26T11:06:28.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:28.263+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:06:28.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:28.263+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:28.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:28.263+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:06:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:28 smithi044 ceph-mon[32404]: pgmap v5114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:28 smithi116 ceph-mon[37097]: pgmap v5114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:28 smithi171 ceph-mon[37789]: pgmap v5114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:30 smithi044 ceph-mon[32404]: pgmap v5115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:30 smithi116 ceph-mon[37097]: pgmap v5115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:30 smithi171 ceph-mon[37789]: pgmap v5115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8735.807946 secs 2024-07-26T11:06:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8735.807946 secs 2024-07-26T11:06:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8735.807946 secs 2024-07-26T11:06:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:32 smithi116 ceph-mon[37097]: pgmap v5116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:32 smithi171 ceph-mon[37789]: pgmap v5116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:32 smithi044 ceph-mon[32404]: pgmap v5116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:34 smithi171 ceph-mon[37789]: pgmap v5117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:34 smithi044 ceph-mon[32404]: pgmap v5117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:34 smithi116 ceph-mon[37097]: pgmap v5117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:35 smithi171 ceph-mon[37789]: pgmap v5118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:35 smithi044 ceph-mon[32404]: pgmap v5118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:35 smithi116 ceph-mon[37097]: pgmap v5118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8740.808094 secs 2024-07-26T11:06:36.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8740.808094 secs 2024-07-26T11:06:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8740.808094 secs 2024-07-26T11:06:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:37 smithi044 ceph-mon[32404]: pgmap v5119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:37 smithi116 ceph-mon[37097]: pgmap v5119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:37 smithi171 ceph-mon[37789]: pgmap v5119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:40 smithi044 ceph-mon[32404]: pgmap v5120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:40 smithi116 ceph-mon[37097]: pgmap v5120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:40 smithi171 ceph-mon[37789]: pgmap v5120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8745.808258 secs 2024-07-26T11:06:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8745.808258 secs 2024-07-26T11:06:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8745.808258 secs 2024-07-26T11:06:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:42 smithi044 ceph-mon[32404]: pgmap v5121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:42 smithi116 ceph-mon[37097]: pgmap v5121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:42 smithi171 ceph-mon[37789]: pgmap v5121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:43.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:43.268+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:06:43.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:43.268+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:43.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:43.268+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:06:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:44 smithi116 ceph-mon[37097]: pgmap v5122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:44 smithi171 ceph-mon[37789]: pgmap v5122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:44 smithi044 ceph-mon[32404]: pgmap v5122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:46 smithi116 ceph-mon[37097]: pgmap v5123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8750.808394 secs 2024-07-26T11:06:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:46 smithi171 ceph-mon[37789]: pgmap v5123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8750.808394 secs 2024-07-26T11:06:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:46 smithi044 ceph-mon[32404]: pgmap v5123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8750.808394 secs 2024-07-26T11:06:48.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:48 smithi044 ceph-mon[32404]: pgmap v5124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:48 smithi171 ceph-mon[37789]: pgmap v5124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:48 smithi116 ceph-mon[37097]: pgmap v5124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:06:50.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:50 smithi044 ceph-mon[32404]: pgmap v5125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:50 smithi116 ceph-mon[37097]: pgmap v5125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:50 smithi171 ceph-mon[37789]: pgmap v5125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:06:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8755.808555 secs 2024-07-26T11:06:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8755.808555 secs 2024-07-26T11:06:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8755.808555 secs 2024-07-26T11:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: pgmap v5126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: pgmap v5126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: pgmap v5126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:52.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:52.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.528 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.528 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.528 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:52.528 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:53 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:06:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:53 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:06:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:53 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:06:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:53 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:06:53.559 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:53 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:06:53.559 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:53 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:06:54.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:54 smithi171 ceph-mon[37789]: pgmap v5127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:54 smithi044 ceph-mon[32404]: pgmap v5127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:54 smithi116 ceph-mon[37097]: pgmap v5127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:06:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: pgmap v5128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8760.808721 secs 2024-07-26T11:06:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:56.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: pgmap v5128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8760.808721 secs 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:56.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:56.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: pgmap v5128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:06:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8760.808721 secs 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:56.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:06:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:57 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:06:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:57 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:06:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:57 smithi044 ceph-mon[32404]: pgmap v5129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:06:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:57 smithi044 ceph-mon[32404]: pgmap v5130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:57 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:06:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:57 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:06:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:57 smithi116 ceph-mon[37097]: pgmap v5129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:06:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:57 smithi116 ceph-mon[37097]: pgmap v5130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:57 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:06:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:57 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:06:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:57 smithi171 ceph-mon[37789]: pgmap v5129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:06:57.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:57 smithi171 ceph-mon[37789]: pgmap v5130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:58.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:58.271+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:06:58.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:58.271+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:06:58.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:06:58.271+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:06:59 smithi044 ceph-mon[32404]: pgmap v5131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:06:59 smithi116 ceph-mon[37097]: pgmap v5131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:06:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:06:59 smithi171 ceph-mon[37789]: pgmap v5131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:07:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:01 smithi116 ceph-mon[37097]: pgmap v5132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:07:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8765.808858 secs 2024-07-26T11:07:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:01 smithi171 ceph-mon[37789]: pgmap v5132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:07:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8765.808858 secs 2024-07-26T11:07:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:01 smithi044 ceph-mon[32404]: pgmap v5132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:07:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8765.808858 secs 2024-07-26T11:07:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:03 smithi171 ceph-mon[37789]: pgmap v5133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:07:03.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:03 smithi044 ceph-mon[32404]: pgmap v5133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:07:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:03 smithi116 ceph-mon[37097]: pgmap v5133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:07:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:05 smithi171 ceph-mon[37789]: pgmap v5134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:07:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:05 smithi044 ceph-mon[32404]: pgmap v5134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:07:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:05 smithi116 ceph-mon[37097]: pgmap v5134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:07:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8770.808998 secs 2024-07-26T11:07:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8770.808998 secs 2024-07-26T11:07:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8770.808998 secs 2024-07-26T11:07:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:07 smithi171 ceph-mon[37789]: pgmap v5135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:07 smithi044 ceph-mon[32404]: pgmap v5135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:07 smithi116 ceph-mon[37097]: pgmap v5135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:07:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:08 smithi171 ceph-mon[37789]: pgmap v5136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:08 smithi044 ceph-mon[32404]: pgmap v5136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:08 smithi116 ceph-mon[37097]: pgmap v5136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:10 smithi044 ceph-mon[32404]: pgmap v5137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:10 smithi116 ceph-mon[37097]: pgmap v5137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:10 smithi171 ceph-mon[37789]: pgmap v5137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8775.809163 secs 2024-07-26T11:07:12.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8775.809163 secs 2024-07-26T11:07:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8775.809163 secs 2024-07-26T11:07:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:12 smithi044 ceph-mon[32404]: pgmap v5138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:12 smithi116 ceph-mon[37097]: pgmap v5138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:12 smithi171 ceph-mon[37789]: pgmap v5138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:13.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:13.275+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:07:13.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:13.275+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:13.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:13.275+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:07:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:14 smithi044 ceph-mon[32404]: pgmap v5139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:14 smithi116 ceph-mon[37097]: pgmap v5139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:15.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:14 smithi171 ceph-mon[37789]: pgmap v5139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8780.809323 secs 2024-07-26T11:07:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:16 smithi044 ceph-mon[32404]: pgmap v5140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8780.809323 secs 2024-07-26T11:07:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:16 smithi116 ceph-mon[37097]: pgmap v5140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8780.809323 secs 2024-07-26T11:07:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:16 smithi171 ceph-mon[37789]: pgmap v5140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:19.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:18 smithi044 ceph-mon[32404]: pgmap v5141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:18 smithi116 ceph-mon[37097]: pgmap v5141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:18 smithi171 ceph-mon[37789]: pgmap v5141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:20 smithi044 ceph-mon[32404]: pgmap v5142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:20 smithi116 ceph-mon[37097]: pgmap v5142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:20 smithi171 ceph-mon[37789]: pgmap v5142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8785.809458 secs 2024-07-26T11:07:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8785.809458 secs 2024-07-26T11:07:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8785.809458 secs 2024-07-26T11:07:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:22 smithi116 ceph-mon[37097]: pgmap v5143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:22 smithi171 ceph-mon[37789]: pgmap v5143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:22 smithi044 ceph-mon[32404]: pgmap v5143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:24 smithi116 ceph-mon[37097]: pgmap v5144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:24 smithi171 ceph-mon[37789]: pgmap v5144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:24 smithi044 ceph-mon[32404]: pgmap v5144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8790.809662 secs 2024-07-26T11:07:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:26 smithi171 ceph-mon[37789]: pgmap v5145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8790.809662 secs 2024-07-26T11:07:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:26 smithi044 ceph-mon[32404]: pgmap v5145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8790.809662 secs 2024-07-26T11:07:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:26 smithi116 ceph-mon[37097]: pgmap v5145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:28.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:28.285+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:07:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:28.285+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:28.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:28.285+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:07:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:28 smithi171 ceph-mon[37789]: pgmap v5146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:28 smithi044 ceph-mon[32404]: pgmap v5146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:28 smithi116 ceph-mon[37097]: pgmap v5146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:30 smithi171 ceph-mon[37789]: pgmap v5147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:30 smithi044 ceph-mon[32404]: pgmap v5147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:30 smithi116 ceph-mon[37097]: pgmap v5147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8795.809751 secs 2024-07-26T11:07:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8795.809751 secs 2024-07-26T11:07:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8795.809751 secs 2024-07-26T11:07:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:32 smithi171 ceph-mon[37789]: pgmap v5148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:32 smithi044 ceph-mon[32404]: pgmap v5148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:32 smithi116 ceph-mon[37097]: pgmap v5148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:34 smithi171 ceph-mon[37789]: pgmap v5149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:34 smithi044 ceph-mon[32404]: pgmap v5149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:34 smithi116 ceph-mon[37097]: pgmap v5149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8800.809901 secs 2024-07-26T11:07:37.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:36 smithi171 ceph-mon[37789]: pgmap v5150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:37.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8800.809901 secs 2024-07-26T11:07:37.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:36 smithi044 ceph-mon[32404]: pgmap v5150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8800.809901 secs 2024-07-26T11:07:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:36 smithi116 ceph-mon[37097]: pgmap v5150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:38 smithi171 ceph-mon[37789]: pgmap v5151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:38 smithi044 ceph-mon[32404]: pgmap v5151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:38 smithi116 ceph-mon[37097]: pgmap v5151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:40 smithi171 ceph-mon[37789]: pgmap v5152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:40 smithi044 ceph-mon[32404]: pgmap v5152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:41.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:40 smithi116 ceph-mon[37097]: pgmap v5152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8805.810052 secs 2024-07-26T11:07:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8805.810052 secs 2024-07-26T11:07:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8805.810052 secs 2024-07-26T11:07:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:42 smithi044 ceph-mon[32404]: pgmap v5153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:42 smithi116 ceph-mon[37097]: pgmap v5153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:43.290+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:07:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:43.290+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:07:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:43.290+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:42 smithi171 ceph-mon[37789]: pgmap v5153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:44 smithi044 ceph-mon[32404]: pgmap v5154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:44 smithi116 ceph-mon[37097]: pgmap v5154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:44 smithi171 ceph-mon[37789]: pgmap v5154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8810.810213 secs 2024-07-26T11:07:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:46 smithi044 ceph-mon[32404]: pgmap v5155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:07:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8810.810213 secs 2024-07-26T11:07:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:46 smithi116 ceph-mon[37097]: pgmap v5155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:07:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8810.810213 secs 2024-07-26T11:07:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:46 smithi171 ceph-mon[37789]: pgmap v5155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:07:49.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:48 smithi044 ceph-mon[32404]: pgmap v5156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:48 smithi116 ceph-mon[37097]: pgmap v5156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:48 smithi171 ceph-mon[37789]: pgmap v5156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:51 smithi044 ceph-mon[32404]: pgmap v5157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8815.810360 secs 2024-07-26T11:07:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:51 smithi116 ceph-mon[37097]: pgmap v5157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8815.810360 secs 2024-07-26T11:07:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:51 smithi171 ceph-mon[37789]: pgmap v5157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:07:51.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8815.810360 secs 2024-07-26T11:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:53 smithi116 ceph-mon[37097]: pgmap v5158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:53 smithi171 ceph-mon[37789]: pgmap v5158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:53 smithi044 ceph-mon[32404]: pgmap v5158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:55 smithi171 ceph-mon[37789]: pgmap v5159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:55 smithi044 ceph-mon[32404]: pgmap v5159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:55 smithi116 ceph-mon[37097]: pgmap v5159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:07:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8820.810511 secs 2024-07-26T11:07:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8820.810511 secs 2024-07-26T11:07:56.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8820.810511 secs 2024-07-26T11:07:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:57 smithi171 ceph-mon[37789]: pgmap v5160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:07:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:57 smithi044 ceph-mon[32404]: pgmap v5160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:57.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:07:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:57 smithi116 ceph-mon[37097]: pgmap v5160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:07:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:57.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:07:58.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:58.299+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:07:58.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:58.299+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:07:58.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:07:58.299+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:58 smithi171 ceph-mon[37789]: pgmap v5161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:07:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:07:58 smithi171 ceph-mon[37789]: pgmap v5162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:07:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:58 smithi044 ceph-mon[32404]: pgmap v5161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:07:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:07:58 smithi044 ceph-mon[32404]: pgmap v5162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:07:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:58 smithi116 ceph-mon[37097]: pgmap v5161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:07:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:07:58 smithi116 ceph-mon[37097]: pgmap v5162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:00 smithi171 ceph-mon[37789]: pgmap v5163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:00 smithi044 ceph-mon[32404]: pgmap v5163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:00 smithi116 ceph-mon[37097]: pgmap v5163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8825.810673 secs 2024-07-26T11:08:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8825.810673 secs 2024-07-26T11:08:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8825.810673 secs 2024-07-26T11:08:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:02 smithi171 ceph-mon[37789]: pgmap v5164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:02 smithi044 ceph-mon[32404]: pgmap v5164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:02 smithi116 ceph-mon[37097]: pgmap v5164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:04 smithi171 ceph-mon[37789]: pgmap v5165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:04 smithi044 ceph-mon[32404]: pgmap v5165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:04 smithi116 ceph-mon[37097]: pgmap v5165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:06 smithi171 ceph-mon[37789]: pgmap v5166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8830.810783 secs 2024-07-26T11:08:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:06 smithi044 ceph-mon[32404]: pgmap v5166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8830.810783 secs 2024-07-26T11:08:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:06 smithi116 ceph-mon[37097]: pgmap v5166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:08:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8830.810783 secs 2024-07-26T11:08:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:08 smithi044 ceph-mon[32404]: pgmap v5167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:08:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:08 smithi116 ceph-mon[37097]: pgmap v5167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:08:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:08 smithi171 ceph-mon[37789]: pgmap v5167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:08:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:10 smithi044 ceph-mon[32404]: pgmap v5168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:10 smithi116 ceph-mon[37097]: pgmap v5168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:10 smithi171 ceph-mon[37789]: pgmap v5168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8835.810895 secs 2024-07-26T11:08:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8835.810895 secs 2024-07-26T11:08:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8835.810895 secs 2024-07-26T11:08:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:12 smithi044 ceph-mon[32404]: pgmap v5169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:12 smithi116 ceph-mon[37097]: pgmap v5169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:12 smithi171 ceph-mon[37789]: pgmap v5169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:13.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:13.305+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:08:13.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:13.305+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:13.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:13.305+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:08:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:14 smithi044 ceph-mon[32404]: pgmap v5170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:14 smithi116 ceph-mon[37097]: pgmap v5170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:14 smithi171 ceph-mon[37789]: pgmap v5170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:16.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:16.254+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:08:16.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:16.254+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:08:16.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:16.255+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:08:16.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:16.255+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:08:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:16 smithi044 ceph-mon[32404]: pgmap v5171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8840.811065 secs 2024-07-26T11:08:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:16 smithi116 ceph-mon[37097]: pgmap v5171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8840.811065 secs 2024-07-26T11:08:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:16 smithi171 ceph-mon[37789]: pgmap v5171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8840.811065 secs 2024-07-26T11:08:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:18 smithi044 ceph-mon[32404]: pgmap v5172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:18 smithi116 ceph-mon[37097]: pgmap v5172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:18 smithi171 ceph-mon[37789]: pgmap v5172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:20 smithi044 ceph-mon[32404]: pgmap v5173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:20 smithi116 ceph-mon[37097]: pgmap v5173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:20 smithi171 ceph-mon[37789]: pgmap v5173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8845.811215 secs 2024-07-26T11:08:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8845.811215 secs 2024-07-26T11:08:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8845.811215 secs 2024-07-26T11:08:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:22 smithi044 ceph-mon[32404]: pgmap v5174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:22 smithi116 ceph-mon[37097]: pgmap v5174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:22 smithi171 ceph-mon[37789]: pgmap v5174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:24 smithi044 ceph-mon[32404]: pgmap v5175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:24 smithi116 ceph-mon[37097]: pgmap v5175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:24.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:24 smithi171 ceph-mon[37789]: pgmap v5175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:26 smithi044 ceph-mon[32404]: pgmap v5176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8850.811366 secs 2024-07-26T11:08:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:26 smithi116 ceph-mon[37097]: pgmap v5176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8850.811366 secs 2024-07-26T11:08:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:26 smithi171 ceph-mon[37789]: pgmap v5176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8850.811366 secs 2024-07-26T11:08:28.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:28.315+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:08:28.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:28.315+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:08:28.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:28.315+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:28 smithi044 ceph-mon[32404]: pgmap v5177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:28 smithi116 ceph-mon[37097]: pgmap v5177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:28 smithi171 ceph-mon[37789]: pgmap v5177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:30 smithi044 ceph-mon[32404]: pgmap v5178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:30 smithi116 ceph-mon[37097]: pgmap v5178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:30 smithi171 ceph-mon[37789]: pgmap v5178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8855.811527 secs 2024-07-26T11:08:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8855.811527 secs 2024-07-26T11:08:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8855.811527 secs 2024-07-26T11:08:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:32 smithi044 ceph-mon[32404]: pgmap v5179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:32 smithi116 ceph-mon[37097]: pgmap v5179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:32 smithi171 ceph-mon[37789]: pgmap v5179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:34 smithi044 ceph-mon[32404]: pgmap v5180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:34 smithi116 ceph-mon[37097]: pgmap v5180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:34 smithi171 ceph-mon[37789]: pgmap v5180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:08:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:36 smithi044 ceph-mon[32404]: pgmap v5181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:08:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8860.811673 secs 2024-07-26T11:08:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:36 smithi116 ceph-mon[37097]: pgmap v5181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:08:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8860.811673 secs 2024-07-26T11:08:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:36 smithi171 ceph-mon[37789]: pgmap v5181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:08:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8860.811673 secs 2024-07-26T11:08:38.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:38 smithi044 ceph-mon[32404]: pgmap v5182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:38 smithi116 ceph-mon[37097]: pgmap v5182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:38 smithi171 ceph-mon[37789]: pgmap v5182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:40 smithi044 ceph-mon[32404]: pgmap v5183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:40 smithi116 ceph-mon[37097]: pgmap v5183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:40 smithi171 ceph-mon[37789]: pgmap v5183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8865.811836 secs 2024-07-26T11:08:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8865.811836 secs 2024-07-26T11:08:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8865.811836 secs 2024-07-26T11:08:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:42 smithi044 ceph-mon[32404]: pgmap v5184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:08:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:42 smithi116 ceph-mon[37097]: pgmap v5184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:08:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:42 smithi171 ceph-mon[37789]: pgmap v5184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:08:43.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:43.318+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:08:43.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:43.318+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:08:43.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:43.318+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:44 smithi044 ceph-mon[32404]: pgmap v5185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:08:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:44 smithi116 ceph-mon[37097]: pgmap v5185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:08:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:44 smithi171 ceph-mon[37789]: pgmap v5185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:08:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:46 smithi044 ceph-mon[32404]: pgmap v5186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8870.812002 secs 2024-07-26T11:08:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:46 smithi116 ceph-mon[37097]: pgmap v5186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8870.812002 secs 2024-07-26T11:08:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:46 smithi171 ceph-mon[37789]: pgmap v5186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8870.812002 secs 2024-07-26T11:08:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:48 smithi044 ceph-mon[32404]: pgmap v5187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:08:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:48 smithi116 ceph-mon[37097]: pgmap v5187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:08:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:48 smithi171 ceph-mon[37789]: pgmap v5187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:08:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:50 smithi044 ceph-mon[32404]: pgmap v5188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:08:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:50 smithi116 ceph-mon[37097]: pgmap v5188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:08:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:50 smithi171 ceph-mon[37789]: pgmap v5188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:08:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8875.812131 secs 2024-07-26T11:08:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8875.812131 secs 2024-07-26T11:08:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8875.812131 secs 2024-07-26T11:08:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:52 smithi171 ceph-mon[37789]: pgmap v5189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:52 smithi044 ceph-mon[32404]: pgmap v5189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:52 smithi116 ceph-mon[37097]: pgmap v5189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:08:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:54 smithi171 ceph-mon[37789]: pgmap v5190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:08:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:54 smithi044 ceph-mon[32404]: pgmap v5190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:08:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:54 smithi116 ceph-mon[37097]: pgmap v5190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:08:56.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:56 smithi116 ceph-mon[37097]: pgmap v5191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:56.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8880.812313 secs 2024-07-26T11:08:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:56 smithi171 ceph-mon[37789]: pgmap v5191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8880.812313 secs 2024-07-26T11:08:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:56 smithi044 ceph-mon[32404]: pgmap v5191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:08:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8880.812313 secs 2024-07-26T11:08:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:58.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:58.327+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:08:58.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:58.327+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:08:58.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:08:58.327+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:58 smithi044 ceph-mon[32404]: pgmap v5192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:08:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:08:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:58 smithi171 ceph-mon[37789]: pgmap v5192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:08:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:08:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:58 smithi116 ceph-mon[37097]: pgmap v5192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:08:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:08:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:08:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:08:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:09:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:00.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:08:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:09:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:08:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:09:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:00 smithi171 ceph-mon[37789]: pgmap v5193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:00 smithi171 ceph-mon[37789]: pgmap v5194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:00 smithi044 ceph-mon[32404]: pgmap v5193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:00 smithi044 ceph-mon[32404]: pgmap v5194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:00 smithi116 ceph-mon[37097]: pgmap v5193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:00 smithi116 ceph-mon[37097]: pgmap v5194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:09:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8885.812480 secs 2024-07-26T11:09:01.856 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:01 smithi171 ceph-mon[37789]: pgmap v5195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8885.812480 secs 2024-07-26T11:09:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:01 smithi044 ceph-mon[32404]: pgmap v5195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8885.812480 secs 2024-07-26T11:09:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:01 smithi116 ceph-mon[37097]: pgmap v5195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:03 smithi044 ceph-mon[32404]: pgmap v5196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T11:09:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:03 smithi116 ceph-mon[37097]: pgmap v5196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T11:09:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:03 smithi171 ceph-mon[37789]: pgmap v5196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T11:09:06.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:05 smithi171 ceph-mon[37789]: pgmap v5197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:05 smithi044 ceph-mon[32404]: pgmap v5197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:05 smithi116 ceph-mon[37097]: pgmap v5197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8890.812682 secs 2024-07-26T11:09:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8890.812682 secs 2024-07-26T11:09:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8890.812682 secs 2024-07-26T11:09:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:07 smithi044 ceph-mon[32404]: pgmap v5198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:07 smithi116 ceph-mon[37097]: pgmap v5198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:07 smithi171 ceph-mon[37789]: pgmap v5198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:09:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:09 smithi044 ceph-mon[32404]: pgmap v5199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:09 smithi116 ceph-mon[37097]: pgmap v5199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:09 smithi171 ceph-mon[37789]: pgmap v5199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:09:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8895.812847 secs 2024-07-26T11:09:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:11 smithi044 ceph-mon[32404]: pgmap v5200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8895.812847 secs 2024-07-26T11:09:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:11 smithi116 ceph-mon[37097]: pgmap v5200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8895.812847 secs 2024-07-26T11:09:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:11 smithi171 ceph-mon[37789]: pgmap v5200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:13.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:13.331+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:09:13.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:13.331+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:09:13.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:13.331+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:13 smithi044 ceph-mon[32404]: pgmap v5201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:13 smithi116 ceph-mon[37097]: pgmap v5201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:13 smithi171 ceph-mon[37789]: pgmap v5201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:15 smithi044 ceph-mon[32404]: pgmap v5202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:15 smithi116 ceph-mon[37097]: pgmap v5202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:15 smithi171 ceph-mon[37789]: pgmap v5202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8900.812951 secs 2024-07-26T11:09:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8900.812951 secs 2024-07-26T11:09:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8900.812951 secs 2024-07-26T11:09:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:17 smithi044 ceph-mon[32404]: pgmap v5203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:17 smithi116 ceph-mon[37097]: pgmap v5203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:17 smithi171 ceph-mon[37789]: pgmap v5203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:19 smithi044 ceph-mon[32404]: pgmap v5204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:19 smithi116 ceph-mon[37097]: pgmap v5204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:19 smithi171 ceph-mon[37789]: pgmap v5204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8905.813087 secs 2024-07-26T11:09:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:21 smithi044 ceph-mon[32404]: pgmap v5205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8905.813087 secs 2024-07-26T11:09:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:21 smithi116 ceph-mon[37097]: pgmap v5205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8905.813087 secs 2024-07-26T11:09:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:21 smithi171 ceph-mon[37789]: pgmap v5205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:23 smithi044 ceph-mon[32404]: pgmap v5206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:23 smithi116 ceph-mon[37097]: pgmap v5206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:23 smithi171 ceph-mon[37789]: pgmap v5206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:26.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:25 smithi044 ceph-mon[32404]: pgmap v5207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:25 smithi116 ceph-mon[37097]: pgmap v5207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:25 smithi171 ceph-mon[37789]: pgmap v5207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8910.813256 secs 2024-07-26T11:09:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8910.813256 secs 2024-07-26T11:09:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8910.813256 secs 2024-07-26T11:09:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:27 smithi044 ceph-mon[32404]: pgmap v5208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:27 smithi116 ceph-mon[37097]: pgmap v5208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:27 smithi171 ceph-mon[37789]: pgmap v5208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:28.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:28.341+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:09:28.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:28.341+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:28.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:28.341+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:09:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:30 smithi171 ceph-mon[37789]: pgmap v5209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:30 smithi044 ceph-mon[32404]: pgmap v5209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:30 smithi116 ceph-mon[37097]: pgmap v5209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8915.813410 secs 2024-07-26T11:09:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8915.813410 secs 2024-07-26T11:09:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8915.813410 secs 2024-07-26T11:09:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:32 smithi171 ceph-mon[37789]: pgmap v5210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:32 smithi044 ceph-mon[32404]: pgmap v5210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:32 smithi116 ceph-mon[37097]: pgmap v5210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:34 smithi171 ceph-mon[37789]: pgmap v5211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:34 smithi044 ceph-mon[32404]: pgmap v5211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:34 smithi116 ceph-mon[37097]: pgmap v5211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:36 smithi044 ceph-mon[32404]: pgmap v5212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8920.813584 secs 2024-07-26T11:09:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:36 smithi116 ceph-mon[37097]: pgmap v5212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8920.813584 secs 2024-07-26T11:09:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:36 smithi171 ceph-mon[37789]: pgmap v5212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8920.813584 secs 2024-07-26T11:09:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:38 smithi044 ceph-mon[32404]: pgmap v5213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:38 smithi116 ceph-mon[37097]: pgmap v5213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:38 smithi171 ceph-mon[37789]: pgmap v5213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:40.643 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:40 smithi044 ceph-mon[32404]: pgmap v5214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:40.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:40 smithi116 ceph-mon[37097]: pgmap v5214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:40 smithi171 ceph-mon[37789]: pgmap v5214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8925.813757 secs 2024-07-26T11:09:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8925.813757 secs 2024-07-26T11:09:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8925.813757 secs 2024-07-26T11:09:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:42 smithi044 ceph-mon[32404]: pgmap v5215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:42 smithi116 ceph-mon[37097]: pgmap v5215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:42 smithi171 ceph-mon[37789]: pgmap v5215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:43.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:43.346+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:09:43.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:43.346+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:43.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:43.346+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:09:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:44 smithi044 ceph-mon[32404]: pgmap v5216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:44 smithi116 ceph-mon[37097]: pgmap v5216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:44 smithi171 ceph-mon[37789]: pgmap v5216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:46.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:46.271+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:09:46.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:46.271+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:09:46.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:46.272+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:09:46.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:46.272+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:09:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:46 smithi044 ceph-mon[32404]: pgmap v5217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8930.813918 secs 2024-07-26T11:09:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:46 smithi116 ceph-mon[37097]: pgmap v5217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8930.813918 secs 2024-07-26T11:09:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:46 smithi171 ceph-mon[37789]: pgmap v5217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8930.813918 secs 2024-07-26T11:09:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:48 smithi044 ceph-mon[32404]: pgmap v5218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:48 smithi116 ceph-mon[37097]: pgmap v5218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:48 smithi171 ceph-mon[37789]: pgmap v5218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:50 smithi044 ceph-mon[32404]: pgmap v5219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:50.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:50 smithi116 ceph-mon[37097]: pgmap v5219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:50 smithi171 ceph-mon[37789]: pgmap v5219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8935.814073 secs 2024-07-26T11:09:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8935.814073 secs 2024-07-26T11:09:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8935.814073 secs 2024-07-26T11:09:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:52 smithi044 ceph-mon[32404]: pgmap v5220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:52 smithi116 ceph-mon[37097]: pgmap v5220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:52 smithi171 ceph-mon[37789]: pgmap v5220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:09:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:54 smithi044 ceph-mon[32404]: pgmap v5221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:54 smithi116 ceph-mon[37097]: pgmap v5221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:54 smithi171 ceph-mon[37789]: pgmap v5221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:56 smithi044 ceph-mon[32404]: pgmap v5222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8940.814212 secs 2024-07-26T11:09:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:56 smithi116 ceph-mon[37097]: pgmap v5222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8940.814212 secs 2024-07-26T11:09:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:56 smithi171 ceph-mon[37789]: pgmap v5222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:09:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8940.814212 secs 2024-07-26T11:09:58.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:58.348+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:09:58.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:58.348+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:09:58.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:09:58.348+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:58 smithi044 ceph-mon[32404]: pgmap v5223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:58 smithi116 ceph-mon[37097]: pgmap v5223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:58 smithi171 ceph-mon[37789]: pgmap v5223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:09:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:09:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:09:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:09:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:00 smithi044 ceph-mon[32404]: pgmap v5224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:00 smithi116 ceph-mon[37097]: pgmap v5224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:00 smithi171 ceph-mon[37789]: pgmap v5224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:10:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8945.814357 secs 2024-07-26T11:10:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8945.814357 secs 2024-07-26T11:10:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8945.814357 secs 2024-07-26T11:10:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:02 smithi044 ceph-mon[32404]: pgmap v5225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:02 smithi116 ceph-mon[37097]: pgmap v5225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:02 smithi171 ceph-mon[37789]: pgmap v5225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:04.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:04.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:03 smithi044 ceph-mon[32404]: pgmap v5226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:03 smithi116 ceph-mon[37097]: pgmap v5226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:03 smithi171 ceph-mon[37789]: pgmap v5226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:04 smithi116 ceph-mon[37097]: pgmap v5227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:04 smithi116 ceph-mon[37097]: pgmap v5228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:05.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:05.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:04 smithi171 ceph-mon[37789]: pgmap v5227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:05.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:04 smithi171 ceph-mon[37789]: pgmap v5228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:05.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:04 smithi044 ceph-mon[32404]: pgmap v5227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:10:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:04 smithi044 ceph-mon[32404]: pgmap v5228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:10:07.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8950.814501 secs 2024-07-26T11:10:07.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:06 smithi171 ceph-mon[37789]: pgmap v5229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:10:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8950.814501 secs 2024-07-26T11:10:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:06 smithi044 ceph-mon[32404]: pgmap v5229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:10:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8950.814501 secs 2024-07-26T11:10:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:06 smithi116 ceph-mon[37097]: pgmap v5229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:10:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:08 smithi044 ceph-mon[32404]: pgmap v5230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:10:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:08 smithi116 ceph-mon[37097]: pgmap v5230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:10:09.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:08 smithi171 ceph-mon[37789]: pgmap v5230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:10:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:11 smithi044 ceph-mon[32404]: pgmap v5231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8955.814694 secs 2024-07-26T11:10:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:11 smithi116 ceph-mon[37097]: pgmap v5231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8955.814694 secs 2024-07-26T11:10:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:11 smithi171 ceph-mon[37789]: pgmap v5231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8955.814694 secs 2024-07-26T11:10:13.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:13.351+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:10:13.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:13.351+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:10:13.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:13.351+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:13 smithi044 ceph-mon[32404]: pgmap v5232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:13 smithi116 ceph-mon[37097]: pgmap v5232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:13 smithi171 ceph-mon[37789]: pgmap v5232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:10:15.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:15 smithi044 ceph-mon[32404]: pgmap v5233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:10:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:15 smithi116 ceph-mon[37097]: pgmap v5233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:10:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:15 smithi171 ceph-mon[37789]: pgmap v5233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:10:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8960.814874 secs 2024-07-26T11:10:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8960.814874 secs 2024-07-26T11:10:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8960.814874 secs 2024-07-26T11:10:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:17 smithi044 ceph-mon[32404]: pgmap v5234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:17 smithi116 ceph-mon[37097]: pgmap v5234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:17 smithi171 ceph-mon[37789]: pgmap v5234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:19 smithi044 ceph-mon[32404]: pgmap v5235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:19 smithi116 ceph-mon[37097]: pgmap v5235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:19 smithi171 ceph-mon[37789]: pgmap v5235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:21 smithi044 ceph-mon[32404]: pgmap v5236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8965.815055 secs 2024-07-26T11:10:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:21 smithi116 ceph-mon[37097]: pgmap v5236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8965.815055 secs 2024-07-26T11:10:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:21 smithi171 ceph-mon[37789]: pgmap v5236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8965.815055 secs 2024-07-26T11:10:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:23 smithi044 ceph-mon[32404]: pgmap v5237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:23 smithi116 ceph-mon[37097]: pgmap v5237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:23 smithi171 ceph-mon[37789]: pgmap v5237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:25 smithi044 ceph-mon[32404]: pgmap v5238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:25 smithi116 ceph-mon[37097]: pgmap v5238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:25 smithi171 ceph-mon[37789]: pgmap v5238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8970.815221 secs 2024-07-26T11:10:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8970.815221 secs 2024-07-26T11:10:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8970.815221 secs 2024-07-26T11:10:27.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:27 smithi044 ceph-mon[32404]: pgmap v5239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:27.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:27 smithi116 ceph-mon[37097]: pgmap v5239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:27 smithi171 ceph-mon[37789]: pgmap v5239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:28.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:28.361+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:10:28.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:28.361+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:10:28.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:28.361+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:29 smithi044 ceph-mon[32404]: pgmap v5240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:29 smithi116 ceph-mon[37097]: pgmap v5240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:29 smithi171 ceph-mon[37789]: pgmap v5240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:31 smithi044 ceph-mon[32404]: pgmap v5241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8975.815391 secs 2024-07-26T11:10:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:31 smithi116 ceph-mon[37097]: pgmap v5241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8975.815391 secs 2024-07-26T11:10:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:31 smithi171 ceph-mon[37789]: pgmap v5241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8975.815391 secs 2024-07-26T11:10:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:33 smithi044 ceph-mon[32404]: pgmap v5242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:33 smithi116 ceph-mon[37097]: pgmap v5242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:33 smithi171 ceph-mon[37789]: pgmap v5242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:35 smithi044 ceph-mon[32404]: pgmap v5243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:35 smithi116 ceph-mon[37097]: pgmap v5243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:35 smithi171 ceph-mon[37789]: pgmap v5243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:36.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8980.815571 secs 2024-07-26T11:10:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8980.815571 secs 2024-07-26T11:10:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8980.815571 secs 2024-07-26T11:10:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:37 smithi044 ceph-mon[32404]: pgmap v5244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:10:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:37 smithi116 ceph-mon[37097]: pgmap v5244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:10:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:37 smithi171 ceph-mon[37789]: pgmap v5244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:10:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:39 smithi044 ceph-mon[32404]: pgmap v5245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:39 smithi116 ceph-mon[37097]: pgmap v5245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:39 smithi171 ceph-mon[37789]: pgmap v5245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:41 smithi044 ceph-mon[32404]: pgmap v5246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8985.815741 secs 2024-07-26T11:10:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:41 smithi116 ceph-mon[37097]: pgmap v5246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8985.815741 secs 2024-07-26T11:10:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:41 smithi171 ceph-mon[37789]: pgmap v5246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:10:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8985.815741 secs 2024-07-26T11:10:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:43.365+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:10:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:43.365+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:10:43.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:43.365+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:43 smithi044 ceph-mon[32404]: pgmap v5247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:43 smithi116 ceph-mon[37097]: pgmap v5247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:43 smithi171 ceph-mon[37789]: pgmap v5247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:45 smithi044 ceph-mon[32404]: pgmap v5248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:45 smithi116 ceph-mon[37097]: pgmap v5248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:45 smithi171 ceph-mon[37789]: pgmap v5248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8990.815835 secs 2024-07-26T11:10:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8990.815835 secs 2024-07-26T11:10:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8990.815835 secs 2024-07-26T11:10:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:47 smithi044 ceph-mon[32404]: pgmap v5249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:10:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:47 smithi116 ceph-mon[37097]: pgmap v5249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:10:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:47 smithi171 ceph-mon[37789]: pgmap v5249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:10:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:49 smithi044 ceph-mon[32404]: pgmap v5250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:49 smithi116 ceph-mon[37097]: pgmap v5250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:49 smithi171 ceph-mon[37789]: pgmap v5250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:51 smithi044 ceph-mon[32404]: pgmap v5251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:10:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 8995.816017 secs 2024-07-26T11:10:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:51 smithi116 ceph-mon[37097]: pgmap v5251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:10:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 8995.816017 secs 2024-07-26T11:10:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:51 smithi171 ceph-mon[37789]: pgmap v5251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:10:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 8995.816017 secs 2024-07-26T11:10:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:53 smithi044 ceph-mon[32404]: pgmap v5252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:53 smithi116 ceph-mon[37097]: pgmap v5252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:53 smithi171 ceph-mon[37789]: pgmap v5252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:10:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:55 smithi044 ceph-mon[32404]: pgmap v5253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:55 smithi116 ceph-mon[37097]: pgmap v5253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:55 smithi171 ceph-mon[37789]: pgmap v5253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:10:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9000.816194 secs 2024-07-26T11:10:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9000.816194 secs 2024-07-26T11:10:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9000.816194 secs 2024-07-26T11:10:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:57 smithi171 ceph-mon[37789]: pgmap v5254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:57.954 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:57 smithi044 ceph-mon[32404]: pgmap v5254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:57 smithi116 ceph-mon[37097]: pgmap v5254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:10:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:58.372+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:10:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:58.372+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:10:58.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:10:58.372+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:10:59 smithi171 ceph-mon[37789]: pgmap v5255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:10:59 smithi044 ceph-mon[32404]: pgmap v5255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:00.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:10:59 smithi116 ceph-mon[37097]: pgmap v5255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:01 smithi171 ceph-mon[37789]: pgmap v5256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9005.816363 secs 2024-07-26T11:11:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:01 smithi044 ceph-mon[32404]: pgmap v5256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9005.816363 secs 2024-07-26T11:11:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:01 smithi116 ceph-mon[37097]: pgmap v5256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:11:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9005.816363 secs 2024-07-26T11:11:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:03 smithi171 ceph-mon[37789]: pgmap v5257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:03 smithi044 ceph-mon[32404]: pgmap v5257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:03 smithi116 ceph-mon[37097]: pgmap v5257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:05 smithi171 ceph-mon[37789]: pgmap v5258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:11:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:05 smithi044 ceph-mon[32404]: pgmap v5258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:11:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:05 smithi116 ceph-mon[37097]: pgmap v5258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:11:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:06 smithi171 ceph-mon[37789]: pgmap v5259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:11:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:06 smithi171 ceph-mon[37789]: pgmap v5260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:06.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9010.816527 secs 2024-07-26T11:11:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:06 smithi044 ceph-mon[32404]: pgmap v5259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:11:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:06 smithi044 ceph-mon[32404]: pgmap v5260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9010.816527 secs 2024-07-26T11:11:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:06 smithi116 ceph-mon[37097]: pgmap v5259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:11:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:06 smithi116 ceph-mon[37097]: pgmap v5260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9010.816527 secs 2024-07-26T11:11:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:08 smithi171 ceph-mon[37789]: pgmap v5261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:08 smithi044 ceph-mon[32404]: pgmap v5261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:08 smithi116 ceph-mon[37097]: pgmap v5261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:10 smithi171 ceph-mon[37789]: pgmap v5262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:10 smithi044 ceph-mon[32404]: pgmap v5262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:10 smithi116 ceph-mon[37097]: pgmap v5262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9015.816645 secs 2024-07-26T11:11:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:11 smithi044 ceph-mon[32404]: pgmap v5263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:11:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9015.816645 secs 2024-07-26T11:11:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:11 smithi116 ceph-mon[37097]: pgmap v5263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:11:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9015.816645 secs 2024-07-26T11:11:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:11 smithi171 ceph-mon[37789]: pgmap v5263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:11:13.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:13.375+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:11:13.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:13.375+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:11:13.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:13.375+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:14 smithi044 ceph-mon[32404]: pgmap v5264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:14 smithi116 ceph-mon[37097]: pgmap v5264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:14 smithi171 ceph-mon[37789]: pgmap v5264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58072000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58072000 comp rx=0 tx=0).stop 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 msgr2=0x7fcf540c18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540b5840 0x7fcf540c18d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:14.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:14.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.713+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:14.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:14.715+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58074070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:16.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:16.287+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:11:16.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:16.287+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:11:16.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:16.287+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:11:16.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:16.287+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:11:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:16 smithi044 ceph-mon[32404]: pgmap v5265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:11:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9020.816787 secs 2024-07-26T11:11:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:16 smithi116 ceph-mon[37097]: pgmap v5265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:11:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9020.816787 secs 2024-07-26T11:11:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:16 smithi171 ceph-mon[37789]: pgmap v5265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:11:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9020.816787 secs 2024-07-26T11:11:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:18 smithi044 ceph-mon[32404]: pgmap v5266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:18.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:18 smithi116 ceph-mon[37097]: pgmap v5266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:18 smithi171 ceph-mon[37789]: pgmap v5266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:20 smithi044 ceph-mon[32404]: pgmap v5267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:20 smithi116 ceph-mon[37097]: pgmap v5267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:20 smithi171 ceph-mon[37789]: pgmap v5267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9025.816934 secs 2024-07-26T11:11:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9025.816934 secs 2024-07-26T11:11:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9025.816934 secs 2024-07-26T11:11:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:22 smithi044 ceph-mon[32404]: pgmap v5268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:22 smithi116 ceph-mon[37097]: pgmap v5268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:22 smithi171 ceph-mon[37789]: pgmap v5268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:24 smithi044 ceph-mon[32404]: pgmap v5269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:24.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:24 smithi116 ceph-mon[37097]: pgmap v5269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:24 smithi171 ceph-mon[37789]: pgmap v5269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:26 smithi044 ceph-mon[32404]: pgmap v5270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9030.817086 secs 2024-07-26T11:11:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:26 smithi116 ceph-mon[37097]: pgmap v5270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9030.817086 secs 2024-07-26T11:11:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:26 smithi171 ceph-mon[37789]: pgmap v5270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9030.817086 secs 2024-07-26T11:11:28.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:28.382+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:11:28.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:28.382+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:28.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:28.382+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:11:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:28 smithi044 ceph-mon[32404]: pgmap v5271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:28.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:28 smithi116 ceph-mon[37097]: pgmap v5271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:28 smithi171 ceph-mon[37789]: pgmap v5271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:30 smithi044 ceph-mon[32404]: pgmap v5272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:30 smithi116 ceph-mon[37097]: pgmap v5272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:30 smithi171 ceph-mon[37789]: pgmap v5272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9035.817238 secs 2024-07-26T11:11:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9035.817238 secs 2024-07-26T11:11:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9035.817238 secs 2024-07-26T11:11:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:32 smithi044 ceph-mon[32404]: pgmap v5273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:32 smithi116 ceph-mon[37097]: pgmap v5273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:32 smithi171 ceph-mon[37789]: pgmap v5273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:34 smithi044 ceph-mon[32404]: pgmap v5274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:34.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:34 smithi116 ceph-mon[37097]: pgmap v5274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:34 smithi171 ceph-mon[37789]: pgmap v5274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:36 smithi044 ceph-mon[32404]: pgmap v5275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9040.817390 secs 2024-07-26T11:11:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:36 smithi116 ceph-mon[37097]: pgmap v5275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9040.817390 secs 2024-07-26T11:11:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:36 smithi171 ceph-mon[37789]: pgmap v5275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9040.817390 secs 2024-07-26T11:11:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:38 smithi044 ceph-mon[32404]: pgmap v5276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:38 smithi116 ceph-mon[37097]: pgmap v5276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:38 smithi171 ceph-mon[37789]: pgmap v5276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:40 smithi044 ceph-mon[32404]: pgmap v5277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:40 smithi116 ceph-mon[37097]: pgmap v5277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:40 smithi171 ceph-mon[37789]: pgmap v5277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9045.817537 secs 2024-07-26T11:11:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9045.817537 secs 2024-07-26T11:11:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9045.817537 secs 2024-07-26T11:11:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:42 smithi044 ceph-mon[32404]: pgmap v5278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:42 smithi116 ceph-mon[37097]: pgmap v5278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:42 smithi171 ceph-mon[37789]: pgmap v5278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:43.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:43.386+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:11:43.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:43.386+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:43.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:43.386+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:11:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:44 smithi044 ceph-mon[32404]: pgmap v5279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:44 smithi116 ceph-mon[37097]: pgmap v5279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:44 smithi171 ceph-mon[37789]: pgmap v5279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:46 smithi044 ceph-mon[32404]: pgmap v5280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9050.817713 secs 2024-07-26T11:11:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:46 smithi116 ceph-mon[37097]: pgmap v5280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9050.817713 secs 2024-07-26T11:11:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:46 smithi171 ceph-mon[37789]: pgmap v5280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9050.817713 secs 2024-07-26T11:11:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:48 smithi044 ceph-mon[32404]: pgmap v5281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:48 smithi116 ceph-mon[37097]: pgmap v5281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:48.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:48 smithi171 ceph-mon[37789]: pgmap v5281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:50 smithi044 ceph-mon[32404]: pgmap v5282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:50 smithi116 ceph-mon[37097]: pgmap v5282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:50 smithi171 ceph-mon[37789]: pgmap v5282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9055.817875 secs 2024-07-26T11:11:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9055.817875 secs 2024-07-26T11:11:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9055.817875 secs 2024-07-26T11:11:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:52 smithi171 ceph-mon[37789]: pgmap v5283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:52 smithi044 ceph-mon[32404]: pgmap v5283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:52 smithi116 ceph-mon[37097]: pgmap v5283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:54 smithi171 ceph-mon[37789]: pgmap v5284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:54 smithi044 ceph-mon[32404]: pgmap v5284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:54 smithi116 ceph-mon[37097]: pgmap v5284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:11:56.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:56 smithi116 ceph-mon[37097]: pgmap v5285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:56.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9060.818019 secs 2024-07-26T11:11:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:56 smithi171 ceph-mon[37789]: pgmap v5285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9060.818019 secs 2024-07-26T11:11:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:56 smithi044 ceph-mon[32404]: pgmap v5285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:11:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9060.818019 secs 2024-07-26T11:11:58.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:58.393+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:11:58.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:58.393+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:58.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:11:58.393+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:11:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:11:58 smithi171 ceph-mon[37789]: pgmap v5286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:11:58 smithi044 ceph-mon[32404]: pgmap v5286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:11:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:11:58 smithi116 ceph-mon[37097]: pgmap v5286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:00 smithi171 ceph-mon[37789]: pgmap v5287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:00 smithi044 ceph-mon[32404]: pgmap v5287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:00.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:00 smithi116 ceph-mon[37097]: pgmap v5287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9065.818183 secs 2024-07-26T11:12:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9065.818183 secs 2024-07-26T11:12:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9065.818183 secs 2024-07-26T11:12:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:02 smithi171 ceph-mon[37789]: pgmap v5288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:02 smithi044 ceph-mon[32404]: pgmap v5288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:02 smithi116 ceph-mon[37097]: pgmap v5288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:04 smithi171 ceph-mon[37789]: pgmap v5289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:04 smithi044 ceph-mon[32404]: pgmap v5289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:04 smithi116 ceph-mon[37097]: pgmap v5289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:06 smithi044 ceph-mon[32404]: pgmap v5290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9070.818269 secs 2024-07-26T11:12:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:12:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:06 smithi116 ceph-mon[37097]: pgmap v5290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9070.818269 secs 2024-07-26T11:12:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:06 smithi171 ceph-mon[37789]: pgmap v5290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9070.818269 secs 2024-07-26T11:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:12:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:07 smithi044 ceph-mon[32404]: pgmap v5291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:12:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:07 smithi044 ceph-mon[32404]: pgmap v5292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:07 smithi116 ceph-mon[37097]: pgmap v5291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:12:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:07 smithi116 ceph-mon[37097]: pgmap v5292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:07 smithi171 ceph-mon[37789]: pgmap v5291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:12:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:07 smithi171 ceph-mon[37789]: pgmap v5292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:09 smithi044 ceph-mon[32404]: pgmap v5293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:09 smithi116 ceph-mon[37097]: pgmap v5293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:09 smithi171 ceph-mon[37789]: pgmap v5293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:11 smithi044 ceph-mon[32404]: pgmap v5294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9075.818438 secs 2024-07-26T11:12:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:11 smithi116 ceph-mon[37097]: pgmap v5294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9075.818438 secs 2024-07-26T11:12:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:11 smithi171 ceph-mon[37789]: pgmap v5294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9075.818438 secs 2024-07-26T11:12:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:13.398+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:12:13.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:13.398+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:12:13.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:13.398+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:13 smithi044 ceph-mon[32404]: pgmap v5295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:13 smithi116 ceph-mon[37097]: pgmap v5295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:13 smithi171 ceph-mon[37789]: pgmap v5295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:15 smithi044 ceph-mon[32404]: pgmap v5296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:15.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:15 smithi116 ceph-mon[37097]: pgmap v5296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:15 smithi171 ceph-mon[37789]: pgmap v5296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:12:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9080.818579 secs 2024-07-26T11:12:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9080.818579 secs 2024-07-26T11:12:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9080.818579 secs 2024-07-26T11:12:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:17 smithi044 ceph-mon[32404]: pgmap v5297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:12:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:17 smithi116 ceph-mon[37097]: pgmap v5297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:12:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:17 smithi171 ceph-mon[37789]: pgmap v5297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:12:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:19 smithi044 ceph-mon[32404]: pgmap v5298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:19 smithi116 ceph-mon[37097]: pgmap v5298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:19 smithi171 ceph-mon[37789]: pgmap v5298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:21 smithi044 ceph-mon[32404]: pgmap v5299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:21.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9085.818741 secs 2024-07-26T11:12:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:21 smithi116 ceph-mon[37097]: pgmap v5299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9085.818741 secs 2024-07-26T11:12:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:21 smithi171 ceph-mon[37789]: pgmap v5299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9085.818741 secs 2024-07-26T11:12:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:23 smithi044 ceph-mon[32404]: pgmap v5300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:23 smithi116 ceph-mon[37097]: pgmap v5300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:23 smithi171 ceph-mon[37789]: pgmap v5300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:25 smithi044 ceph-mon[32404]: pgmap v5301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:25 smithi116 ceph-mon[37097]: pgmap v5301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:25 smithi171 ceph-mon[37789]: pgmap v5301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:26.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9090.818877 secs 2024-07-26T11:12:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9090.818877 secs 2024-07-26T11:12:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9090.818877 secs 2024-07-26T11:12:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:27 smithi044 ceph-mon[32404]: pgmap v5302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:27 smithi116 ceph-mon[37097]: pgmap v5302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:27 smithi171 ceph-mon[37789]: pgmap v5302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:28.404+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:12:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:28.404+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:28.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:28.404+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:12:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:29 smithi044 ceph-mon[32404]: pgmap v5303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:29 smithi116 ceph-mon[37097]: pgmap v5303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:29 smithi171 ceph-mon[37789]: pgmap v5303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:31 smithi044 ceph-mon[32404]: pgmap v5304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9095.819021 secs 2024-07-26T11:12:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:31 smithi116 ceph-mon[37097]: pgmap v5304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9095.819021 secs 2024-07-26T11:12:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:31 smithi171 ceph-mon[37789]: pgmap v5304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9095.819021 secs 2024-07-26T11:12:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:33 smithi044 ceph-mon[32404]: pgmap v5305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:33 smithi116 ceph-mon[37097]: pgmap v5305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:33 smithi171 ceph-mon[37789]: pgmap v5305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:35 smithi044 ceph-mon[32404]: pgmap v5306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:35 smithi116 ceph-mon[37097]: pgmap v5306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:35 smithi171 ceph-mon[37789]: pgmap v5306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:36.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9100.819175 secs 2024-07-26T11:12:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9100.819175 secs 2024-07-26T11:12:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9100.819175 secs 2024-07-26T11:12:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:37 smithi044 ceph-mon[32404]: pgmap v5307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:37.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:37 smithi116 ceph-mon[37097]: pgmap v5307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:37 smithi171 ceph-mon[37789]: pgmap v5307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:39 smithi044 ceph-mon[32404]: pgmap v5308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:39.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:39 smithi116 ceph-mon[37097]: pgmap v5308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:39 smithi171 ceph-mon[37789]: pgmap v5308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:41 smithi044 ceph-mon[32404]: pgmap v5309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9105.819294 secs 2024-07-26T11:12:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:41 smithi116 ceph-mon[37097]: pgmap v5309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9105.819294 secs 2024-07-26T11:12:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:41 smithi171 ceph-mon[37789]: pgmap v5309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9105.819294 secs 2024-07-26T11:12:43.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:43.409+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:12:43.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:43.409+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:12:43.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:43.409+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:43 smithi044 ceph-mon[32404]: pgmap v5310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:43 smithi116 ceph-mon[37097]: pgmap v5310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:43 smithi171 ceph-mon[37789]: pgmap v5310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:45 smithi044 ceph-mon[32404]: pgmap v5311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:45 smithi116 ceph-mon[37097]: pgmap v5311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:45 smithi171 ceph-mon[37789]: pgmap v5311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9110.819454 secs 2024-07-26T11:12:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9110.819454 secs 2024-07-26T11:12:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9110.819454 secs 2024-07-26T11:12:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:47 smithi044 ceph-mon[32404]: pgmap v5312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:47 smithi116 ceph-mon[37097]: pgmap v5312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:47 smithi171 ceph-mon[37789]: pgmap v5312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:49 smithi044 ceph-mon[32404]: pgmap v5313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:49 smithi116 ceph-mon[37097]: pgmap v5313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:49 smithi171 ceph-mon[37789]: pgmap v5313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:51 smithi044 ceph-mon[32404]: pgmap v5314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:51.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9115.819601 secs 2024-07-26T11:12:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:51 smithi116 ceph-mon[37097]: pgmap v5314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9115.819601 secs 2024-07-26T11:12:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:51 smithi171 ceph-mon[37789]: pgmap v5314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9115.819601 secs 2024-07-26T11:12:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:53 smithi044 ceph-mon[32404]: pgmap v5315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:53 smithi116 ceph-mon[37097]: pgmap v5315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:53 smithi171 ceph-mon[37789]: pgmap v5315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:12:55.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:55 smithi116 ceph-mon[37097]: pgmap v5316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:55 smithi171 ceph-mon[37789]: pgmap v5316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:55 smithi044 ceph-mon[32404]: pgmap v5316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:12:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9120.819750 secs 2024-07-26T11:12:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9120.819750 secs 2024-07-26T11:12:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9120.819750 secs 2024-07-26T11:12:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:57 smithi171 ceph-mon[37789]: pgmap v5317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:57 smithi044 ceph-mon[32404]: pgmap v5317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:57 smithi116 ceph-mon[37097]: pgmap v5317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:12:58.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:58.419+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:12:58.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:58.419+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:58.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:12:58.419+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:12:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:12:59 smithi171 ceph-mon[37789]: pgmap v5318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:12:59 smithi044 ceph-mon[32404]: pgmap v5318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:12:59 smithi116 ceph-mon[37097]: pgmap v5318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:01 smithi171 ceph-mon[37789]: pgmap v5319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9125.819908 secs 2024-07-26T11:13:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:01 smithi044 ceph-mon[32404]: pgmap v5319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9125.819908 secs 2024-07-26T11:13:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:01 smithi116 ceph-mon[37097]: pgmap v5319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9125.819908 secs 2024-07-26T11:13:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:03 smithi171 ceph-mon[37789]: pgmap v5320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:03 smithi044 ceph-mon[32404]: pgmap v5320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:03 smithi116 ceph-mon[37097]: pgmap v5320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:05 smithi171 ceph-mon[37789]: pgmap v5321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:05 smithi044 ceph-mon[32404]: pgmap v5321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:05 smithi116 ceph-mon[37097]: pgmap v5321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9130.820012 secs 2024-07-26T11:13:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9130.820012 secs 2024-07-26T11:13:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9130.820012 secs 2024-07-26T11:13:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:07 smithi171 ceph-mon[37789]: pgmap v5322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:13:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:07 smithi044 ceph-mon[32404]: pgmap v5322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:08.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:08.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:08.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:13:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:07 smithi116 ceph-mon[37097]: pgmap v5322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:08.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:13:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:08 smithi171 ceph-mon[37789]: pgmap v5323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:13:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:08 smithi171 ceph-mon[37789]: pgmap v5324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:08 smithi044 ceph-mon[32404]: pgmap v5323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:13:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:08 smithi044 ceph-mon[32404]: pgmap v5324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:08 smithi116 ceph-mon[37097]: pgmap v5323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:13:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:08 smithi116 ceph-mon[37097]: pgmap v5324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:10 smithi171 ceph-mon[37789]: pgmap v5325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:10 smithi044 ceph-mon[32404]: pgmap v5325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:10 smithi116 ceph-mon[37097]: pgmap v5325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9135.820141 secs 2024-07-26T11:13:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9135.820141 secs 2024-07-26T11:13:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9135.820141 secs 2024-07-26T11:13:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:12 smithi044 ceph-mon[32404]: pgmap v5326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:12 smithi116 ceph-mon[37097]: pgmap v5326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:12 smithi171 ceph-mon[37789]: pgmap v5326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:13.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:13.420+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:13:13.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:13.420+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:13.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:13.420+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:13:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:14 smithi044 ceph-mon[32404]: pgmap v5327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:14 smithi116 ceph-mon[37097]: pgmap v5327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:14 smithi171 ceph-mon[37789]: pgmap v5327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9140.820299 secs 2024-07-26T11:13:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:16 smithi044 ceph-mon[32404]: pgmap v5328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9140.820299 secs 2024-07-26T11:13:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:16 smithi116 ceph-mon[37097]: pgmap v5328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9140.820299 secs 2024-07-26T11:13:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:16 smithi171 ceph-mon[37789]: pgmap v5328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:13:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:17 smithi044 ceph-mon[32404]: pgmap v5329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:13:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:17 smithi116 ceph-mon[37097]: pgmap v5329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:13:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:17 smithi171 ceph-mon[37789]: pgmap v5329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:13:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:20 smithi044 ceph-mon[32404]: pgmap v5330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:20 smithi116 ceph-mon[37097]: pgmap v5330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:20 smithi171 ceph-mon[37789]: pgmap v5330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9145.820446 secs 2024-07-26T11:13:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9145.820446 secs 2024-07-26T11:13:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9145.820446 secs 2024-07-26T11:13:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:22 smithi044 ceph-mon[32404]: pgmap v5331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:22 smithi116 ceph-mon[37097]: pgmap v5331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:22 smithi171 ceph-mon[37789]: pgmap v5331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:24 smithi044 ceph-mon[32404]: pgmap v5332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:24 smithi116 ceph-mon[37097]: pgmap v5332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:24 smithi171 ceph-mon[37789]: pgmap v5332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:25 smithi044 ceph-mon[32404]: pgmap v5333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:25 smithi116 ceph-mon[37097]: pgmap v5333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:25 smithi171 ceph-mon[37789]: pgmap v5333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:26.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9150.820609 secs 2024-07-26T11:13:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9150.820609 secs 2024-07-26T11:13:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9150.820609 secs 2024-07-26T11:13:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:27 smithi044 ceph-mon[32404]: pgmap v5334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:27 smithi116 ceph-mon[37097]: pgmap v5334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:27 smithi171 ceph-mon[37789]: pgmap v5334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:28.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:28.428+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:13:28.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:28.428+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:28.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:28.428+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:13:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:30 smithi044 ceph-mon[32404]: pgmap v5335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:30 smithi116 ceph-mon[37097]: pgmap v5335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:30 smithi171 ceph-mon[37789]: pgmap v5335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9155.820752 secs 2024-07-26T11:13:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9155.820752 secs 2024-07-26T11:13:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9155.820752 secs 2024-07-26T11:13:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:32 smithi044 ceph-mon[32404]: pgmap v5336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:32 smithi116 ceph-mon[37097]: pgmap v5336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:32 smithi171 ceph-mon[37789]: pgmap v5336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:33.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:33 smithi044 ceph-mon[32404]: pgmap v5337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:33 smithi116 ceph-mon[37097]: pgmap v5337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:33 smithi171 ceph-mon[37789]: pgmap v5337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9160.820906 secs 2024-07-26T11:13:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:36 smithi044 ceph-mon[32404]: pgmap v5338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9160.820906 secs 2024-07-26T11:13:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:36 smithi116 ceph-mon[37097]: pgmap v5338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9160.820906 secs 2024-07-26T11:13:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:36 smithi171 ceph-mon[37789]: pgmap v5338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:38 smithi044 ceph-mon[32404]: pgmap v5339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:38 smithi116 ceph-mon[37097]: pgmap v5339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:38 smithi171 ceph-mon[37789]: pgmap v5339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:40 smithi044 ceph-mon[32404]: pgmap v5340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:40 smithi116 ceph-mon[37097]: pgmap v5340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:40 smithi171 ceph-mon[37789]: pgmap v5340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9165.821043 secs 2024-07-26T11:13:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9165.821043 secs 2024-07-26T11:13:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9165.821043 secs 2024-07-26T11:13:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:42 smithi044 ceph-mon[32404]: pgmap v5341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:42 smithi116 ceph-mon[37097]: pgmap v5341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:42 smithi171 ceph-mon[37789]: pgmap v5341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:43.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:43.432+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:13:43.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:43.432+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:13:43.435 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:43.432+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:43 smithi044 ceph-mon[32404]: pgmap v5342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:43 smithi116 ceph-mon[37097]: pgmap v5342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:43 smithi171 ceph-mon[37789]: pgmap v5342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9170.821200 secs 2024-07-26T11:13:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:46 smithi044 ceph-mon[32404]: pgmap v5343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9170.821200 secs 2024-07-26T11:13:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:46 smithi116 ceph-mon[37097]: pgmap v5343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9170.821200 secs 2024-07-26T11:13:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:46 smithi171 ceph-mon[37789]: pgmap v5343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:48.962 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:48 smithi044 ceph-mon[32404]: pgmap v5344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:48 smithi116 ceph-mon[37097]: pgmap v5344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:48 smithi171 ceph-mon[37789]: pgmap v5344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:49 smithi044 ceph-mon[32404]: pgmap v5345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:49 smithi116 ceph-mon[37097]: pgmap v5345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:49 smithi171 ceph-mon[37789]: pgmap v5345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9175.821362 secs 2024-07-26T11:13:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9175.821362 secs 2024-07-26T11:13:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9175.821362 secs 2024-07-26T11:13:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:52 smithi044 ceph-mon[32404]: pgmap v5346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:52 smithi116 ceph-mon[37097]: pgmap v5346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:52 smithi171 ceph-mon[37789]: pgmap v5346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:54 smithi044 ceph-mon[32404]: pgmap v5347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:54 smithi116 ceph-mon[37097]: pgmap v5347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:54 smithi171 ceph-mon[37789]: pgmap v5347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9180.821527 secs 2024-07-26T11:13:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:56 smithi044 ceph-mon[32404]: pgmap v5348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9180.821527 secs 2024-07-26T11:13:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:56 smithi116 ceph-mon[37097]: pgmap v5348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9180.821527 secs 2024-07-26T11:13:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:56 smithi171 ceph-mon[37789]: pgmap v5348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:13:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:13:57 smithi044 ceph-mon[32404]: pgmap v5349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:58.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:13:57 smithi116 ceph-mon[37097]: pgmap v5349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:13:57 smithi171 ceph-mon[37789]: pgmap v5349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:13:58.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:58.437+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:13:58.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:58.437+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:58.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:13:58.437+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:14:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:00 smithi044 ceph-mon[32404]: pgmap v5350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:00 smithi116 ceph-mon[37097]: pgmap v5350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:00 smithi171 ceph-mon[37789]: pgmap v5350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9185.821697 secs 2024-07-26T11:14:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9185.821697 secs 2024-07-26T11:14:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9185.821697 secs 2024-07-26T11:14:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:02 smithi044 ceph-mon[32404]: pgmap v5351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:02 smithi116 ceph-mon[37097]: pgmap v5351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:02 smithi171 ceph-mon[37789]: pgmap v5351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:03 smithi044 ceph-mon[32404]: pgmap v5352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:14:04.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:03 smithi116 ceph-mon[37097]: pgmap v5352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:14:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:03 smithi171 ceph-mon[37789]: pgmap v5352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:14:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9190.821855 secs 2024-07-26T11:14:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:06 smithi044 ceph-mon[32404]: pgmap v5353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9190.821855 secs 2024-07-26T11:14:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:06 smithi116 ceph-mon[37097]: pgmap v5353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9190.821855 secs 2024-07-26T11:14:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:06 smithi171 ceph-mon[37789]: pgmap v5353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:07 smithi044 ceph-mon[32404]: pgmap v5354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:07 smithi116 ceph-mon[37097]: pgmap v5354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:07 smithi171 ceph-mon[37789]: pgmap v5354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:09 smithi044 ceph-mon[32404]: pgmap v5355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:09 smithi044 ceph-mon[32404]: pgmap v5356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:14:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:09 smithi116 ceph-mon[37097]: pgmap v5355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:09 smithi116 ceph-mon[37097]: pgmap v5356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:14:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:10.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:10.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:09 smithi171 ceph-mon[37789]: pgmap v5355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:10.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:09 smithi171 ceph-mon[37789]: pgmap v5356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:10.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:14:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9195.821949 secs 2024-07-26T11:14:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:11 smithi044 ceph-mon[32404]: pgmap v5357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9195.821949 secs 2024-07-26T11:14:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:11 smithi116 ceph-mon[37097]: pgmap v5357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9195.821949 secs 2024-07-26T11:14:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:11 smithi171 ceph-mon[37789]: pgmap v5357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:13.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:13.439+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:14:13.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:13.439+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:14:13.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:13.439+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:13 smithi044 ceph-mon[32404]: pgmap v5358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:13 smithi116 ceph-mon[37097]: pgmap v5358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:13 smithi171 ceph-mon[37789]: pgmap v5358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:14:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:15 smithi044 ceph-mon[32404]: pgmap v5359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:15 smithi116 ceph-mon[37097]: pgmap v5359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:16.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:16.320+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:14:16.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:16.320+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:14:16.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:16.320+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:14:16.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:16.320+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:14:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:15 smithi171 ceph-mon[37789]: pgmap v5359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9200.822104 secs 2024-07-26T11:14:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9200.822104 secs 2024-07-26T11:14:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9200.822104 secs 2024-07-26T11:14:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:17 smithi044 ceph-mon[32404]: pgmap v5360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:18.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:17 smithi116 ceph-mon[37097]: pgmap v5360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:17 smithi171 ceph-mon[37789]: pgmap v5360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:14:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:19 smithi044 ceph-mon[32404]: pgmap v5361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:19 smithi116 ceph-mon[37097]: pgmap v5361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:19 smithi171 ceph-mon[37789]: pgmap v5361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:14:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9205.822261 secs 2024-07-26T11:14:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:21 smithi044 ceph-mon[32404]: pgmap v5362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:14:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9205.822261 secs 2024-07-26T11:14:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:21 smithi116 ceph-mon[37097]: pgmap v5362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:14:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9205.822261 secs 2024-07-26T11:14:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:21 smithi171 ceph-mon[37789]: pgmap v5362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:14:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:23 smithi044 ceph-mon[32404]: pgmap v5363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:23 smithi116 ceph-mon[37097]: pgmap v5363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:23 smithi171 ceph-mon[37789]: pgmap v5363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:25 smithi044 ceph-mon[32404]: pgmap v5364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:25 smithi116 ceph-mon[37097]: pgmap v5364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:25 smithi171 ceph-mon[37789]: pgmap v5364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9210.822429 secs 2024-07-26T11:14:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9210.822429 secs 2024-07-26T11:14:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9210.822429 secs 2024-07-26T11:14:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:27 smithi044 ceph-mon[32404]: pgmap v5365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:27 smithi116 ceph-mon[37097]: pgmap v5365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:27 smithi171 ceph-mon[37789]: pgmap v5365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:28.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:28.442+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:14:28.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:28.442+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:28.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:28.442+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:14:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:29 smithi044 ceph-mon[32404]: pgmap v5366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:29 smithi116 ceph-mon[37097]: pgmap v5366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:29 smithi171 ceph-mon[37789]: pgmap v5366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9215.822593 secs 2024-07-26T11:14:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:31 smithi044 ceph-mon[32404]: pgmap v5367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9215.822593 secs 2024-07-26T11:14:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:31 smithi116 ceph-mon[37097]: pgmap v5367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9215.822593 secs 2024-07-26T11:14:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:31 smithi171 ceph-mon[37789]: pgmap v5367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:34.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:33 smithi044 ceph-mon[32404]: pgmap v5368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:33 smithi116 ceph-mon[37097]: pgmap v5368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:33 smithi171 ceph-mon[37789]: pgmap v5368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:35 smithi044 ceph-mon[32404]: pgmap v5369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:35 smithi116 ceph-mon[37097]: pgmap v5369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:35 smithi171 ceph-mon[37789]: pgmap v5369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9220.822779 secs 2024-07-26T11:14:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9220.822779 secs 2024-07-26T11:14:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9220.822779 secs 2024-07-26T11:14:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:37 smithi044 ceph-mon[32404]: pgmap v5370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:37 smithi116 ceph-mon[37097]: pgmap v5370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:37 smithi171 ceph-mon[37789]: pgmap v5370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:39 smithi044 ceph-mon[32404]: pgmap v5371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:39 smithi116 ceph-mon[37097]: pgmap v5371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:39 smithi171 ceph-mon[37789]: pgmap v5371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9225.822954 secs 2024-07-26T11:14:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:41 smithi044 ceph-mon[32404]: pgmap v5372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:14:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9225.822954 secs 2024-07-26T11:14:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:41 smithi116 ceph-mon[37097]: pgmap v5372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:14:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9225.822954 secs 2024-07-26T11:14:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:41 smithi171 ceph-mon[37789]: pgmap v5372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:14:43.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:43.446+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:14:43.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:43.447+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:14:43.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:43.447+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:43 smithi044 ceph-mon[32404]: pgmap v5373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:44 smithi116 ceph-mon[37097]: pgmap v5373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:44 smithi171 ceph-mon[37789]: pgmap v5373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:46 smithi044 ceph-mon[32404]: pgmap v5374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:46 smithi116 ceph-mon[37097]: pgmap v5374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:46 smithi171 ceph-mon[37789]: pgmap v5374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:14:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9230.823144 secs 2024-07-26T11:14:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9230.823144 secs 2024-07-26T11:14:47.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9230.823144 secs 2024-07-26T11:14:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:48 smithi171 ceph-mon[37789]: pgmap v5375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:48 smithi044 ceph-mon[32404]: pgmap v5375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:48 smithi116 ceph-mon[37097]: pgmap v5375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:14:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:50 smithi044 ceph-mon[32404]: pgmap v5376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:50 smithi116 ceph-mon[37097]: pgmap v5376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:50 smithi171 ceph-mon[37789]: pgmap v5376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9235.823277 secs 2024-07-26T11:14:51.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9235.823277 secs 2024-07-26T11:14:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9235.823277 secs 2024-07-26T11:14:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:52 smithi044 ceph-mon[32404]: pgmap v5377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:52 smithi116 ceph-mon[37097]: pgmap v5377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:52 smithi171 ceph-mon[37789]: pgmap v5377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:14:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:54 smithi044 ceph-mon[32404]: pgmap v5378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:54.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:54 smithi116 ceph-mon[37097]: pgmap v5378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:54 smithi171 ceph-mon[37789]: pgmap v5378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:56 smithi044 ceph-mon[32404]: pgmap v5379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:56.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9240.823439 secs 2024-07-26T11:14:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:56 smithi116 ceph-mon[37097]: pgmap v5379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:56.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9240.823439 secs 2024-07-26T11:14:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:56 smithi171 ceph-mon[37789]: pgmap v5379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:14:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9240.823439 secs 2024-07-26T11:14:58.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:58.456+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:14:58.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:58.456+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:58.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:14:58.456+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:14:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:14:58 smithi044 ceph-mon[32404]: pgmap v5380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:14:58 smithi116 ceph-mon[37097]: pgmap v5380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:14:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:14:58 smithi171 ceph-mon[37789]: pgmap v5380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:00 smithi044 ceph-mon[32404]: pgmap v5381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:00 smithi116 ceph-mon[37097]: pgmap v5381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:00 smithi171 ceph-mon[37789]: pgmap v5381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9245.823666 secs 2024-07-26T11:15:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9245.823666 secs 2024-07-26T11:15:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9245.823666 secs 2024-07-26T11:15:02.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:02 smithi116 ceph-mon[37097]: pgmap v5382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:02 smithi171 ceph-mon[37789]: pgmap v5382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:02 smithi044 ceph-mon[32404]: pgmap v5382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:04 smithi171 ceph-mon[37789]: pgmap v5383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:04 smithi044 ceph-mon[32404]: pgmap v5383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:05.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:04 smithi116 ceph-mon[37097]: pgmap v5383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:06 smithi171 ceph-mon[37789]: pgmap v5384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9250.823826 secs 2024-07-26T11:15:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:06 smithi044 ceph-mon[32404]: pgmap v5384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:07.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9250.823826 secs 2024-07-26T11:15:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:06 smithi116 ceph-mon[37097]: pgmap v5384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:15:07.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9250.823826 secs 2024-07-26T11:15:08.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:08 smithi171 ceph-mon[37789]: pgmap v5385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:08 smithi044 ceph-mon[32404]: pgmap v5385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:08 smithi116 ceph-mon[37097]: pgmap v5385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:10 smithi171 ceph-mon[37789]: pgmap v5386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:10.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:15:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:10 smithi044 ceph-mon[32404]: pgmap v5386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:11.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:15:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:10 smithi116 ceph-mon[37097]: pgmap v5386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:15:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:11 smithi171 ceph-mon[37789]: pgmap v5387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:15:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:11 smithi171 ceph-mon[37789]: pgmap v5388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9255.824014 secs 2024-07-26T11:15:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:11 smithi044 ceph-mon[32404]: pgmap v5387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:15:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:11 smithi044 ceph-mon[32404]: pgmap v5388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9255.824014 secs 2024-07-26T11:15:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:11 smithi116 ceph-mon[37097]: pgmap v5387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:15:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:11 smithi116 ceph-mon[37097]: pgmap v5388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9255.824014 secs 2024-07-26T11:15:13.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:13.461+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:15:13.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:13.461+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:13.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:13.461+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:15:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:13 smithi044 ceph-mon[32404]: pgmap v5389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:13 smithi171 ceph-mon[37789]: pgmap v5389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:13 smithi116 ceph-mon[37097]: pgmap v5389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:15 smithi044 ceph-mon[32404]: pgmap v5390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:16.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:15 smithi116 ceph-mon[37097]: pgmap v5390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:15 smithi171 ceph-mon[37789]: pgmap v5390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9260.824131 secs 2024-07-26T11:15:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9260.824131 secs 2024-07-26T11:15:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9260.824131 secs 2024-07-26T11:15:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:17 smithi044 ceph-mon[32404]: pgmap v5391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:17 smithi116 ceph-mon[37097]: pgmap v5391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:17 smithi171 ceph-mon[37789]: pgmap v5391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:15:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:19 smithi044 ceph-mon[32404]: pgmap v5392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:19 smithi116 ceph-mon[37097]: pgmap v5392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:19 smithi171 ceph-mon[37789]: pgmap v5392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:15:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:21 smithi044 ceph-mon[32404]: pgmap v5393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9265.824290 secs 2024-07-26T11:15:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:21 smithi116 ceph-mon[37097]: pgmap v5393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9265.824290 secs 2024-07-26T11:15:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:21 smithi171 ceph-mon[37789]: pgmap v5393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:15:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9265.824290 secs 2024-07-26T11:15:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:23 smithi044 ceph-mon[32404]: pgmap v5394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:23 smithi116 ceph-mon[37097]: pgmap v5394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:23 smithi171 ceph-mon[37789]: pgmap v5394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:26.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:25 smithi044 ceph-mon[32404]: pgmap v5395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:25 smithi116 ceph-mon[37097]: pgmap v5395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:25 smithi171 ceph-mon[37789]: pgmap v5395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:26.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9270.824431 secs 2024-07-26T11:15:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9270.824431 secs 2024-07-26T11:15:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9270.824431 secs 2024-07-26T11:15:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:27 smithi044 ceph-mon[32404]: pgmap v5396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:27 smithi116 ceph-mon[37097]: pgmap v5396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:27 smithi171 ceph-mon[37789]: pgmap v5396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:28.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:28.470+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:15:28.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:28.470+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:15:28.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:28.470+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:28 smithi044 ceph-mon[32404]: pgmap v5397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:28 smithi116 ceph-mon[37097]: pgmap v5397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:28 smithi171 ceph-mon[37789]: pgmap v5397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:31 smithi044 ceph-mon[32404]: pgmap v5398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9275.824573 secs 2024-07-26T11:15:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:31 smithi116 ceph-mon[37097]: pgmap v5398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9275.824573 secs 2024-07-26T11:15:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:31 smithi171 ceph-mon[37789]: pgmap v5398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9275.824573 secs 2024-07-26T11:15:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:33 smithi044 ceph-mon[32404]: pgmap v5399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:33 smithi116 ceph-mon[37097]: pgmap v5399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:33 smithi171 ceph-mon[37789]: pgmap v5399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:35 smithi044 ceph-mon[32404]: pgmap v5400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:35 smithi116 ceph-mon[37097]: pgmap v5400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:35 smithi171 ceph-mon[37789]: pgmap v5400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:36.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9280.824734 secs 2024-07-26T11:15:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9280.824734 secs 2024-07-26T11:15:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9280.824734 secs 2024-07-26T11:15:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:37 smithi044 ceph-mon[32404]: pgmap v5401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:37 smithi116 ceph-mon[37097]: pgmap v5401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:37 smithi171 ceph-mon[37789]: pgmap v5401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:39 smithi044 ceph-mon[32404]: pgmap v5402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:39 smithi116 ceph-mon[37097]: pgmap v5402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:39 smithi171 ceph-mon[37789]: pgmap v5402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:41 smithi044 ceph-mon[32404]: pgmap v5403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9285.824879 secs 2024-07-26T11:15:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:41 smithi116 ceph-mon[37097]: pgmap v5403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9285.824879 secs 2024-07-26T11:15:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:41 smithi171 ceph-mon[37789]: pgmap v5403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9285.824879 secs 2024-07-26T11:15:43.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:43.475+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:15:43.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:43.475+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:43.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:43.475+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:15:43.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:43 smithi044 ceph-mon[32404]: pgmap v5404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:43 smithi116 ceph-mon[37097]: pgmap v5404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:43 smithi171 ceph-mon[37789]: pgmap v5404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:45 smithi044 ceph-mon[32404]: pgmap v5405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:45 smithi116 ceph-mon[37097]: pgmap v5405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:45 smithi171 ceph-mon[37789]: pgmap v5405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9290.825045 secs 2024-07-26T11:15:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9290.825045 secs 2024-07-26T11:15:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9290.825045 secs 2024-07-26T11:15:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:47 smithi044 ceph-mon[32404]: pgmap v5406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:47 smithi116 ceph-mon[37097]: pgmap v5406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:47 smithi171 ceph-mon[37789]: pgmap v5406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:49 smithi044 ceph-mon[32404]: pgmap v5407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:49.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:49 smithi116 ceph-mon[37097]: pgmap v5407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:49 smithi171 ceph-mon[37789]: pgmap v5407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:51 smithi044 ceph-mon[32404]: pgmap v5408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9295.825217 secs 2024-07-26T11:15:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:51 smithi116 ceph-mon[37097]: pgmap v5408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9295.825217 secs 2024-07-26T11:15:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:51 smithi171 ceph-mon[37789]: pgmap v5408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9295.825217 secs 2024-07-26T11:15:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:53 smithi044 ceph-mon[32404]: pgmap v5409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:53 smithi116 ceph-mon[37097]: pgmap v5409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:53 smithi171 ceph-mon[37789]: pgmap v5409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:55 smithi044 ceph-mon[32404]: pgmap v5410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:55 smithi116 ceph-mon[37097]: pgmap v5410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:55 smithi171 ceph-mon[37789]: pgmap v5410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:15:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9300.825401 secs 2024-07-26T11:15:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9300.825401 secs 2024-07-26T11:15:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9300.825401 secs 2024-07-26T11:15:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:57 smithi116 ceph-mon[37097]: pgmap v5411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:57 smithi171 ceph-mon[37789]: pgmap v5411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:57 smithi044 ceph-mon[32404]: pgmap v5411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:15:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:58.485+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:15:58.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:58.485+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:15:58.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:15:58.485+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:15:59 smithi171 ceph-mon[37789]: pgmap v5412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:15:59 smithi044 ceph-mon[32404]: pgmap v5412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:59.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:15:59 smithi116 ceph-mon[37097]: pgmap v5412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:16:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:01 smithi171 ceph-mon[37789]: pgmap v5413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9305.825565 secs 2024-07-26T11:16:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:01 smithi044 ceph-mon[32404]: pgmap v5413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:01.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9305.825565 secs 2024-07-26T11:16:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:01 smithi116 ceph-mon[37097]: pgmap v5413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9305.825565 secs 2024-07-26T11:16:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:03 smithi171 ceph-mon[37789]: pgmap v5414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:03.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:03 smithi044 ceph-mon[32404]: pgmap v5414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:03 smithi116 ceph-mon[37097]: pgmap v5414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:05 smithi171 ceph-mon[37789]: pgmap v5415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:05 smithi044 ceph-mon[32404]: pgmap v5415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:05 smithi116 ceph-mon[37097]: pgmap v5415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:16:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9310.825744 secs 2024-07-26T11:16:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9310.825744 secs 2024-07-26T11:16:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9310.825744 secs 2024-07-26T11:16:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:07 smithi171 ceph-mon[37789]: pgmap v5416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:07 smithi044 ceph-mon[32404]: pgmap v5416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:07 smithi116 ceph-mon[37097]: pgmap v5416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:09 smithi171 ceph-mon[37789]: pgmap v5417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:16:09.762 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:09 smithi044 ceph-mon[32404]: pgmap v5417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:16:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:09 smithi116 ceph-mon[37097]: pgmap v5417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:16:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:11 smithi171 ceph-mon[37789]: pgmap v5418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9315.825822 secs 2024-07-26T11:16:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:11 smithi044 ceph-mon[32404]: pgmap v5418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9315.825822 secs 2024-07-26T11:16:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:11 smithi116 ceph-mon[37097]: pgmap v5418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:16:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9315.825822 secs 2024-07-26T11:16:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:12.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: pgmap v5419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:16:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: pgmap v5420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:16:12.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: pgmap v5419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: pgmap v5420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:16:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:16:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:16:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:16:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: pgmap v5419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:16:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: pgmap v5420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:16:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:16:13.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:13.490+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:16:13.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:13.490+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:16:13.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:13.490+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:14 smithi044 ceph-mon[32404]: pgmap v5421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:16:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:14 smithi116 ceph-mon[37097]: pgmap v5421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:16:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:14 smithi171 ceph-mon[37789]: pgmap v5421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:16:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:16 smithi044 ceph-mon[32404]: pgmap v5422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:16:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9320.825999 secs 2024-07-26T11:16:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:16 smithi116 ceph-mon[37097]: pgmap v5422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:16:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9320.825999 secs 2024-07-26T11:16:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:16 smithi171 ceph-mon[37789]: pgmap v5422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:16:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9320.825999 secs 2024-07-26T11:16:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:18 smithi116 ceph-mon[37097]: pgmap v5423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:16:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:18 smithi171 ceph-mon[37789]: pgmap v5423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:16:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:18 smithi044 ceph-mon[32404]: pgmap v5423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:16:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:20 smithi171 ceph-mon[37789]: pgmap v5424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:16:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:20 smithi044 ceph-mon[32404]: pgmap v5424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:16:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:20 smithi116 ceph-mon[37097]: pgmap v5424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:16:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9325.826180 secs 2024-07-26T11:16:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9325.826180 secs 2024-07-26T11:16:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9325.826180 secs 2024-07-26T11:16:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:22 smithi171 ceph-mon[37789]: pgmap v5425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:16:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:22 smithi044 ceph-mon[32404]: pgmap v5425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:16:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:22 smithi116 ceph-mon[37097]: pgmap v5425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:16:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:24 smithi171 ceph-mon[37789]: pgmap v5426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:24 smithi044 ceph-mon[32404]: pgmap v5426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:24 smithi116 ceph-mon[37097]: pgmap v5426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:26.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:26 smithi116 ceph-mon[37097]: pgmap v5427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:26.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9330.826358 secs 2024-07-26T11:16:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:26 smithi171 ceph-mon[37789]: pgmap v5427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9330.826358 secs 2024-07-26T11:16:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:26 smithi044 ceph-mon[32404]: pgmap v5427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9330.826358 secs 2024-07-26T11:16:28.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:28.495+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:16:28.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:28.495+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:28.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:28.495+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:16:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:28 smithi044 ceph-mon[32404]: pgmap v5428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:28 smithi171 ceph-mon[37789]: pgmap v5428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:28 smithi116 ceph-mon[37097]: pgmap v5428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:30 smithi171 ceph-mon[37789]: pgmap v5429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:30 smithi044 ceph-mon[32404]: pgmap v5429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:30 smithi116 ceph-mon[37097]: pgmap v5429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9335.826477 secs 2024-07-26T11:16:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9335.826477 secs 2024-07-26T11:16:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9335.826477 secs 2024-07-26T11:16:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:32 smithi171 ceph-mon[37789]: pgmap v5430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:32 smithi044 ceph-mon[32404]: pgmap v5430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:32 smithi116 ceph-mon[37097]: pgmap v5430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:34 smithi171 ceph-mon[37789]: pgmap v5431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:34 smithi044 ceph-mon[32404]: pgmap v5431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:34 smithi116 ceph-mon[37097]: pgmap v5431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:36 smithi044 ceph-mon[32404]: pgmap v5432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9340.826672 secs 2024-07-26T11:16:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:36 smithi116 ceph-mon[37097]: pgmap v5432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9340.826672 secs 2024-07-26T11:16:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:36 smithi171 ceph-mon[37789]: pgmap v5432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9340.826672 secs 2024-07-26T11:16:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:38 smithi044 ceph-mon[32404]: pgmap v5433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:39.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:38 smithi116 ceph-mon[37097]: pgmap v5433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:38 smithi171 ceph-mon[37789]: pgmap v5433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:40 smithi044 ceph-mon[32404]: pgmap v5434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:40 smithi116 ceph-mon[37097]: pgmap v5434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:40 smithi171 ceph-mon[37789]: pgmap v5434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9345.826836 secs 2024-07-26T11:16:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9345.826836 secs 2024-07-26T11:16:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9345.826836 secs 2024-07-26T11:16:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:42 smithi044 ceph-mon[32404]: pgmap v5435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:42 smithi116 ceph-mon[37097]: pgmap v5435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:42 smithi171 ceph-mon[37789]: pgmap v5435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:43.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:43.499+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:16:43.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:43.499+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:43.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:43.499+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:16:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:44 smithi044 ceph-mon[32404]: pgmap v5436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:44 smithi116 ceph-mon[37097]: pgmap v5436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:44 smithi171 ceph-mon[37789]: pgmap v5436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:46 smithi044 ceph-mon[32404]: pgmap v5437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9350.826972 secs 2024-07-26T11:16:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:46 smithi116 ceph-mon[37097]: pgmap v5437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:47.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9350.826972 secs 2024-07-26T11:16:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:46 smithi171 ceph-mon[37789]: pgmap v5437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9350.826972 secs 2024-07-26T11:16:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:48 smithi044 ceph-mon[32404]: pgmap v5438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:49.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:48 smithi116 ceph-mon[37097]: pgmap v5438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:48 smithi171 ceph-mon[37789]: pgmap v5438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:50 smithi044 ceph-mon[32404]: pgmap v5439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:50 smithi116 ceph-mon[37097]: pgmap v5439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:50 smithi171 ceph-mon[37789]: pgmap v5439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:16:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9355.827130 secs 2024-07-26T11:16:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9355.827130 secs 2024-07-26T11:16:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9355.827130 secs 2024-07-26T11:16:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:52 smithi044 ceph-mon[32404]: pgmap v5440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:52 smithi116 ceph-mon[37097]: pgmap v5440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:52 smithi171 ceph-mon[37789]: pgmap v5440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:16:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:54 smithi044 ceph-mon[32404]: pgmap v5441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:54 smithi116 ceph-mon[37097]: pgmap v5441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:54 smithi171 ceph-mon[37789]: pgmap v5441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:16:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:55 smithi044 ceph-mon[32404]: pgmap v5442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:55 smithi116 ceph-mon[37097]: pgmap v5442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:55 smithi171 ceph-mon[37789]: pgmap v5442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:16:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9360.827244 secs 2024-07-26T11:16:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9360.827244 secs 2024-07-26T11:16:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9360.827244 secs 2024-07-26T11:16:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:16:57 smithi044 ceph-mon[32404]: pgmap v5443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:16:57 smithi116 ceph-mon[37097]: pgmap v5443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:16:57 smithi171 ceph-mon[37789]: pgmap v5443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:16:58.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:58.509+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:16:58.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:58.509+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:58.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:16:58.509+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:17:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:00 smithi044 ceph-mon[32404]: pgmap v5444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:00 smithi116 ceph-mon[37097]: pgmap v5444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:00 smithi171 ceph-mon[37789]: pgmap v5444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9365.827392 secs 2024-07-26T11:17:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9365.827392 secs 2024-07-26T11:17:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9365.827392 secs 2024-07-26T11:17:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:02 smithi044 ceph-mon[32404]: pgmap v5445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:02 smithi116 ceph-mon[37097]: pgmap v5445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:02 smithi171 ceph-mon[37789]: pgmap v5445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:04 smithi044 ceph-mon[32404]: pgmap v5446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:04 smithi116 ceph-mon[37097]: pgmap v5446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:04 smithi171 ceph-mon[37789]: pgmap v5446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:06 smithi044 ceph-mon[32404]: pgmap v5447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9370.827547 secs 2024-07-26T11:17:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:06 smithi116 ceph-mon[37097]: pgmap v5447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9370.827547 secs 2024-07-26T11:17:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:06 smithi171 ceph-mon[37789]: pgmap v5447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9370.827547 secs 2024-07-26T11:17:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:08 smithi044 ceph-mon[32404]: pgmap v5448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:08 smithi116 ceph-mon[37097]: pgmap v5448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:08 smithi171 ceph-mon[37789]: pgmap v5448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:10 smithi044 ceph-mon[32404]: pgmap v5449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:10 smithi116 ceph-mon[37097]: pgmap v5449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:10 smithi171 ceph-mon[37789]: pgmap v5449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9375.827656 secs 2024-07-26T11:17:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9375.827656 secs 2024-07-26T11:17:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9375.827656 secs 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: pgmap v5450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: pgmap v5450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: pgmap v5450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:12.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:17:13.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:13.513+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:17:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:13.513+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:17:13.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:13.513+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: pgmap v5451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: pgmap v5452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: pgmap v5451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: pgmap v5452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:17:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:17:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:17:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:17:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:17:13.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:13.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:13.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: pgmap v5451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:17:13.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: pgmap v5452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:13.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:17:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:14 smithi044 ceph-mon[32404]: pgmap v5453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:17:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:14 smithi116 ceph-mon[37097]: pgmap v5453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:17:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:14 smithi171 ceph-mon[37789]: pgmap v5453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:17:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:16.352+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:17:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:16.352+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:17:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:16.352+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:17:16.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:16.353+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:17:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9380.827802 secs 2024-07-26T11:17:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9380.827802 secs 2024-07-26T11:17:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9380.827802 secs 2024-07-26T11:17:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:17 smithi044 ceph-mon[32404]: pgmap v5454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:17:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:17 smithi116 ceph-mon[37097]: pgmap v5454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:17:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:17 smithi171 ceph-mon[37789]: pgmap v5454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:17:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:18 smithi044 ceph-mon[32404]: pgmap v5455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:17:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:18 smithi116 ceph-mon[37097]: pgmap v5455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:17:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:18 smithi171 ceph-mon[37789]: pgmap v5455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:17:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:20 smithi044 ceph-mon[32404]: pgmap v5456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:17:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:20 smithi116 ceph-mon[37097]: pgmap v5456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:17:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:20 smithi171 ceph-mon[37789]: pgmap v5456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:17:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9385.827908 secs 2024-07-26T11:17:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9385.827908 secs 2024-07-26T11:17:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9385.827908 secs 2024-07-26T11:17:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:22 smithi044 ceph-mon[32404]: pgmap v5457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:17:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:22 smithi116 ceph-mon[37097]: pgmap v5457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:17:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:22 smithi171 ceph-mon[37789]: pgmap v5457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:17:25.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:24 smithi044 ceph-mon[32404]: pgmap v5458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:25.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:24 smithi116 ceph-mon[37097]: pgmap v5458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:24 smithi171 ceph-mon[37789]: pgmap v5458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9390.828075 secs 2024-07-26T11:17:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9390.828075 secs 2024-07-26T11:17:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9390.828075 secs 2024-07-26T11:17:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:27 smithi044 ceph-mon[32404]: pgmap v5459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:27 smithi116 ceph-mon[37097]: pgmap v5459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:27 smithi171 ceph-mon[37789]: pgmap v5459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:28.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:28.522+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:17:28.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:28.522+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:17:28.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:28.522+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:28 smithi044 ceph-mon[32404]: pgmap v5460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:28 smithi116 ceph-mon[37097]: pgmap v5460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:28 smithi171 ceph-mon[37789]: pgmap v5460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:30 smithi044 ceph-mon[32404]: pgmap v5461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:30 smithi116 ceph-mon[37097]: pgmap v5461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:30 smithi171 ceph-mon[37789]: pgmap v5461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9395.828252 secs 2024-07-26T11:17:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9395.828252 secs 2024-07-26T11:17:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9395.828252 secs 2024-07-26T11:17:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:32 smithi044 ceph-mon[32404]: pgmap v5462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:32 smithi116 ceph-mon[37097]: pgmap v5462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:32 smithi171 ceph-mon[37789]: pgmap v5462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:34 smithi044 ceph-mon[32404]: pgmap v5463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:35.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:34 smithi116 ceph-mon[37097]: pgmap v5463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:34 smithi171 ceph-mon[37789]: pgmap v5463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9400.828433 secs 2024-07-26T11:17:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9400.828433 secs 2024-07-26T11:17:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9400.828433 secs 2024-07-26T11:17:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:37 smithi044 ceph-mon[32404]: pgmap v5464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:37 smithi116 ceph-mon[37097]: pgmap v5464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:37 smithi171 ceph-mon[37789]: pgmap v5464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:38 smithi044 ceph-mon[32404]: pgmap v5465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:39.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:38 smithi116 ceph-mon[37097]: pgmap v5465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:38 smithi171 ceph-mon[37789]: pgmap v5465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:40 smithi044 ceph-mon[32404]: pgmap v5466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:40 smithi116 ceph-mon[37097]: pgmap v5466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:40 smithi171 ceph-mon[37789]: pgmap v5466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9405.828605 secs 2024-07-26T11:17:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9405.828605 secs 2024-07-26T11:17:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9405.828605 secs 2024-07-26T11:17:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:42 smithi044 ceph-mon[32404]: pgmap v5467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:42 smithi116 ceph-mon[37097]: pgmap v5467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:42 smithi171 ceph-mon[37789]: pgmap v5467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:43.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:43.525+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:17:43.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:43.525+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:17:43.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:43.525+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:44 smithi044 ceph-mon[32404]: pgmap v5468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:44 smithi116 ceph-mon[37097]: pgmap v5468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:44 smithi171 ceph-mon[37789]: pgmap v5468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9410.828665 secs 2024-07-26T11:17:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9410.828665 secs 2024-07-26T11:17:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9410.828665 secs 2024-07-26T11:17:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:47 smithi044 ceph-mon[32404]: pgmap v5469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:47 smithi116 ceph-mon[37097]: pgmap v5469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:47 smithi171 ceph-mon[37789]: pgmap v5469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:48 smithi044 ceph-mon[32404]: pgmap v5470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:48 smithi116 ceph-mon[37097]: pgmap v5470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:48 smithi171 ceph-mon[37789]: pgmap v5470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:50 smithi044 ceph-mon[32404]: pgmap v5471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:50 smithi116 ceph-mon[37097]: pgmap v5471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:50 smithi171 ceph-mon[37789]: pgmap v5471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:17:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9415.828848 secs 2024-07-26T11:17:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9415.828848 secs 2024-07-26T11:17:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9415.828848 secs 2024-07-26T11:17:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:52 smithi044 ceph-mon[32404]: pgmap v5472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:52 smithi116 ceph-mon[37097]: pgmap v5472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:52 smithi171 ceph-mon[37789]: pgmap v5472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:17:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:54 smithi044 ceph-mon[32404]: pgmap v5473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:54 smithi116 ceph-mon[37097]: pgmap v5473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:54 smithi171 ceph-mon[37789]: pgmap v5473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:17:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9420.829002 secs 2024-07-26T11:17:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9420.829002 secs 2024-07-26T11:17:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9420.829002 secs 2024-07-26T11:17:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:57 smithi044 ceph-mon[32404]: pgmap v5474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:57 smithi116 ceph-mon[37097]: pgmap v5474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:57 smithi171 ceph-mon[37789]: pgmap v5474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:17:58.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:58.535+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:17:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:58.535+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:17:58.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:17:58.535+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:58.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:17:58 smithi044 ceph-mon[32404]: pgmap v5475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:59.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:17:58 smithi116 ceph-mon[37097]: pgmap v5475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:17:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:17:58 smithi171 ceph-mon[37789]: pgmap v5475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:00 smithi044 ceph-mon[32404]: pgmap v5476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:00 smithi116 ceph-mon[37097]: pgmap v5476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:00 smithi171 ceph-mon[37789]: pgmap v5476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:01.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:01.360+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:18:01.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:01.360+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:18:01.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:01.361+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:18:01.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:01.361+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:18:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9425.829140 secs 2024-07-26T11:18:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9425.829140 secs 2024-07-26T11:18:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9425.829140 secs 2024-07-26T11:18:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:02 smithi044 ceph-mon[32404]: pgmap v5477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:02 smithi116 ceph-mon[37097]: pgmap v5477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:02 smithi171 ceph-mon[37789]: pgmap v5477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:04 smithi044 ceph-mon[32404]: pgmap v5478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:04 smithi116 ceph-mon[37097]: pgmap v5478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:04 smithi171 ceph-mon[37789]: pgmap v5478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9430.829321 secs 2024-07-26T11:18:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9430.829321 secs 2024-07-26T11:18:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9430.829321 secs 2024-07-26T11:18:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:07 smithi044 ceph-mon[32404]: pgmap v5479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:07 smithi116 ceph-mon[37097]: pgmap v5479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:07 smithi171 ceph-mon[37789]: pgmap v5479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:18:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:08 smithi044 ceph-mon[32404]: pgmap v5480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:08 smithi116 ceph-mon[37097]: pgmap v5480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:08 smithi171 ceph-mon[37789]: pgmap v5480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:18:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:10 smithi044 ceph-mon[32404]: pgmap v5481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:11.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:10 smithi116 ceph-mon[37097]: pgmap v5481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:10 smithi171 ceph-mon[37789]: pgmap v5481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9435.829460 secs 2024-07-26T11:18:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9435.829460 secs 2024-07-26T11:18:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9435.829460 secs 2024-07-26T11:18:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:12 smithi044 ceph-mon[32404]: pgmap v5482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:12 smithi116 ceph-mon[37097]: pgmap v5482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:12 smithi171 ceph-mon[37789]: pgmap v5482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:13.539+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:18:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:13.539+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:18:13.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:13.539+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:13.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:14 smithi044 ceph-mon[32404]: pgmap v5483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:18:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:14 smithi044 ceph-mon[32404]: pgmap v5484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:18:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:14 smithi116 ceph-mon[37097]: pgmap v5483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:18:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:14 smithi116 ceph-mon[37097]: pgmap v5484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:18:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:14 smithi171 ceph-mon[37789]: pgmap v5483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:18:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:14 smithi171 ceph-mon[37789]: pgmap v5484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:18:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9440.829684 secs 2024-07-26T11:18:17.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:16 smithi044 ceph-mon[32404]: pgmap v5485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:18:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9440.829684 secs 2024-07-26T11:18:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:16 smithi116 ceph-mon[37097]: pgmap v5485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:18:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9440.829684 secs 2024-07-26T11:18:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:16 smithi171 ceph-mon[37789]: pgmap v5485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:18:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:18 smithi044 ceph-mon[32404]: pgmap v5486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:18:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:18 smithi116 ceph-mon[37097]: pgmap v5486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:18:19.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:18 smithi171 ceph-mon[37789]: pgmap v5486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:18:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:20 smithi044 ceph-mon[32404]: pgmap v5487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:20 smithi116 ceph-mon[37097]: pgmap v5487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:20 smithi171 ceph-mon[37789]: pgmap v5487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9445.829785 secs 2024-07-26T11:18:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9445.829785 secs 2024-07-26T11:18:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9445.829785 secs 2024-07-26T11:18:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:22 smithi044 ceph-mon[32404]: pgmap v5488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:23.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:22 smithi116 ceph-mon[37097]: pgmap v5488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:22 smithi171 ceph-mon[37789]: pgmap v5488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:18:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:25 smithi171 ceph-mon[37789]: pgmap v5489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:18:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:25 smithi044 ceph-mon[32404]: pgmap v5489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:18:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:25 smithi116 ceph-mon[37097]: pgmap v5489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:18:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9450.829894 secs 2024-07-26T11:18:26.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9450.829894 secs 2024-07-26T11:18:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9450.829894 secs 2024-07-26T11:18:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:27 smithi044 ceph-mon[32404]: pgmap v5490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:27 smithi116 ceph-mon[37097]: pgmap v5490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:27 smithi171 ceph-mon[37789]: pgmap v5490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:28.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:28.541+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:18:28.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:28.541+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:18:28.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:28.541+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:29 smithi044 ceph-mon[32404]: pgmap v5491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:29 smithi116 ceph-mon[37097]: pgmap v5491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:29 smithi171 ceph-mon[37789]: pgmap v5491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:31 smithi044 ceph-mon[32404]: pgmap v5492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9455.830019 secs 2024-07-26T11:18:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:31 smithi116 ceph-mon[37097]: pgmap v5492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9455.830019 secs 2024-07-26T11:18:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:31 smithi171 ceph-mon[37789]: pgmap v5492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9455.830019 secs 2024-07-26T11:18:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:33 smithi044 ceph-mon[32404]: pgmap v5493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:33 smithi116 ceph-mon[37097]: pgmap v5493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:33 smithi171 ceph-mon[37789]: pgmap v5493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:35 smithi044 ceph-mon[32404]: pgmap v5494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:35 smithi116 ceph-mon[37097]: pgmap v5494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:35 smithi171 ceph-mon[37789]: pgmap v5494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:36.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9460.830204 secs 2024-07-26T11:18:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9460.830204 secs 2024-07-26T11:18:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9460.830204 secs 2024-07-26T11:18:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:37 smithi044 ceph-mon[32404]: pgmap v5495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:37 smithi116 ceph-mon[37097]: pgmap v5495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:37 smithi171 ceph-mon[37789]: pgmap v5495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:39 smithi044 ceph-mon[32404]: pgmap v5496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:39 smithi116 ceph-mon[37097]: pgmap v5496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:39 smithi171 ceph-mon[37789]: pgmap v5496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:41 smithi044 ceph-mon[32404]: pgmap v5497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9465.830383 secs 2024-07-26T11:18:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:41 smithi116 ceph-mon[37097]: pgmap v5497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9465.830383 secs 2024-07-26T11:18:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:41 smithi171 ceph-mon[37789]: pgmap v5497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9465.830383 secs 2024-07-26T11:18:43.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:43.544+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:18:43.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:43.544+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:43.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:43.544+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:18:43.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:43 smithi044 ceph-mon[32404]: pgmap v5498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:44.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:43 smithi116 ceph-mon[37097]: pgmap v5498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:43 smithi171 ceph-mon[37789]: pgmap v5498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:45 smithi044 ceph-mon[32404]: pgmap v5499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:45 smithi116 ceph-mon[37097]: pgmap v5499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:45 smithi171 ceph-mon[37789]: pgmap v5499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9470.830569 secs 2024-07-26T11:18:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9470.830569 secs 2024-07-26T11:18:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9470.830569 secs 2024-07-26T11:18:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:47 smithi044 ceph-mon[32404]: pgmap v5500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:47 smithi116 ceph-mon[37097]: pgmap v5500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:47 smithi171 ceph-mon[37789]: pgmap v5500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:49 smithi044 ceph-mon[32404]: pgmap v5501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:49 smithi116 ceph-mon[37097]: pgmap v5501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:49 smithi171 ceph-mon[37789]: pgmap v5501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:51 smithi044 ceph-mon[32404]: pgmap v5502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9475.830749 secs 2024-07-26T11:18:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:51 smithi116 ceph-mon[37097]: pgmap v5502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:52.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9475.830749 secs 2024-07-26T11:18:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:51 smithi171 ceph-mon[37789]: pgmap v5502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:18:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9475.830749 secs 2024-07-26T11:18:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:52 smithi044 ceph-mon[32404]: pgmap v5503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:52 smithi116 ceph-mon[37097]: pgmap v5503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:52 smithi171 ceph-mon[37789]: pgmap v5503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:54 smithi044 ceph-mon[32404]: pgmap v5504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:54 smithi116 ceph-mon[37097]: pgmap v5504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:54 smithi171 ceph-mon[37789]: pgmap v5504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9480.830904 secs 2024-07-26T11:18:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:56 smithi044 ceph-mon[32404]: pgmap v5505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9480.830904 secs 2024-07-26T11:18:57.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:56 smithi116 ceph-mon[37097]: pgmap v5505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9480.830904 secs 2024-07-26T11:18:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:56 smithi171 ceph-mon[37789]: pgmap v5505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:18:58.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:58.554+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:18:58.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:58.554+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:58.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:18:58.554+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:18:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:18:58 smithi044 ceph-mon[32404]: pgmap v5506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:18:58 smithi116 ceph-mon[37097]: pgmap v5506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:18:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:18:58 smithi171 ceph-mon[37789]: pgmap v5506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:00 smithi044 ceph-mon[32404]: pgmap v5507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:00 smithi116 ceph-mon[37097]: pgmap v5507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:00 smithi171 ceph-mon[37789]: pgmap v5507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9485.831016 secs 2024-07-26T11:19:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9485.831016 secs 2024-07-26T11:19:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9485.831016 secs 2024-07-26T11:19:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:03 smithi044 ceph-mon[32404]: pgmap v5508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:03 smithi116 ceph-mon[37097]: pgmap v5508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:03 smithi171 ceph-mon[37789]: pgmap v5508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:05 smithi116 ceph-mon[37097]: pgmap v5509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:05 smithi171 ceph-mon[37789]: pgmap v5509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:05 smithi044 ceph-mon[32404]: pgmap v5509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9490.831177 secs 2024-07-26T11:19:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:07 smithi171 ceph-mon[37789]: pgmap v5510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:19:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9490.831177 secs 2024-07-26T11:19:07.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:07 smithi044 ceph-mon[32404]: pgmap v5510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:19:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9490.831177 secs 2024-07-26T11:19:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:07 smithi116 ceph-mon[37097]: pgmap v5510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:19:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:09 smithi171 ceph-mon[37789]: pgmap v5511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:09 smithi044 ceph-mon[32404]: pgmap v5511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:09 smithi116 ceph-mon[37097]: pgmap v5511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:11 smithi171 ceph-mon[37789]: pgmap v5512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:11 smithi044 ceph-mon[32404]: pgmap v5512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:11.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:11 smithi116 ceph-mon[37097]: pgmap v5512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9495.831343 secs 2024-07-26T11:19:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9495.831343 secs 2024-07-26T11:19:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9495.831343 secs 2024-07-26T11:19:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:13 smithi171 ceph-mon[37789]: pgmap v5513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:13.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:13 smithi044 ceph-mon[32404]: pgmap v5513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:13.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:13 smithi116 ceph-mon[37097]: pgmap v5513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:13.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:13.559+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:19:13.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:13.559+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:13.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:13.559+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:19:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:15 smithi171 ceph-mon[37789]: pgmap v5514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:15 smithi044 ceph-mon[32404]: pgmap v5514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:15 smithi116 ceph-mon[37097]: pgmap v5514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:19:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9500.831475 secs 2024-07-26T11:19:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:17 smithi171 ceph-mon[37789]: pgmap v5515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:19:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:17 smithi171 ceph-mon[37789]: pgmap v5516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9500.831475 secs 2024-07-26T11:19:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:17 smithi044 ceph-mon[32404]: pgmap v5515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:19:17.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:17 smithi044 ceph-mon[32404]: pgmap v5516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9500.831475 secs 2024-07-26T11:19:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:17 smithi116 ceph-mon[37097]: pgmap v5515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:19:17.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:17 smithi116 ceph-mon[37097]: pgmap v5516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:19 smithi171 ceph-mon[37789]: pgmap v5517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:19 smithi044 ceph-mon[32404]: pgmap v5517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:19 smithi116 ceph-mon[37097]: pgmap v5517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:21 smithi171 ceph-mon[37789]: pgmap v5518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:21 smithi044 ceph-mon[32404]: pgmap v5518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:21 smithi116 ceph-mon[37097]: pgmap v5518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9505.831659 secs 2024-07-26T11:19:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9505.831659 secs 2024-07-26T11:19:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9505.831659 secs 2024-07-26T11:19:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:23 smithi044 ceph-mon[32404]: pgmap v5519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:23 smithi116 ceph-mon[37097]: pgmap v5519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:23 smithi171 ceph-mon[37789]: pgmap v5519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:25 smithi044 ceph-mon[32404]: pgmap v5520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:25 smithi116 ceph-mon[37097]: pgmap v5520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:25 smithi171 ceph-mon[37789]: pgmap v5520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:19:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9510.831784 secs 2024-07-26T11:19:27.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:27 smithi044 ceph-mon[32404]: pgmap v5521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:19:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9510.831784 secs 2024-07-26T11:19:27.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:27 smithi116 ceph-mon[37097]: pgmap v5521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:19:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9510.831784 secs 2024-07-26T11:19:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:27 smithi171 ceph-mon[37789]: pgmap v5521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:19:28.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:28.569+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:19:28.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:28.569+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:28.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:28.569+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:19:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:29 smithi044 ceph-mon[32404]: pgmap v5522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:29 smithi116 ceph-mon[37097]: pgmap v5522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:29 smithi171 ceph-mon[37789]: pgmap v5522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:31 smithi044 ceph-mon[32404]: pgmap v5523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:31 smithi116 ceph-mon[37097]: pgmap v5523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:31 smithi171 ceph-mon[37789]: pgmap v5523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9515.831949 secs 2024-07-26T11:19:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9515.831949 secs 2024-07-26T11:19:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9515.831949 secs 2024-07-26T11:19:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:33 smithi044 ceph-mon[32404]: pgmap v5524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:33.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:33 smithi116 ceph-mon[37097]: pgmap v5524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:33 smithi171 ceph-mon[37789]: pgmap v5524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:35 smithi044 ceph-mon[32404]: pgmap v5525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:35 smithi116 ceph-mon[37097]: pgmap v5525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:35 smithi171 ceph-mon[37789]: pgmap v5525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9520.832055 secs 2024-07-26T11:19:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:37 smithi044 ceph-mon[32404]: pgmap v5526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9520.832055 secs 2024-07-26T11:19:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:37 smithi116 ceph-mon[37097]: pgmap v5526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9520.832055 secs 2024-07-26T11:19:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:37 smithi171 ceph-mon[37789]: pgmap v5526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:39 smithi044 ceph-mon[32404]: pgmap v5527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:39 smithi116 ceph-mon[37097]: pgmap v5527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:39 smithi171 ceph-mon[37789]: pgmap v5527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:41 smithi044 ceph-mon[32404]: pgmap v5528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:41 smithi116 ceph-mon[37097]: pgmap v5528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:41 smithi171 ceph-mon[37789]: pgmap v5528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9525.832236 secs 2024-07-26T11:19:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9525.832236 secs 2024-07-26T11:19:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9525.832236 secs 2024-07-26T11:19:43.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:43 smithi044 ceph-mon[32404]: pgmap v5529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:43 smithi116 ceph-mon[37097]: pgmap v5529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:43.574+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:19:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:43.574+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:19:43.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:43.574+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:43 smithi171 ceph-mon[37789]: pgmap v5529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:45 smithi044 ceph-mon[32404]: pgmap v5530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:45 smithi116 ceph-mon[37097]: pgmap v5530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:45 smithi171 ceph-mon[37789]: pgmap v5530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9530.832406 secs 2024-07-26T11:19:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9530.832406 secs 2024-07-26T11:19:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9530.832406 secs 2024-07-26T11:19:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:47 smithi044 ceph-mon[32404]: pgmap v5531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:47 smithi116 ceph-mon[37097]: pgmap v5531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:47 smithi171 ceph-mon[37789]: pgmap v5531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:19:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:49 smithi044 ceph-mon[32404]: pgmap v5532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:49 smithi116 ceph-mon[37097]: pgmap v5532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:49 smithi171 ceph-mon[37789]: pgmap v5532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:51 smithi044 ceph-mon[32404]: pgmap v5533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:51.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9535.832532 secs 2024-07-26T11:19:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:51 smithi116 ceph-mon[37097]: pgmap v5533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9535.832532 secs 2024-07-26T11:19:51.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:51 smithi171 ceph-mon[37789]: pgmap v5533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:19:51.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9535.832532 secs 2024-07-26T11:19:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:53 smithi044 ceph-mon[32404]: pgmap v5534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:53 smithi116 ceph-mon[37097]: pgmap v5534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:53 smithi171 ceph-mon[37789]: pgmap v5534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:55 smithi044 ceph-mon[32404]: pgmap v5535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:55 smithi116 ceph-mon[37097]: pgmap v5535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:55 smithi171 ceph-mon[37789]: pgmap v5535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9540.832712 secs 2024-07-26T11:19:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9540.832712 secs 2024-07-26T11:19:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9540.832712 secs 2024-07-26T11:19:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:57 smithi044 ceph-mon[32404]: pgmap v5536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:19:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:57 smithi116 ceph-mon[37097]: pgmap v5536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:19:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:57 smithi171 ceph-mon[37789]: pgmap v5536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:19:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:58.583+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:19:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:58.583+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:58.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:19:58.583+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:19:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:19:59 smithi044 ceph-mon[32404]: pgmap v5537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:59.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:19:59 smithi116 ceph-mon[37097]: pgmap v5537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:19:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:19:59 smithi171 ceph-mon[37789]: pgmap v5537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:20:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:01 smithi044 ceph-mon[32404]: pgmap v5538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:01.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9545.832904 secs 2024-07-26T11:20:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:01 smithi116 ceph-mon[37097]: pgmap v5538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9545.832904 secs 2024-07-26T11:20:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:01 smithi171 ceph-mon[37789]: pgmap v5538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9545.832904 secs 2024-07-26T11:20:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:03 smithi044 ceph-mon[32404]: pgmap v5539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:03 smithi116 ceph-mon[37097]: pgmap v5539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:03 smithi171 ceph-mon[37789]: pgmap v5539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:05 smithi044 ceph-mon[32404]: pgmap v5540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:05 smithi116 ceph-mon[37097]: pgmap v5540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:05 smithi171 ceph-mon[37789]: pgmap v5540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9550.833028 secs 2024-07-26T11:20:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9550.833028 secs 2024-07-26T11:20:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9550.833028 secs 2024-07-26T11:20:07.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:07 smithi116 ceph-mon[37097]: pgmap v5541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:07 smithi171 ceph-mon[37789]: pgmap v5541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:07 smithi044 ceph-mon[32404]: pgmap v5541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:09 smithi171 ceph-mon[37789]: pgmap v5542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:09 smithi044 ceph-mon[32404]: pgmap v5542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:09 smithi116 ceph-mon[37097]: pgmap v5542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:11 smithi171 ceph-mon[37789]: pgmap v5543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9555.833194 secs 2024-07-26T11:20:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:11 smithi044 ceph-mon[32404]: pgmap v5543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:11.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9555.833194 secs 2024-07-26T11:20:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:11 smithi116 ceph-mon[37097]: pgmap v5543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9555.833194 secs 2024-07-26T11:20:13.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:13 smithi044 ceph-mon[32404]: pgmap v5544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:13 smithi171 ceph-mon[37789]: pgmap v5544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:13.588+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:20:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:13.588+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:13.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:13.588+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:20:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:13 smithi116 ceph-mon[37097]: pgmap v5544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:15 smithi171 ceph-mon[37789]: pgmap v5545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:15 smithi044 ceph-mon[32404]: pgmap v5545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:15 smithi116 ceph-mon[37097]: pgmap v5545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:16.385+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:20:16.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:16.385+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:20:16.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:16.386+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:20:16.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:16.386+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:20:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9560.833378 secs 2024-07-26T11:20:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9560.833378 secs 2024-07-26T11:20:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9560.833378 secs 2024-07-26T11:20:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:17 smithi044 ceph-mon[32404]: pgmap v5546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:17 smithi116 ceph-mon[37097]: pgmap v5546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:17 smithi171 ceph-mon[37789]: pgmap v5546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:19 smithi044 ceph-mon[32404]: pgmap v5547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:19 smithi116 ceph-mon[37097]: pgmap v5547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:19 smithi171 ceph-mon[37789]: pgmap v5547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:20 smithi044 ceph-mon[32404]: pgmap v5548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:20 smithi116 ceph-mon[37097]: pgmap v5548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:20 smithi171 ceph-mon[37789]: pgmap v5548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9565.833527 secs 2024-07-26T11:20:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:21 smithi044 ceph-mon[32404]: pgmap v5549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:21 smithi044 ceph-mon[32404]: pgmap v5550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:20:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9565.833527 secs 2024-07-26T11:20:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:21 smithi116 ceph-mon[37097]: pgmap v5549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:21 smithi116 ceph-mon[37097]: pgmap v5550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:20:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9565.833527 secs 2024-07-26T11:20:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:22.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:21 smithi171 ceph-mon[37789]: pgmap v5549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:22.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:21 smithi171 ceph-mon[37789]: pgmap v5550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:20:22.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:20:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:23 smithi171 ceph-mon[37789]: pgmap v5551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:23 smithi044 ceph-mon[32404]: pgmap v5551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:23 smithi116 ceph-mon[37097]: pgmap v5551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:25 smithi171 ceph-mon[37789]: pgmap v5552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:25 smithi044 ceph-mon[32404]: pgmap v5552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:25 smithi116 ceph-mon[37097]: pgmap v5552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9570.833697 secs 2024-07-26T11:20:27.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9570.833697 secs 2024-07-26T11:20:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9570.833697 secs 2024-07-26T11:20:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:27 smithi171 ceph-mon[37789]: pgmap v5553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:20:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:27 smithi044 ceph-mon[32404]: pgmap v5553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:20:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:27 smithi116 ceph-mon[37097]: pgmap v5553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:20:28.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:28.598+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:20:28.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:28.598+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:28.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:28.598+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:20:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:29 smithi171 ceph-mon[37789]: pgmap v5554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:29 smithi044 ceph-mon[32404]: pgmap v5554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:30.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:29 smithi116 ceph-mon[37097]: pgmap v5554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:20:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9575.833819 secs 2024-07-26T11:20:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:31 smithi044 ceph-mon[32404]: pgmap v5555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:20:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9575.833819 secs 2024-07-26T11:20:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:31 smithi116 ceph-mon[37097]: pgmap v5555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9575.833819 secs 2024-07-26T11:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:31 smithi171 ceph-mon[37789]: pgmap v5555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:20:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:33 smithi044 ceph-mon[32404]: pgmap v5556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:20:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:33 smithi116 ceph-mon[37097]: pgmap v5556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:20:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:33 smithi171 ceph-mon[37789]: pgmap v5556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:20:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:35 smithi044 ceph-mon[32404]: pgmap v5557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:35 smithi116 ceph-mon[37097]: pgmap v5557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:35 smithi171 ceph-mon[37789]: pgmap v5557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:37.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9580.833999 secs 2024-07-26T11:20:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9580.833999 secs 2024-07-26T11:20:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9580.833999 secs 2024-07-26T11:20:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:37 smithi044 ceph-mon[32404]: pgmap v5558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:37 smithi116 ceph-mon[37097]: pgmap v5558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:37 smithi171 ceph-mon[37789]: pgmap v5558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:39 smithi044 ceph-mon[32404]: pgmap v5559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:39 smithi116 ceph-mon[37097]: pgmap v5559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:39 smithi171 ceph-mon[37789]: pgmap v5559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9585.834164 secs 2024-07-26T11:20:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:41 smithi044 ceph-mon[32404]: pgmap v5560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9585.834164 secs 2024-07-26T11:20:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:41 smithi116 ceph-mon[37097]: pgmap v5560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9585.834164 secs 2024-07-26T11:20:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:41 smithi171 ceph-mon[37789]: pgmap v5560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:43.602+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:20:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:43.602+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:20:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:43.602+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:43 smithi044 ceph-mon[32404]: pgmap v5561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:43 smithi116 ceph-mon[37097]: pgmap v5561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:43 smithi171 ceph-mon[37789]: pgmap v5561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:45 smithi044 ceph-mon[32404]: pgmap v5562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:45 smithi116 ceph-mon[37097]: pgmap v5562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:45 smithi171 ceph-mon[37789]: pgmap v5562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9590.834347 secs 2024-07-26T11:20:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9590.834347 secs 2024-07-26T11:20:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9590.834347 secs 2024-07-26T11:20:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:47 smithi044 ceph-mon[32404]: pgmap v5563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:47 smithi116 ceph-mon[37097]: pgmap v5563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:47 smithi171 ceph-mon[37789]: pgmap v5563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:49 smithi044 ceph-mon[32404]: pgmap v5564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:49 smithi116 ceph-mon[37097]: pgmap v5564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:49 smithi171 ceph-mon[37789]: pgmap v5564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:20:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9595.834519 secs 2024-07-26T11:20:52.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:51 smithi044 ceph-mon[32404]: pgmap v5565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9595.834519 secs 2024-07-26T11:20:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:51 smithi116 ceph-mon[37097]: pgmap v5565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9595.834519 secs 2024-07-26T11:20:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:51 smithi171 ceph-mon[37789]: pgmap v5565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:20:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:53 smithi044 ceph-mon[32404]: pgmap v5566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:53 smithi116 ceph-mon[37097]: pgmap v5566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:53 smithi171 ceph-mon[37789]: pgmap v5566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:20:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:55 smithi044 ceph-mon[32404]: pgmap v5567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:55 smithi116 ceph-mon[37097]: pgmap v5567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:55 smithi171 ceph-mon[37789]: pgmap v5567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:20:57.175 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9600.834678 secs 2024-07-26T11:20:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9600.834678 secs 2024-07-26T11:20:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9600.834678 secs 2024-07-26T11:20:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:57 smithi044 ceph-mon[32404]: pgmap v5568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:57 smithi116 ceph-mon[37097]: pgmap v5568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:57 smithi171 ceph-mon[37789]: pgmap v5568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:20:58.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:58.611+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:20:58.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:58.611+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:58.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:20:58.611+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:21:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:20:59 smithi044 ceph-mon[32404]: pgmap v5569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:20:59 smithi116 ceph-mon[37097]: pgmap v5569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:20:59 smithi171 ceph-mon[37789]: pgmap v5569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9605.834862 secs 2024-07-26T11:21:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:01 smithi044 ceph-mon[32404]: pgmap v5570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9605.834862 secs 2024-07-26T11:21:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:01 smithi116 ceph-mon[37097]: pgmap v5570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9605.834862 secs 2024-07-26T11:21:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:01 smithi171 ceph-mon[37789]: pgmap v5570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:03 smithi044 ceph-mon[32404]: pgmap v5571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:03 smithi116 ceph-mon[37097]: pgmap v5571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:03 smithi171 ceph-mon[37789]: pgmap v5571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:05 smithi044 ceph-mon[32404]: pgmap v5572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:05 smithi116 ceph-mon[37097]: pgmap v5572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:05 smithi171 ceph-mon[37789]: pgmap v5572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:07.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9610.834991 secs 2024-07-26T11:21:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9610.834991 secs 2024-07-26T11:21:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9610.834991 secs 2024-07-26T11:21:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:07 smithi044 ceph-mon[32404]: pgmap v5573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:07 smithi116 ceph-mon[37097]: pgmap v5573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:07 smithi171 ceph-mon[37789]: pgmap v5573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:09 smithi044 ceph-mon[32404]: pgmap v5574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:09 smithi116 ceph-mon[37097]: pgmap v5574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:09 smithi171 ceph-mon[37789]: pgmap v5574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9615.835150 secs 2024-07-26T11:21:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:11 smithi044 ceph-mon[32404]: pgmap v5575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9615.835150 secs 2024-07-26T11:21:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:11 smithi116 ceph-mon[37097]: pgmap v5575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9615.835150 secs 2024-07-26T11:21:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:11 smithi171 ceph-mon[37789]: pgmap v5575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:13.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:13.613+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:21:13.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:13.613+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:13.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:13.613+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:21:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:13 smithi044 ceph-mon[32404]: pgmap v5576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:13 smithi116 ceph-mon[37097]: pgmap v5576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:13 smithi171 ceph-mon[37789]: pgmap v5576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:15 smithi044 ceph-mon[32404]: pgmap v5577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:16 smithi116 ceph-mon[37097]: pgmap v5577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:15 smithi171 ceph-mon[37789]: pgmap v5577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:21:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9620.835315 secs 2024-07-26T11:21:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9620.835315 secs 2024-07-26T11:21:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9620.835315 secs 2024-07-26T11:21:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:18 smithi044 ceph-mon[32404]: pgmap v5578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:18 smithi116 ceph-mon[37097]: pgmap v5578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:18 smithi171 ceph-mon[37789]: pgmap v5578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:21:20.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:20 smithi116 ceph-mon[37097]: pgmap v5579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:20 smithi171 ceph-mon[37789]: pgmap v5579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:20 smithi044 ceph-mon[32404]: pgmap v5579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:21:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:21 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:21 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:21 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9625.835477 secs 2024-07-26T11:21:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:22 smithi171 ceph-mon[37789]: pgmap v5580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:21:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9625.835477 secs 2024-07-26T11:21:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:22 smithi044 ceph-mon[32404]: pgmap v5580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:21:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9625.835477 secs 2024-07-26T11:21:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:22 smithi116 ceph-mon[37097]: pgmap v5580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:22.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:21:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:23 smithi171 ceph-mon[37789]: pgmap v5581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:21:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:23 smithi171 ceph-mon[37789]: pgmap v5582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:23 smithi044 ceph-mon[32404]: pgmap v5581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:21:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:23 smithi044 ceph-mon[32404]: pgmap v5582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:23 smithi116 ceph-mon[37097]: pgmap v5581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:21:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:23 smithi116 ceph-mon[37097]: pgmap v5582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:25 smithi171 ceph-mon[37789]: pgmap v5583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:25 smithi044 ceph-mon[32404]: pgmap v5583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:25 smithi116 ceph-mon[37097]: pgmap v5583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:27.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9630.835675 secs 2024-07-26T11:21:27.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:27 smithi116 ceph-mon[37097]: pgmap v5584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:21:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9630.835675 secs 2024-07-26T11:21:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:27 smithi171 ceph-mon[37789]: pgmap v5584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:21:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9630.835675 secs 2024-07-26T11:21:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:27 smithi044 ceph-mon[32404]: pgmap v5584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:21:28.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:28.623+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:21:28.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:28.623+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:28.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:28.623+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:21:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:29 smithi171 ceph-mon[37789]: pgmap v5585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:29 smithi044 ceph-mon[32404]: pgmap v5585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:29 smithi116 ceph-mon[37097]: pgmap v5585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:31 smithi171 ceph-mon[37789]: pgmap v5586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:31 smithi044 ceph-mon[32404]: pgmap v5586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:31 smithi116 ceph-mon[37097]: pgmap v5586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:21:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9635.835805 secs 2024-07-26T11:21:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9635.835805 secs 2024-07-26T11:21:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9635.835805 secs 2024-07-26T11:21:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:33 smithi171 ceph-mon[37789]: pgmap v5587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:21:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:33 smithi044 ceph-mon[32404]: pgmap v5587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:21:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:33 smithi116 ceph-mon[37097]: pgmap v5587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:21:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:35 smithi171 ceph-mon[37789]: pgmap v5588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:35 smithi044 ceph-mon[32404]: pgmap v5588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:35 smithi116 ceph-mon[37097]: pgmap v5588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:37.417 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9640.835950 secs 2024-07-26T11:21:37.418 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:37 smithi171 ceph-mon[37789]: pgmap v5589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9640.835950 secs 2024-07-26T11:21:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:37 smithi044 ceph-mon[32404]: pgmap v5589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9640.835950 secs 2024-07-26T11:21:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:37 smithi116 ceph-mon[37097]: pgmap v5589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:39 smithi044 ceph-mon[32404]: pgmap v5590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:39 smithi116 ceph-mon[37097]: pgmap v5590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:39.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:39 smithi171 ceph-mon[37789]: pgmap v5590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:41 smithi044 ceph-mon[32404]: pgmap v5591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:41 smithi116 ceph-mon[37097]: pgmap v5591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:41 smithi171 ceph-mon[37789]: pgmap v5591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9645.836104 secs 2024-07-26T11:21:42.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9645.836104 secs 2024-07-26T11:21:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9645.836104 secs 2024-07-26T11:21:43.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:43 smithi044 ceph-mon[32404]: pgmap v5592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:43.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:43 smithi116 ceph-mon[37097]: pgmap v5592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:43 smithi171 ceph-mon[37789]: pgmap v5592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:43.625+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:21:43.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:43.625+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:21:43.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:43.625+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:45 smithi044 ceph-mon[32404]: pgmap v5593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:45 smithi116 ceph-mon[37097]: pgmap v5593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:45 smithi171 ceph-mon[37789]: pgmap v5593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9650.836262 secs 2024-07-26T11:21:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:47 smithi044 ceph-mon[32404]: pgmap v5594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9650.836262 secs 2024-07-26T11:21:47.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:47 smithi116 ceph-mon[37097]: pgmap v5594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9650.836262 secs 2024-07-26T11:21:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:47 smithi171 ceph-mon[37789]: pgmap v5594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:49 smithi044 ceph-mon[32404]: pgmap v5595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:49 smithi116 ceph-mon[37097]: pgmap v5595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:49 smithi171 ceph-mon[37789]: pgmap v5595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:51 smithi044 ceph-mon[32404]: pgmap v5596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:51 smithi116 ceph-mon[37097]: pgmap v5596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:51 smithi171 ceph-mon[37789]: pgmap v5596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9655.836432 secs 2024-07-26T11:21:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9655.836432 secs 2024-07-26T11:21:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9655.836432 secs 2024-07-26T11:21:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:53 smithi044 ceph-mon[32404]: pgmap v5597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:53 smithi116 ceph-mon[37097]: pgmap v5597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:53 smithi171 ceph-mon[37789]: pgmap v5597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:55 smithi044 ceph-mon[32404]: pgmap v5598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:55 smithi116 ceph-mon[37097]: pgmap v5598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:55 smithi171 ceph-mon[37789]: pgmap v5598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:21:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9660.836564 secs 2024-07-26T11:21:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:57 smithi044 ceph-mon[32404]: pgmap v5599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9660.836564 secs 2024-07-26T11:21:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:57 smithi116 ceph-mon[37097]: pgmap v5599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9660.836564 secs 2024-07-26T11:21:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:57 smithi171 ceph-mon[37789]: pgmap v5599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:21:58.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:58.633+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:21:58.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:58.633+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:58.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:21:58.633+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:21:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:21:59 smithi044 ceph-mon[32404]: pgmap v5600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:21:59 smithi116 ceph-mon[37097]: pgmap v5600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:21:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:21:59 smithi171 ceph-mon[37789]: pgmap v5600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:01 smithi044 ceph-mon[32404]: pgmap v5601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:01 smithi116 ceph-mon[37097]: pgmap v5601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:01 smithi171 ceph-mon[37789]: pgmap v5601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9665.836735 secs 2024-07-26T11:22:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9665.836735 secs 2024-07-26T11:22:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9665.836735 secs 2024-07-26T11:22:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:03 smithi044 ceph-mon[32404]: pgmap v5602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:03 smithi116 ceph-mon[37097]: pgmap v5602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:03 smithi171 ceph-mon[37789]: pgmap v5602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:05 smithi044 ceph-mon[32404]: pgmap v5603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:05 smithi116 ceph-mon[37097]: pgmap v5603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:05 smithi171 ceph-mon[37789]: pgmap v5603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9670.836880 secs 2024-07-26T11:22:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9670.836880 secs 2024-07-26T11:22:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9670.836880 secs 2024-07-26T11:22:07.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:07 smithi171 ceph-mon[37789]: pgmap v5604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:07 smithi044 ceph-mon[32404]: pgmap v5604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:07 smithi116 ceph-mon[37097]: pgmap v5604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:09 smithi044 ceph-mon[32404]: pgmap v5605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:09 smithi116 ceph-mon[37097]: pgmap v5605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:09 smithi171 ceph-mon[37789]: pgmap v5605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:11 smithi044 ceph-mon[32404]: pgmap v5606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9675.837026 secs 2024-07-26T11:22:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:11 smithi116 ceph-mon[37097]: pgmap v5606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9675.837026 secs 2024-07-26T11:22:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:11 smithi171 ceph-mon[37789]: pgmap v5606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9675.837026 secs 2024-07-26T11:22:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:13 smithi044 ceph-mon[32404]: pgmap v5607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:13 smithi116 ceph-mon[37097]: pgmap v5607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:13 smithi171 ceph-mon[37789]: pgmap v5607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:13.636+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:22:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:13.637+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:22:13.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:13.637+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:15 smithi044 ceph-mon[32404]: pgmap v5608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:15 smithi116 ceph-mon[37097]: pgmap v5608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:15 smithi171 ceph-mon[37789]: pgmap v5608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9680.837186 secs 2024-07-26T11:22:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9680.837186 secs 2024-07-26T11:22:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9680.837186 secs 2024-07-26T11:22:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:17 smithi044 ceph-mon[32404]: pgmap v5609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:17 smithi116 ceph-mon[37097]: pgmap v5609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:17 smithi171 ceph-mon[37789]: pgmap v5609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:19.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:19 smithi116 ceph-mon[37097]: pgmap v5610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:19 smithi171 ceph-mon[37789]: pgmap v5610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:19 smithi044 ceph-mon[32404]: pgmap v5610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:21 smithi171 ceph-mon[37789]: pgmap v5611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9685.837324 secs 2024-07-26T11:22:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:21 smithi044 ceph-mon[32404]: pgmap v5611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9685.837324 secs 2024-07-26T11:22:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:21 smithi116 ceph-mon[37097]: pgmap v5611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:21.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9685.837324 secs 2024-07-26T11:22:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:23 smithi171 ceph-mon[37789]: pgmap v5612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:23.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:23.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:22:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:23 smithi044 ceph-mon[32404]: pgmap v5612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:23.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:22:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:23 smithi116 ceph-mon[37097]: pgmap v5612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:23.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:22:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:24 smithi171 ceph-mon[37789]: pgmap v5613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:24 smithi044 ceph-mon[32404]: pgmap v5613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:24 smithi116 ceph-mon[37097]: pgmap v5613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:26 smithi171 ceph-mon[37789]: pgmap v5614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9690.837413 secs 2024-07-26T11:22:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:26 smithi044 ceph-mon[32404]: pgmap v5614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9690.837413 secs 2024-07-26T11:22:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:26 smithi116 ceph-mon[37097]: pgmap v5614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9690.837413 secs 2024-07-26T11:22:28.575 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:28 smithi116 ceph-mon[37097]: pgmap v5615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:28 smithi171 ceph-mon[37789]: pgmap v5615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:28.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:28.646+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:22:28.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:28.647+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:22:28.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:28.647+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:28.649 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:28 smithi044 ceph-mon[32404]: pgmap v5615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:30 smithi171 ceph-mon[37789]: pgmap v5616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:30 smithi044 ceph-mon[32404]: pgmap v5616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:30 smithi116 ceph-mon[37097]: pgmap v5616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9695.837559 secs 2024-07-26T11:22:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9695.837559 secs 2024-07-26T11:22:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9695.837559 secs 2024-07-26T11:22:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:32 smithi171 ceph-mon[37789]: pgmap v5617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:32 smithi044 ceph-mon[32404]: pgmap v5617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:32 smithi116 ceph-mon[37097]: pgmap v5617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:22:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:34 smithi171 ceph-mon[37789]: pgmap v5618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:34 smithi044 ceph-mon[32404]: pgmap v5618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:34 smithi116 ceph-mon[37097]: pgmap v5618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:22:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:36 smithi171 ceph-mon[37789]: pgmap v5619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9700.837703 secs 2024-07-26T11:22:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:36 smithi044 ceph-mon[32404]: pgmap v5619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9700.837703 secs 2024-07-26T11:22:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:36 smithi116 ceph-mon[37097]: pgmap v5619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:36.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9700.837703 secs 2024-07-26T11:22:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:38 smithi044 ceph-mon[32404]: pgmap v5620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:38 smithi116 ceph-mon[37097]: pgmap v5620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:38 smithi171 ceph-mon[37789]: pgmap v5620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:40 smithi044 ceph-mon[32404]: pgmap v5621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:40 smithi116 ceph-mon[37097]: pgmap v5621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:40 smithi171 ceph-mon[37789]: pgmap v5621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9705.837858 secs 2024-07-26T11:22:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9705.837858 secs 2024-07-26T11:22:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9705.837858 secs 2024-07-26T11:22:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:42 smithi044 ceph-mon[32404]: pgmap v5622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:42 smithi116 ceph-mon[37097]: pgmap v5622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:42 smithi171 ceph-mon[37789]: pgmap v5622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:43.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:43.652+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:22:43.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:43.652+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:22:43.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:43.652+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:44 smithi044 ceph-mon[32404]: pgmap v5623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:44 smithi116 ceph-mon[37097]: pgmap v5623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:44 smithi171 ceph-mon[37789]: pgmap v5623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:22:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:46 smithi044 ceph-mon[32404]: pgmap v5624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:46.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9710.838020 secs 2024-07-26T11:22:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:46 smithi116 ceph-mon[37097]: pgmap v5624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9710.838020 secs 2024-07-26T11:22:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:46 smithi171 ceph-mon[37789]: pgmap v5624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9710.838020 secs 2024-07-26T11:22:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:48 smithi044 ceph-mon[32404]: pgmap v5625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:48 smithi116 ceph-mon[37097]: pgmap v5625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:48 smithi171 ceph-mon[37789]: pgmap v5625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:22:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:50 smithi044 ceph-mon[32404]: pgmap v5626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:50 smithi116 ceph-mon[37097]: pgmap v5626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:50 smithi171 ceph-mon[37789]: pgmap v5626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:22:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9715.838080 secs 2024-07-26T11:22:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9715.838080 secs 2024-07-26T11:22:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9715.838080 secs 2024-07-26T11:22:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:52 smithi044 ceph-mon[32404]: pgmap v5627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:52 smithi116 ceph-mon[37097]: pgmap v5627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:52 smithi171 ceph-mon[37789]: pgmap v5627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:22:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:54 smithi044 ceph-mon[32404]: pgmap v5628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:22:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:54 smithi116 ceph-mon[37097]: pgmap v5628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:22:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:54 smithi171 ceph-mon[37789]: pgmap v5628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:22:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:56 smithi044 ceph-mon[32404]: pgmap v5629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:22:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9720.838223 secs 2024-07-26T11:22:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:56 smithi116 ceph-mon[37097]: pgmap v5629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:22:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9720.838223 secs 2024-07-26T11:22:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:56 smithi171 ceph-mon[37789]: pgmap v5629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:22:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9720.838223 secs 2024-07-26T11:22:58.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:58.661+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:22:58.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:58.661+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:58.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:22:58.661+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:22:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:22:58 smithi044 ceph-mon[32404]: pgmap v5630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:22:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:22:58 smithi116 ceph-mon[37097]: pgmap v5630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:22:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:22:58 smithi171 ceph-mon[37789]: pgmap v5630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:23:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:00 smithi044 ceph-mon[32404]: pgmap v5631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:23:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:00 smithi116 ceph-mon[37097]: pgmap v5631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:23:01.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:00 smithi171 ceph-mon[37789]: pgmap v5631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:23:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9725.838384 secs 2024-07-26T11:23:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:01 smithi044 ceph-mon[32404]: pgmap v5632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:23:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9725.838384 secs 2024-07-26T11:23:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:01 smithi116 ceph-mon[37097]: pgmap v5632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:23:02.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9725.838384 secs 2024-07-26T11:23:02.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:01 smithi171 ceph-mon[37789]: pgmap v5632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:23:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:04 smithi171 ceph-mon[37789]: pgmap v5633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:23:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:04 smithi044 ceph-mon[32404]: pgmap v5633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:23:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:04 smithi116 ceph-mon[37097]: pgmap v5633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:23:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:06 smithi171 ceph-mon[37789]: pgmap v5634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9730.838529 secs 2024-07-26T11:23:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:06 smithi044 ceph-mon[32404]: pgmap v5634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:06.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9730.838529 secs 2024-07-26T11:23:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:06 smithi116 ceph-mon[37097]: pgmap v5634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9730.838529 secs 2024-07-26T11:23:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:08 smithi171 ceph-mon[37789]: pgmap v5635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:08 smithi044 ceph-mon[32404]: pgmap v5635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:08 smithi116 ceph-mon[37097]: pgmap v5635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:10 smithi171 ceph-mon[37789]: pgmap v5636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:10 smithi044 ceph-mon[32404]: pgmap v5636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:10 smithi116 ceph-mon[37097]: pgmap v5636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9735.838688 secs 2024-07-26T11:23:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9735.838688 secs 2024-07-26T11:23:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9735.838688 secs 2024-07-26T11:23:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:12 smithi171 ceph-mon[37789]: pgmap v5637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:12 smithi044 ceph-mon[32404]: pgmap v5637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:12 smithi116 ceph-mon[37097]: pgmap v5637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:13.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:13.667+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:23:13.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:13.667+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:13.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:13.667+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:23:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:14 smithi171 ceph-mon[37789]: pgmap v5638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:23:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:14 smithi044 ceph-mon[32404]: pgmap v5638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:23:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:14 smithi116 ceph-mon[37097]: pgmap v5638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:23:16.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:16.418+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:23:16.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:16.418+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:23:16.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:16.419+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:23:16.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:16.419+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:23:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:16 smithi171 ceph-mon[37789]: pgmap v5639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9740.838841 secs 2024-07-26T11:23:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:16 smithi044 ceph-mon[32404]: pgmap v5639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:16.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9740.838841 secs 2024-07-26T11:23:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:16 smithi116 ceph-mon[37097]: pgmap v5639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9740.838841 secs 2024-07-26T11:23:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:18 smithi171 ceph-mon[37789]: pgmap v5640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:18 smithi044 ceph-mon[32404]: pgmap v5640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:18 smithi116 ceph-mon[37097]: pgmap v5640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:20 smithi171 ceph-mon[37789]: pgmap v5641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:20 smithi044 ceph-mon[32404]: pgmap v5641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:20 smithi116 ceph-mon[37097]: pgmap v5641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9745.838982 secs 2024-07-26T11:23:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9745.838982 secs 2024-07-26T11:23:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9745.838982 secs 2024-07-26T11:23:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:22 smithi044 ceph-mon[32404]: pgmap v5642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:23:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:22 smithi116 ceph-mon[37097]: pgmap v5642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:23:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:22 smithi171 ceph-mon[37789]: pgmap v5642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:23:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:24 smithi044 ceph-mon[32404]: pgmap v5643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:23:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:24 smithi116 ceph-mon[37097]: pgmap v5643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:23:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:24 smithi171 ceph-mon[37789]: pgmap v5643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:23:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:25 smithi044 ceph-mon[32404]: pgmap v5644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:25.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:23:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:25 smithi116 ceph-mon[37097]: pgmap v5644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:25.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:23:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:25.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:25 smithi171 ceph-mon[37789]: pgmap v5644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:25.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:23:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9750.839152 secs 2024-07-26T11:23:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9750.839152 secs 2024-07-26T11:23:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9750.839152 secs 2024-07-26T11:23:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:27 smithi044 ceph-mon[32404]: pgmap v5645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:23:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:27 smithi116 ceph-mon[37097]: pgmap v5645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:23:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:27 smithi171 ceph-mon[37789]: pgmap v5645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:23:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:28.677+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:23:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:28.677+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:28.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:28.677+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:23:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:29 smithi044 ceph-mon[32404]: pgmap v5646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:23:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:29 smithi116 ceph-mon[37097]: pgmap v5646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:23:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:29 smithi171 ceph-mon[37789]: pgmap v5646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:23:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:31 smithi116 ceph-mon[37097]: pgmap v5647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9755.839307 secs 2024-07-26T11:23:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:31 smithi171 ceph-mon[37789]: pgmap v5647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9755.839307 secs 2024-07-26T11:23:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:31 smithi044 ceph-mon[32404]: pgmap v5647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9755.839307 secs 2024-07-26T11:23:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:33 smithi171 ceph-mon[37789]: pgmap v5648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:23:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:33 smithi044 ceph-mon[32404]: pgmap v5648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:23:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:33 smithi116 ceph-mon[37097]: pgmap v5648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:23:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:35 smithi171 ceph-mon[37789]: pgmap v5649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:35 smithi044 ceph-mon[32404]: pgmap v5649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:35 smithi116 ceph-mon[37097]: pgmap v5649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:23:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9760.839474 secs 2024-07-26T11:23:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9760.839474 secs 2024-07-26T11:23:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9760.839474 secs 2024-07-26T11:23:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:37 smithi171 ceph-mon[37789]: pgmap v5650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:37 smithi044 ceph-mon[32404]: pgmap v5650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:37 smithi116 ceph-mon[37097]: pgmap v5650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:39 smithi171 ceph-mon[37789]: pgmap v5651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:39 smithi044 ceph-mon[32404]: pgmap v5651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:39 smithi116 ceph-mon[37097]: pgmap v5651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:41 smithi171 ceph-mon[37789]: pgmap v5652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9765.839668 secs 2024-07-26T11:23:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:41 smithi044 ceph-mon[32404]: pgmap v5652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:42.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9765.839668 secs 2024-07-26T11:23:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:41 smithi116 ceph-mon[37097]: pgmap v5652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9765.839668 secs 2024-07-26T11:23:43.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:43.682+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:23:43.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:43.682+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:23:43.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:43.682+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:43 smithi171 ceph-mon[37789]: pgmap v5653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:43 smithi044 ceph-mon[32404]: pgmap v5653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:43 smithi116 ceph-mon[37097]: pgmap v5653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:45.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:45 smithi044 ceph-mon[32404]: pgmap v5654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:45 smithi171 ceph-mon[37789]: pgmap v5654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:46.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:45 smithi116 ceph-mon[37097]: pgmap v5654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9770.839782 secs 2024-07-26T11:23:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9770.839782 secs 2024-07-26T11:23:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9770.839782 secs 2024-07-26T11:23:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:47 smithi171 ceph-mon[37789]: pgmap v5655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:47 smithi044 ceph-mon[32404]: pgmap v5655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:47 smithi116 ceph-mon[37097]: pgmap v5655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:49 smithi171 ceph-mon[37789]: pgmap v5656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:49 smithi044 ceph-mon[32404]: pgmap v5656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:49 smithi116 ceph-mon[37097]: pgmap v5656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:23:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:51 smithi044 ceph-mon[32404]: pgmap v5657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9775.839961 secs 2024-07-26T11:23:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:51 smithi116 ceph-mon[37097]: pgmap v5657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9775.839961 secs 2024-07-26T11:23:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:51 smithi171 ceph-mon[37789]: pgmap v5657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:23:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9775.839961 secs 2024-07-26T11:23:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:53 smithi044 ceph-mon[32404]: pgmap v5658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:53 smithi116 ceph-mon[37097]: pgmap v5658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:53 smithi171 ceph-mon[37789]: pgmap v5658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:23:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:55 smithi044 ceph-mon[32404]: pgmap v5659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:55 smithi116 ceph-mon[37097]: pgmap v5659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:55 smithi171 ceph-mon[37789]: pgmap v5659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:23:56.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9780.840069 secs 2024-07-26T11:23:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9780.840069 secs 2024-07-26T11:23:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9780.840069 secs 2024-07-26T11:23:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:57 smithi044 ceph-mon[32404]: pgmap v5660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:57 smithi116 ceph-mon[37097]: pgmap v5660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:57 smithi171 ceph-mon[37789]: pgmap v5660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:23:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:58.692+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:23:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:58.692+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:23:58.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:23:58.692+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:23:59 smithi044 ceph-mon[32404]: pgmap v5661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:23:59 smithi116 ceph-mon[37097]: pgmap v5661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:23:59 smithi171 ceph-mon[37789]: pgmap v5661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:01 smithi044 ceph-mon[32404]: pgmap v5662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9785.840243 secs 2024-07-26T11:24:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:01 smithi116 ceph-mon[37097]: pgmap v5662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:02.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9785.840243 secs 2024-07-26T11:24:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:01 smithi171 ceph-mon[37789]: pgmap v5662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9785.840243 secs 2024-07-26T11:24:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:03 smithi044 ceph-mon[32404]: pgmap v5663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:03 smithi116 ceph-mon[37097]: pgmap v5663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:03 smithi171 ceph-mon[37789]: pgmap v5663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:05 smithi044 ceph-mon[32404]: pgmap v5664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:05 smithi116 ceph-mon[37097]: pgmap v5664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:05 smithi171 ceph-mon[37789]: pgmap v5664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9790.840424 secs 2024-07-26T11:24:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9790.840424 secs 2024-07-26T11:24:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9790.840424 secs 2024-07-26T11:24:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:07 smithi044 ceph-mon[32404]: pgmap v5665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:07 smithi116 ceph-mon[37097]: pgmap v5665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:07 smithi171 ceph-mon[37789]: pgmap v5665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:09 smithi044 ceph-mon[32404]: pgmap v5666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:09 smithi116 ceph-mon[37097]: pgmap v5666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:09 smithi171 ceph-mon[37789]: pgmap v5666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:11 smithi044 ceph-mon[32404]: pgmap v5667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9795.840527 secs 2024-07-26T11:24:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:11 smithi116 ceph-mon[37097]: pgmap v5667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9795.840527 secs 2024-07-26T11:24:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:11 smithi171 ceph-mon[37789]: pgmap v5667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9795.840527 secs 2024-07-26T11:24:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:13.697+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:24:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:13.697+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:24:13.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:13.697+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:13 smithi044 ceph-mon[32404]: pgmap v5668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:13 smithi116 ceph-mon[37097]: pgmap v5668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:13 smithi171 ceph-mon[37789]: pgmap v5668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:15 smithi044 ceph-mon[32404]: pgmap v5669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:15 smithi116 ceph-mon[37097]: pgmap v5669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:15 smithi171 ceph-mon[37789]: pgmap v5669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9800.840691 secs 2024-07-26T11:24:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9800.840691 secs 2024-07-26T11:24:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9800.840691 secs 2024-07-26T11:24:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:17 smithi044 ceph-mon[32404]: pgmap v5670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:17 smithi116 ceph-mon[37097]: pgmap v5670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:17 smithi171 ceph-mon[37789]: pgmap v5670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:19 smithi044 ceph-mon[32404]: pgmap v5671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:19 smithi116 ceph-mon[37097]: pgmap v5671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:19 smithi171 ceph-mon[37789]: pgmap v5671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:21 smithi044 ceph-mon[32404]: pgmap v5672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9805.840871 secs 2024-07-26T11:24:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:21 smithi116 ceph-mon[37097]: pgmap v5672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9805.840871 secs 2024-07-26T11:24:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:21 smithi171 ceph-mon[37789]: pgmap v5672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9805.840871 secs 2024-07-26T11:24:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:23 smithi044 ceph-mon[32404]: pgmap v5673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:23 smithi116 ceph-mon[37097]: pgmap v5673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:23 smithi171 ceph-mon[37789]: pgmap v5673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:25 smithi044 ceph-mon[32404]: pgmap v5674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:25 smithi116 ceph-mon[37097]: pgmap v5674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:25 smithi171 ceph-mon[37789]: pgmap v5674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9810.840978 secs 2024-07-26T11:24:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:26 smithi044 ceph-mon[32404]: pgmap v5675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:24:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:26 smithi044 ceph-mon[32404]: pgmap v5676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:24:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:24:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9810.840978 secs 2024-07-26T11:24:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:26 smithi116 ceph-mon[37097]: pgmap v5675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:24:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:26 smithi116 ceph-mon[37097]: pgmap v5676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:24:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:24:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9810.840978 secs 2024-07-26T11:24:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:26 smithi171 ceph-mon[37789]: pgmap v5675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:24:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:26 smithi171 ceph-mon[37789]: pgmap v5676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:24:27.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:24:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:27 smithi171 ceph-mon[37789]: pgmap v5677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:24:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:27 smithi044 ceph-mon[32404]: pgmap v5677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:24:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:27 smithi116 ceph-mon[37097]: pgmap v5677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:24:28.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:28.707+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:24:28.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:28.707+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:24:28.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:28.707+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:30.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:29 smithi116 ceph-mon[37097]: pgmap v5678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:24:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:29 smithi171 ceph-mon[37789]: pgmap v5678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:24:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:29 smithi044 ceph-mon[32404]: pgmap v5678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:24:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9815.841095 secs 2024-07-26T11:24:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9815.841095 secs 2024-07-26T11:24:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9815.841095 secs 2024-07-26T11:24:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:32 smithi171 ceph-mon[37789]: pgmap v5679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:24:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:32 smithi044 ceph-mon[32404]: pgmap v5679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:24:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:32 smithi116 ceph-mon[37097]: pgmap v5679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:24:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:33 smithi171 ceph-mon[37789]: pgmap v5680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:24:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:33 smithi044 ceph-mon[32404]: pgmap v5680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:24:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:33 smithi116 ceph-mon[37097]: pgmap v5680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:24:36.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:35 smithi171 ceph-mon[37789]: pgmap v5681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:24:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:35 smithi044 ceph-mon[32404]: pgmap v5681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:24:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:35 smithi116 ceph-mon[37097]: pgmap v5681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:24:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9820.841234 secs 2024-07-26T11:24:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9820.841234 secs 2024-07-26T11:24:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9820.841234 secs 2024-07-26T11:24:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:37 smithi171 ceph-mon[37789]: pgmap v5682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:37 smithi044 ceph-mon[32404]: pgmap v5682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:37 smithi116 ceph-mon[37097]: pgmap v5682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:39 smithi171 ceph-mon[37789]: pgmap v5683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:39 smithi044 ceph-mon[32404]: pgmap v5683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:39 smithi116 ceph-mon[37097]: pgmap v5683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9825.841397 secs 2024-07-26T11:24:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9825.841397 secs 2024-07-26T11:24:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9825.841397 secs 2024-07-26T11:24:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:42 smithi171 ceph-mon[37789]: pgmap v5684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:42 smithi044 ceph-mon[32404]: pgmap v5684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:42 smithi116 ceph-mon[37097]: pgmap v5684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:43.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:43.711+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:24:43.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:43.711+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:43.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:43.711+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:24:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:43 smithi044 ceph-mon[32404]: pgmap v5685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:43 smithi171 ceph-mon[37789]: pgmap v5685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:44.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:43 smithi116 ceph-mon[37097]: pgmap v5685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:24:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:45 smithi171 ceph-mon[37789]: pgmap v5686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:45 smithi044 ceph-mon[32404]: pgmap v5686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:45 smithi116 ceph-mon[37097]: pgmap v5686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:24:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9830.841556 secs 2024-07-26T11:24:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9830.841556 secs 2024-07-26T11:24:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9830.841556 secs 2024-07-26T11:24:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:47 smithi044 ceph-mon[32404]: pgmap v5687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:47 smithi116 ceph-mon[37097]: pgmap v5687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:47 smithi171 ceph-mon[37789]: pgmap v5687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:24:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:49 smithi171 ceph-mon[37789]: pgmap v5688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:49 smithi044 ceph-mon[32404]: pgmap v5688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:49 smithi116 ceph-mon[37097]: pgmap v5688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:24:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9835.841723 secs 2024-07-26T11:24:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9835.841723 secs 2024-07-26T11:24:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9835.841723 secs 2024-07-26T11:24:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:52 smithi171 ceph-mon[37789]: pgmap v5689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:52 smithi044 ceph-mon[32404]: pgmap v5689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:52 smithi116 ceph-mon[37097]: pgmap v5689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:24:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:53 smithi171 ceph-mon[37789]: pgmap v5690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:24:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:53 smithi044 ceph-mon[32404]: pgmap v5690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:24:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:53 smithi116 ceph-mon[37097]: pgmap v5690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:24:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:55 smithi171 ceph-mon[37789]: pgmap v5691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:24:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:55 smithi044 ceph-mon[32404]: pgmap v5691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:24:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:55 smithi116 ceph-mon[37097]: pgmap v5691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:24:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9840.841887 secs 2024-07-26T11:24:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9840.841887 secs 2024-07-26T11:24:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9840.841887 secs 2024-07-26T11:24:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:57 smithi171 ceph-mon[37789]: pgmap v5692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:24:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:57 smithi044 ceph-mon[32404]: pgmap v5692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:24:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:57 smithi116 ceph-mon[37097]: pgmap v5692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:24:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:58.721+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:24:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:58.721+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:24:58.721+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:25:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:24:59 smithi171 ceph-mon[37789]: pgmap v5693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:24:59 smithi044 ceph-mon[32404]: pgmap v5693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:24:59 smithi116 ceph-mon[37097]: pgmap v5693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9845.842034 secs 2024-07-26T11:25:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9845.842034 secs 2024-07-26T11:25:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9845.842034 secs 2024-07-26T11:25:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:02 smithi171 ceph-mon[37789]: pgmap v5694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:25:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:02 smithi044 ceph-mon[32404]: pgmap v5694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:25:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:02 smithi116 ceph-mon[37097]: pgmap v5694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:25:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:04 smithi171 ceph-mon[37789]: pgmap v5695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:25:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:04 smithi044 ceph-mon[32404]: pgmap v5695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:25:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:04 smithi116 ceph-mon[37097]: pgmap v5695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:25:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9850.842186 secs 2024-07-26T11:25:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:06 smithi171 ceph-mon[37789]: pgmap v5696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9850.842186 secs 2024-07-26T11:25:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:06 smithi044 ceph-mon[32404]: pgmap v5696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9850.842186 secs 2024-07-26T11:25:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:06 smithi116 ceph-mon[37097]: pgmap v5696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:08 smithi044 ceph-mon[32404]: pgmap v5697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:08 smithi116 ceph-mon[37097]: pgmap v5697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:08 smithi171 ceph-mon[37789]: pgmap v5697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:11.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:10 smithi171 ceph-mon[37789]: pgmap v5698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:10 smithi044 ceph-mon[32404]: pgmap v5698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:10 smithi116 ceph-mon[37097]: pgmap v5698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9855.842352 secs 2024-07-26T11:25:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9855.842352 secs 2024-07-26T11:25:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9855.842352 secs 2024-07-26T11:25:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:12 smithi044 ceph-mon[32404]: pgmap v5699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:12 smithi116 ceph-mon[37097]: pgmap v5699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:12 smithi171 ceph-mon[37789]: pgmap v5699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:13.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:13.724+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:25:13.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:13.724+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:13.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:13.724+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:25:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:14 smithi044 ceph-mon[32404]: pgmap v5700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:14 smithi116 ceph-mon[37097]: pgmap v5700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:14 smithi171 ceph-mon[37789]: pgmap v5700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9860.842516 secs 2024-07-26T11:25:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:16 smithi044 ceph-mon[32404]: pgmap v5701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9860.842516 secs 2024-07-26T11:25:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:16 smithi116 ceph-mon[37097]: pgmap v5701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9860.842516 secs 2024-07-26T11:25:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:16 smithi171 ceph-mon[37789]: pgmap v5701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:18 smithi044 ceph-mon[32404]: pgmap v5702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:18 smithi116 ceph-mon[37097]: pgmap v5702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:18 smithi171 ceph-mon[37789]: pgmap v5702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:20 smithi044 ceph-mon[32404]: pgmap v5703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:20 smithi116 ceph-mon[37097]: pgmap v5703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:20 smithi171 ceph-mon[37789]: pgmap v5703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:25:22.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9865.842683 secs 2024-07-26T11:25:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9865.842683 secs 2024-07-26T11:25:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9865.842683 secs 2024-07-26T11:25:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:22 smithi044 ceph-mon[32404]: pgmap v5704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:22 smithi116 ceph-mon[37097]: pgmap v5704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:22 smithi171 ceph-mon[37789]: pgmap v5704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:25:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:24 smithi044 ceph-mon[32404]: pgmap v5705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:24 smithi116 ceph-mon[37097]: pgmap v5705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:24 smithi171 ceph-mon[37789]: pgmap v5705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:25:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9870.842848 secs 2024-07-26T11:25:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:26 smithi044 ceph-mon[32404]: pgmap v5706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9870.842848 secs 2024-07-26T11:25:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:26 smithi116 ceph-mon[37097]: pgmap v5706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:27.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9870.842848 secs 2024-07-26T11:25:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:26 smithi171 ceph-mon[37789]: pgmap v5706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:27 smithi044 ceph-mon[32404]: pgmap v5707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:25:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:27 smithi044 ceph-mon[32404]: pgmap v5708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:25:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:27 smithi116 ceph-mon[37097]: pgmap v5707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:25:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:27 smithi116 ceph-mon[37097]: pgmap v5708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:28.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:25:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:27 smithi171 ceph-mon[37789]: pgmap v5707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:25:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:27 smithi171 ceph-mon[37789]: pgmap v5708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:25:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:28.734+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:25:28.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:28.734+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:28.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:28.734+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:25:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:30 smithi044 ceph-mon[32404]: pgmap v5709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:30 smithi116 ceph-mon[37097]: pgmap v5709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:30 smithi171 ceph-mon[37789]: pgmap v5709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9875.842997 secs 2024-07-26T11:25:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:32 smithi044 ceph-mon[32404]: pgmap v5710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9875.842997 secs 2024-07-26T11:25:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:32 smithi116 ceph-mon[37097]: pgmap v5710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9875.842997 secs 2024-07-26T11:25:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:32 smithi171 ceph-mon[37789]: pgmap v5710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:25:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:34 smithi044 ceph-mon[32404]: pgmap v5711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:34 smithi116 ceph-mon[37097]: pgmap v5711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:34 smithi171 ceph-mon[37789]: pgmap v5711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:25:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:36 smithi044 ceph-mon[32404]: pgmap v5712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:25:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:36 smithi116 ceph-mon[37097]: pgmap v5712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:25:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:36 smithi171 ceph-mon[37789]: pgmap v5712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:25:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9880.843144 secs 2024-07-26T11:25:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9880.843144 secs 2024-07-26T11:25:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9880.843144 secs 2024-07-26T11:25:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:38 smithi044 ceph-mon[32404]: pgmap v5713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:38 smithi116 ceph-mon[37097]: pgmap v5713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:38 smithi171 ceph-mon[37789]: pgmap v5713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:25:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:40 smithi044 ceph-mon[32404]: pgmap v5714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:40 smithi116 ceph-mon[37097]: pgmap v5714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:40 smithi171 ceph-mon[37789]: pgmap v5714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9885.843307 secs 2024-07-26T11:25:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:42 smithi044 ceph-mon[32404]: pgmap v5715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:25:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9885.843307 secs 2024-07-26T11:25:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:42 smithi116 ceph-mon[37097]: pgmap v5715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:25:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9885.843307 secs 2024-07-26T11:25:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:42 smithi171 ceph-mon[37789]: pgmap v5715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:25:43.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:43.739+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:25:43.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:43.739+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:25:43.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:43.739+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:44 smithi044 ceph-mon[32404]: pgmap v5716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:25:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:44 smithi116 ceph-mon[37097]: pgmap v5716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:25:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:44 smithi171 ceph-mon[37789]: pgmap v5716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:25:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:46 smithi044 ceph-mon[32404]: pgmap v5717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:25:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:46 smithi116 ceph-mon[37097]: pgmap v5717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:25:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:46 smithi171 ceph-mon[37789]: pgmap v5717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:25:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9890.843476 secs 2024-07-26T11:25:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9890.843476 secs 2024-07-26T11:25:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9890.843476 secs 2024-07-26T11:25:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:48 smithi044 ceph-mon[32404]: pgmap v5718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:48 smithi116 ceph-mon[37097]: pgmap v5718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:48 smithi171 ceph-mon[37789]: pgmap v5718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:50 smithi044 ceph-mon[32404]: pgmap v5719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:50 smithi116 ceph-mon[37097]: pgmap v5719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:50 smithi171 ceph-mon[37789]: pgmap v5719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9895.843677 secs 2024-07-26T11:25:52.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:52 smithi044 ceph-mon[32404]: pgmap v5720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9895.843677 secs 2024-07-26T11:25:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:52 smithi116 ceph-mon[37097]: pgmap v5720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9895.843677 secs 2024-07-26T11:25:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:52 smithi171 ceph-mon[37789]: pgmap v5720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:54 smithi044 ceph-mon[32404]: pgmap v5721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:25:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:54 smithi116 ceph-mon[37097]: pgmap v5721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:25:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:54 smithi171 ceph-mon[37789]: pgmap v5721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:25:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:56 smithi044 ceph-mon[32404]: pgmap v5722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9900.843844 secs 2024-07-26T11:25:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:56 smithi116 ceph-mon[37097]: pgmap v5722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9900.843844 secs 2024-07-26T11:25:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:56 smithi171 ceph-mon[37789]: pgmap v5722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:25:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9900.843844 secs 2024-07-26T11:25:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:25:58 smithi044 ceph-mon[32404]: pgmap v5723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:25:58 smithi116 ceph-mon[37097]: pgmap v5723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:25:58 smithi171 ceph-mon[37789]: pgmap v5723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:25:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:58.748+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:25:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:58.749+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:25:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:25:58.749+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:00 smithi044 ceph-mon[32404]: pgmap v5724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:00 smithi116 ceph-mon[37097]: pgmap v5724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:00 smithi171 ceph-mon[37789]: pgmap v5724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9905.843984 secs 2024-07-26T11:26:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9905.843984 secs 2024-07-26T11:26:01.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9905.843984 secs 2024-07-26T11:26:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:02 smithi171 ceph-mon[37789]: pgmap v5725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:02 smithi044 ceph-mon[32404]: pgmap v5725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:02 smithi116 ceph-mon[37097]: pgmap v5725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:04 smithi171 ceph-mon[37789]: pgmap v5726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:04 smithi044 ceph-mon[32404]: pgmap v5726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:04 smithi116 ceph-mon[37097]: pgmap v5726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:06 smithi171 ceph-mon[37789]: pgmap v5727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9910.844133 secs 2024-07-26T11:26:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:06 smithi044 ceph-mon[32404]: pgmap v5727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9910.844133 secs 2024-07-26T11:26:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:06 smithi116 ceph-mon[37097]: pgmap v5727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9910.844133 secs 2024-07-26T11:26:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:08 smithi171 ceph-mon[37789]: pgmap v5728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:08 smithi044 ceph-mon[32404]: pgmap v5728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:08 smithi116 ceph-mon[37097]: pgmap v5728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:10 smithi171 ceph-mon[37789]: pgmap v5729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:10 smithi044 ceph-mon[32404]: pgmap v5729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:10 smithi116 ceph-mon[37097]: pgmap v5729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9915.844279 secs 2024-07-26T11:26:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9915.844279 secs 2024-07-26T11:26:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9915.844279 secs 2024-07-26T11:26:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:12 smithi171 ceph-mon[37789]: pgmap v5730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:12 smithi044 ceph-mon[32404]: pgmap v5730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:12 smithi116 ceph-mon[37097]: pgmap v5730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:13.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:13.750+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:26:13.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:13.750+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:26:13.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:13.750+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:14 smithi171 ceph-mon[37789]: pgmap v5731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:26:14.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T11:26:14.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:26:14.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58074070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:26:14.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58074070 comp rx=0 tx=0).stop 2024-07-26T11:26:14.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T11:26:14.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:26:14.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:14.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.715+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:14.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:14 smithi044 ceph-mon[32404]: pgmap v5731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:26:14.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.716+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:14.719+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf5807c000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:14 smithi116 ceph-mon[37097]: pgmap v5731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:26:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.450+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:26:16.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.450+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:26:16.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.451+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:26:16.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.451+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:26:16.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.451+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:26:16.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.451+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:26:16.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.451+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:26:16.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:16.452+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:26:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:16 smithi171 ceph-mon[37789]: pgmap v5732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:26:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9920.844465 secs 2024-07-26T11:26:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:16 smithi044 ceph-mon[32404]: pgmap v5732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:26:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9920.844465 secs 2024-07-26T11:26:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:16 smithi116 ceph-mon[37097]: pgmap v5732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:26:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9920.844465 secs 2024-07-26T11:26:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:18 smithi171 ceph-mon[37789]: pgmap v5733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:18 smithi044 ceph-mon[32404]: pgmap v5733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:18 smithi116 ceph-mon[37097]: pgmap v5733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:20 smithi171 ceph-mon[37789]: pgmap v5734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:20 smithi044 ceph-mon[32404]: pgmap v5734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:20 smithi116 ceph-mon[37097]: pgmap v5734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:26:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9925.844696 secs 2024-07-26T11:26:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9925.844696 secs 2024-07-26T11:26:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9925.844696 secs 2024-07-26T11:26:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:22 smithi044 ceph-mon[32404]: pgmap v5735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:26:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:22 smithi116 ceph-mon[37097]: pgmap v5735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:26:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:22 smithi171 ceph-mon[37789]: pgmap v5735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:26:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:24 smithi044 ceph-mon[32404]: pgmap v5736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:26:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:24 smithi116 ceph-mon[37097]: pgmap v5736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:26:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:24 smithi171 ceph-mon[37789]: pgmap v5736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:26:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:26 smithi044 ceph-mon[32404]: pgmap v5737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9930.844809 secs 2024-07-26T11:26:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:26 smithi116 ceph-mon[37097]: pgmap v5737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9930.844809 secs 2024-07-26T11:26:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:26 smithi171 ceph-mon[37789]: pgmap v5737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9930.844809 secs 2024-07-26T11:26:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:28.755+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:26:28.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:28.755+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:26:28.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:28.755+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: pgmap v5738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.758 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: pgmap v5738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: pgmap v5738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: pgmap v5739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: pgmap v5740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:29.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:26:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: pgmap v5739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: pgmap v5740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:29.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:26:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:26:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:26:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:26:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:29.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:29.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: pgmap v5739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:26:29.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: pgmap v5740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:29.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:26:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:31 smithi044 ceph-mon[32404]: pgmap v5741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9935.844990 secs 2024-07-26T11:26:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:31 smithi116 ceph-mon[37097]: pgmap v5741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9935.844990 secs 2024-07-26T11:26:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:31 smithi171 ceph-mon[37789]: pgmap v5741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9935.844990 secs 2024-07-26T11:26:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:33 smithi044 ceph-mon[32404]: pgmap v5742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:33 smithi116 ceph-mon[37097]: pgmap v5742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:33 smithi171 ceph-mon[37789]: pgmap v5742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:35 smithi044 ceph-mon[32404]: pgmap v5743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:35 smithi116 ceph-mon[37097]: pgmap v5743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:35 smithi171 ceph-mon[37789]: pgmap v5743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:26:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9940.845142 secs 2024-07-26T11:26:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9940.845142 secs 2024-07-26T11:26:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9940.845142 secs 2024-07-26T11:26:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:37 smithi044 ceph-mon[32404]: pgmap v5744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:37 smithi116 ceph-mon[37097]: pgmap v5744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:37 smithi171 ceph-mon[37789]: pgmap v5744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:26:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:39 smithi044 ceph-mon[32404]: pgmap v5745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:26:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:39 smithi116 ceph-mon[37097]: pgmap v5745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:26:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:39 smithi171 ceph-mon[37789]: pgmap v5745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:26:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:40 smithi171 ceph-mon[37789]: pgmap v5746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:40 smithi044 ceph-mon[32404]: pgmap v5746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:40 smithi116 ceph-mon[37097]: pgmap v5746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9945.845315 secs 2024-07-26T11:26:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9945.845315 secs 2024-07-26T11:26:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9945.845315 secs 2024-07-26T11:26:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:42 smithi171 ceph-mon[37789]: pgmap v5747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:42 smithi044 ceph-mon[32404]: pgmap v5747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:42 smithi116 ceph-mon[37097]: pgmap v5747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:43.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:43.760+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:26:43.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:43.760+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:26:43.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:43.760+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:45 smithi044 ceph-mon[32404]: pgmap v5748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:45 smithi116 ceph-mon[37097]: pgmap v5748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:45 smithi171 ceph-mon[37789]: pgmap v5748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9950.845500 secs 2024-07-26T11:26:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9950.845500 secs 2024-07-26T11:26:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9950.845500 secs 2024-07-26T11:26:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:47 smithi044 ceph-mon[32404]: pgmap v5749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:47 smithi116 ceph-mon[37097]: pgmap v5749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:47 smithi171 ceph-mon[37789]: pgmap v5749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:49 smithi044 ceph-mon[32404]: pgmap v5750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:49 smithi116 ceph-mon[37097]: pgmap v5750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:49 smithi171 ceph-mon[37789]: pgmap v5750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:50 smithi171 ceph-mon[37789]: pgmap v5751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:50 smithi044 ceph-mon[32404]: pgmap v5751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:50 smithi116 ceph-mon[37097]: pgmap v5751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9955.845699 secs 2024-07-26T11:26:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9955.845699 secs 2024-07-26T11:26:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9955.845699 secs 2024-07-26T11:26:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:52 smithi171 ceph-mon[37789]: pgmap v5752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:52 smithi044 ceph-mon[32404]: pgmap v5752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:52 smithi116 ceph-mon[37097]: pgmap v5752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:26:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:55 smithi044 ceph-mon[32404]: pgmap v5753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:55 smithi116 ceph-mon[37097]: pgmap v5753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:55 smithi171 ceph-mon[37789]: pgmap v5753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:26:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9960.845878 secs 2024-07-26T11:26:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9960.845878 secs 2024-07-26T11:26:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9960.845878 secs 2024-07-26T11:26:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:57 smithi044 ceph-mon[32404]: pgmap v5754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:57 smithi116 ceph-mon[37097]: pgmap v5754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:57 smithi171 ceph-mon[37789]: pgmap v5754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:26:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:58.770+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:26:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:58.770+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:26:58.770+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:26:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:26:59 smithi044 ceph-mon[32404]: pgmap v5755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:26:59 smithi116 ceph-mon[37097]: pgmap v5755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:26:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:26:59 smithi171 ceph-mon[37789]: pgmap v5755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:01 smithi044 ceph-mon[32404]: pgmap v5756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9965.846011 secs 2024-07-26T11:27:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:01 smithi116 ceph-mon[37097]: pgmap v5756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9965.846011 secs 2024-07-26T11:27:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:01 smithi171 ceph-mon[37789]: pgmap v5756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9965.846011 secs 2024-07-26T11:27:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:03 smithi044 ceph-mon[32404]: pgmap v5757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:03 smithi116 ceph-mon[37097]: pgmap v5757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:03 smithi171 ceph-mon[37789]: pgmap v5757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:05.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:05 smithi044 ceph-mon[32404]: pgmap v5758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:05 smithi116 ceph-mon[37097]: pgmap v5758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:05 smithi171 ceph-mon[37789]: pgmap v5758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9970.846163 secs 2024-07-26T11:27:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9970.846163 secs 2024-07-26T11:27:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9970.846163 secs 2024-07-26T11:27:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:07 smithi116 ceph-mon[37097]: pgmap v5759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:27:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:07 smithi171 ceph-mon[37789]: pgmap v5759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:27:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:07 smithi044 ceph-mon[32404]: pgmap v5759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:27:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:09 smithi171 ceph-mon[37789]: pgmap v5760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:09 smithi044 ceph-mon[32404]: pgmap v5760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:09 smithi116 ceph-mon[37097]: pgmap v5760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:11 smithi171 ceph-mon[37789]: pgmap v5761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9975.846322 secs 2024-07-26T11:27:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:11 smithi044 ceph-mon[32404]: pgmap v5761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:12.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9975.846322 secs 2024-07-26T11:27:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:11 smithi116 ceph-mon[37097]: pgmap v5761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9975.846322 secs 2024-07-26T11:27:13.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:13.775+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:27:13.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:13.775+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:13.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:13.775+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:27:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:13 smithi171 ceph-mon[37789]: pgmap v5762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:13 smithi044 ceph-mon[32404]: pgmap v5762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:13 smithi116 ceph-mon[37097]: pgmap v5762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:15 smithi171 ceph-mon[37789]: pgmap v5763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:15 smithi044 ceph-mon[32404]: pgmap v5763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:15 smithi116 ceph-mon[37097]: pgmap v5763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9980.846483 secs 2024-07-26T11:27:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9980.846483 secs 2024-07-26T11:27:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9980.846483 secs 2024-07-26T11:27:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:17 smithi171 ceph-mon[37789]: pgmap v5764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:17 smithi044 ceph-mon[32404]: pgmap v5764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:17 smithi116 ceph-mon[37097]: pgmap v5764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:19 smithi171 ceph-mon[37789]: pgmap v5765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:19 smithi044 ceph-mon[32404]: pgmap v5765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:19 smithi116 ceph-mon[37097]: pgmap v5765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:21 smithi171 ceph-mon[37789]: pgmap v5766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9985.846705 secs 2024-07-26T11:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:21 smithi044 ceph-mon[32404]: pgmap v5766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9985.846705 secs 2024-07-26T11:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:21 smithi116 ceph-mon[37097]: pgmap v5766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9985.846705 secs 2024-07-26T11:27:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:23 smithi171 ceph-mon[37789]: pgmap v5767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:23 smithi044 ceph-mon[32404]: pgmap v5767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:23 smithi116 ceph-mon[37097]: pgmap v5767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:25 smithi171 ceph-mon[37789]: pgmap v5768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:25 smithi044 ceph-mon[32404]: pgmap v5768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:25 smithi116 ceph-mon[37097]: pgmap v5768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9990.846879 secs 2024-07-26T11:27:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9990.846879 secs 2024-07-26T11:27:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9990.846879 secs 2024-07-26T11:27:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:27 smithi044 ceph-mon[32404]: pgmap v5769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:28.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:27 smithi116 ceph-mon[37097]: pgmap v5769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:27 smithi171 ceph-mon[37789]: pgmap v5769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:28.785 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:28.783+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:27:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:28.783+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:27:28.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:28.783+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: pgmap v5770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: pgmap v5770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: pgmap v5770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:27:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:27:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: pgmap v5771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: pgmap v5772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:31.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:27:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:27:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: pgmap v5771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: pgmap v5772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:27:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:27:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:27:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:27:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:27:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:31.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: pgmap v5771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:27:31.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: pgmap v5772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:31.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:27:31.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 9995.846986 secs 2024-07-26T11:27:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 9995.846986 secs 2024-07-26T11:27:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 9995.846986 secs 2024-07-26T11:27:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:32 smithi044 ceph-mon[32404]: pgmap v5773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:27:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:32 smithi116 ceph-mon[37097]: pgmap v5773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:27:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:32 smithi171 ceph-mon[37789]: pgmap v5773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:27:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:34 smithi044 ceph-mon[32404]: pgmap v5774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:34 smithi116 ceph-mon[37097]: pgmap v5774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:34 smithi171 ceph-mon[37789]: pgmap v5774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10000.847134 secs 2024-07-26T11:27:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:36 smithi044 ceph-mon[32404]: pgmap v5775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10000.847134 secs 2024-07-26T11:27:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:36 smithi116 ceph-mon[37097]: pgmap v5775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10000.847134 secs 2024-07-26T11:27:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:36 smithi171 ceph-mon[37789]: pgmap v5775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:27:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:38 smithi044 ceph-mon[32404]: pgmap v5776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:27:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:38 smithi116 ceph-mon[37097]: pgmap v5776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:27:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:38 smithi171 ceph-mon[37789]: pgmap v5776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:27:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:40 smithi044 ceph-mon[32404]: pgmap v5777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:27:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:40 smithi116 ceph-mon[37097]: pgmap v5777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:27:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:40 smithi171 ceph-mon[37789]: pgmap v5777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:27:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10005.847303 secs 2024-07-26T11:27:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10005.847303 secs 2024-07-26T11:27:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10005.847303 secs 2024-07-26T11:27:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:42 smithi171 ceph-mon[37789]: pgmap v5778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:42 smithi116 ceph-mon[37097]: pgmap v5778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:43.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:43.785+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:27:43.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:43.785+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:27:43.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:43.785+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:42 smithi044 ceph-mon[32404]: pgmap v5778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:44 smithi171 ceph-mon[37789]: pgmap v5779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:44 smithi044 ceph-mon[32404]: pgmap v5779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:44 smithi116 ceph-mon[37097]: pgmap v5779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10010.847455 secs 2024-07-26T11:27:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:46 smithi171 ceph-mon[37789]: pgmap v5780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10010.847455 secs 2024-07-26T11:27:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:46 smithi044 ceph-mon[32404]: pgmap v5780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10010.847455 secs 2024-07-26T11:27:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:46 smithi116 ceph-mon[37097]: pgmap v5780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:48 smithi171 ceph-mon[37789]: pgmap v5781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:48 smithi044 ceph-mon[32404]: pgmap v5781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:48 smithi116 ceph-mon[37097]: pgmap v5781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:50 smithi171 ceph-mon[37789]: pgmap v5782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:50 smithi044 ceph-mon[32404]: pgmap v5782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:50 smithi116 ceph-mon[37097]: pgmap v5782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:27:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10015.847585 secs 2024-07-26T11:27:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10015.847585 secs 2024-07-26T11:27:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10015.847585 secs 2024-07-26T11:27:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:52 smithi171 ceph-mon[37789]: pgmap v5783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:52 smithi044 ceph-mon[32404]: pgmap v5783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:52 smithi116 ceph-mon[37097]: pgmap v5783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:27:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:53 smithi171 ceph-mon[37789]: pgmap v5784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:53 smithi044 ceph-mon[32404]: pgmap v5784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:53 smithi116 ceph-mon[37097]: pgmap v5784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:27:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10020.847751 secs 2024-07-26T11:27:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:56 smithi044 ceph-mon[32404]: pgmap v5785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10020.847751 secs 2024-07-26T11:27:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:56 smithi116 ceph-mon[37097]: pgmap v5785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10020.847751 secs 2024-07-26T11:27:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:56 smithi171 ceph-mon[37789]: pgmap v5785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:27:58.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:58.788+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:27:58.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:58.788+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:58.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:27:58.788+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:27:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:27:58 smithi116 ceph-mon[37097]: pgmap v5786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:27:58 smithi171 ceph-mon[37789]: pgmap v5786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:27:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:27:58 smithi044 ceph-mon[32404]: pgmap v5786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:00 smithi171 ceph-mon[37789]: pgmap v5787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:00 smithi044 ceph-mon[32404]: pgmap v5787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:00 smithi116 ceph-mon[37097]: pgmap v5787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10025.847909 secs 2024-07-26T11:28:02.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10025.847909 secs 2024-07-26T11:28:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10025.847909 secs 2024-07-26T11:28:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:02 smithi171 ceph-mon[37789]: pgmap v5788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:02 smithi044 ceph-mon[32404]: pgmap v5788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:02 smithi116 ceph-mon[37097]: pgmap v5788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:05.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:04 smithi044 ceph-mon[32404]: pgmap v5789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:04 smithi171 ceph-mon[37789]: pgmap v5789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:05.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:04 smithi116 ceph-mon[37097]: pgmap v5789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:07.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10030.848013 secs 2024-07-26T11:28:07.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:06 smithi171 ceph-mon[37789]: pgmap v5790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10030.848013 secs 2024-07-26T11:28:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:06 smithi044 ceph-mon[32404]: pgmap v5790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10030.848013 secs 2024-07-26T11:28:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:06 smithi116 ceph-mon[37097]: pgmap v5790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:08 smithi171 ceph-mon[37789]: pgmap v5791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:08 smithi044 ceph-mon[32404]: pgmap v5791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:08 smithi116 ceph-mon[37097]: pgmap v5791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:10 smithi171 ceph-mon[37789]: pgmap v5792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:10 smithi044 ceph-mon[32404]: pgmap v5792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:10 smithi116 ceph-mon[37097]: pgmap v5792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10035.848121 secs 2024-07-26T11:28:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10035.848121 secs 2024-07-26T11:28:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10035.848121 secs 2024-07-26T11:28:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:12 smithi171 ceph-mon[37789]: pgmap v5793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:12 smithi044 ceph-mon[32404]: pgmap v5793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:12 smithi116 ceph-mon[37097]: pgmap v5793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:13.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:13.793+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:28:13.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:13.793+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:28:13.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:13.793+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:13 smithi044 ceph-mon[32404]: pgmap v5794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:13 smithi116 ceph-mon[37097]: pgmap v5794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:13 smithi171 ceph-mon[37789]: pgmap v5794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:17.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10040.848283 secs 2024-07-26T11:28:17.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:16 smithi171 ceph-mon[37789]: pgmap v5795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10040.848283 secs 2024-07-26T11:28:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:16 smithi044 ceph-mon[32404]: pgmap v5795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10040.848283 secs 2024-07-26T11:28:17.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:16 smithi116 ceph-mon[37097]: pgmap v5795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:18 smithi171 ceph-mon[37789]: pgmap v5796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:18 smithi044 ceph-mon[32404]: pgmap v5796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:19.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:18 smithi116 ceph-mon[37097]: pgmap v5796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:21.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:20 smithi044 ceph-mon[32404]: pgmap v5797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:20 smithi171 ceph-mon[37789]: pgmap v5797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:20 smithi116 ceph-mon[37097]: pgmap v5797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10045.848438 secs 2024-07-26T11:28:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10045.848438 secs 2024-07-26T11:28:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10045.848438 secs 2024-07-26T11:28:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:22 smithi171 ceph-mon[37789]: pgmap v5798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:22 smithi044 ceph-mon[32404]: pgmap v5798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:22 smithi116 ceph-mon[37097]: pgmap v5798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:28:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:24 smithi171 ceph-mon[37789]: pgmap v5799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:24 smithi044 ceph-mon[32404]: pgmap v5799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:24 smithi116 ceph-mon[37097]: pgmap v5799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:28:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10050.848604 secs 2024-07-26T11:28:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:26 smithi171 ceph-mon[37789]: pgmap v5800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10050.848604 secs 2024-07-26T11:28:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:26 smithi044 ceph-mon[32404]: pgmap v5800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10050.848604 secs 2024-07-26T11:28:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:26 smithi116 ceph-mon[37097]: pgmap v5800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:27 smithi044 ceph-mon[32404]: pgmap v5801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:27 smithi116 ceph-mon[37097]: pgmap v5801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:27 smithi171 ceph-mon[37789]: pgmap v5801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:28.802+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:28:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:28.802+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:28:28.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:28.802+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:30 smithi171 ceph-mon[37789]: pgmap v5802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:30 smithi044 ceph-mon[32404]: pgmap v5802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:30 smithi116 ceph-mon[37097]: pgmap v5802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10055.848679 secs 2024-07-26T11:28:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:28:32.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10055.848679 secs 2024-07-26T11:28:32.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:32.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:32.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:28:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10055.848679 secs 2024-07-26T11:28:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:28:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:32 smithi171 ceph-mon[37789]: pgmap v5803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:28:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:32 smithi171 ceph-mon[37789]: pgmap v5804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:28:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:32 smithi044 ceph-mon[32404]: pgmap v5803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:28:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:32 smithi044 ceph-mon[32404]: pgmap v5804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:28:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:32 smithi116 ceph-mon[37097]: pgmap v5803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:28:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:32 smithi116 ceph-mon[37097]: pgmap v5804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:28:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:34 smithi171 ceph-mon[37789]: pgmap v5805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:34 smithi044 ceph-mon[32404]: pgmap v5805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:34 smithi116 ceph-mon[37097]: pgmap v5805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:37.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:36 smithi044 ceph-mon[32404]: pgmap v5806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:37.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10060.848842 secs 2024-07-26T11:28:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:36 smithi171 ceph-mon[37789]: pgmap v5806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10060.848842 secs 2024-07-26T11:28:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:36 smithi116 ceph-mon[37097]: pgmap v5806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:37.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10060.848842 secs 2024-07-26T11:28:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:38 smithi171 ceph-mon[37789]: pgmap v5807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:38 smithi044 ceph-mon[32404]: pgmap v5807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:38 smithi116 ceph-mon[37097]: pgmap v5807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:39 smithi044 ceph-mon[32404]: pgmap v5808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:39 smithi116 ceph-mon[37097]: pgmap v5808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:39 smithi171 ceph-mon[37789]: pgmap v5808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:28:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10065.848984 secs 2024-07-26T11:28:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:42 smithi044 ceph-mon[32404]: pgmap v5809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10065.848984 secs 2024-07-26T11:28:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:42 smithi116 ceph-mon[37097]: pgmap v5809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10065.848984 secs 2024-07-26T11:28:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:42 smithi171 ceph-mon[37789]: pgmap v5809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:28:43.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:43.807+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:28:43.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:43.807+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:28:43.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:43.807+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:44 smithi044 ceph-mon[32404]: pgmap v5810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:44 smithi116 ceph-mon[37097]: pgmap v5810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:44 smithi171 ceph-mon[37789]: pgmap v5810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:46 smithi116 ceph-mon[37097]: pgmap v5811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:46 smithi171 ceph-mon[37789]: pgmap v5811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:46 smithi044 ceph-mon[32404]: pgmap v5811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10070.849133 secs 2024-07-26T11:28:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10070.849133 secs 2024-07-26T11:28:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10070.849133 secs 2024-07-26T11:28:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:48 smithi171 ceph-mon[37789]: pgmap v5812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:48 smithi044 ceph-mon[32404]: pgmap v5812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:48 smithi116 ceph-mon[37097]: pgmap v5812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:50 smithi171 ceph-mon[37789]: pgmap v5813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:50 smithi044 ceph-mon[32404]: pgmap v5813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:50 smithi116 ceph-mon[37097]: pgmap v5813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10075.849236 secs 2024-07-26T11:28:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:52 smithi171 ceph-mon[37789]: pgmap v5814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10075.849236 secs 2024-07-26T11:28:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:52 smithi044 ceph-mon[32404]: pgmap v5814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10075.849236 secs 2024-07-26T11:28:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:52 smithi116 ceph-mon[37097]: pgmap v5814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:28:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:54 smithi171 ceph-mon[37789]: pgmap v5815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:54 smithi044 ceph-mon[32404]: pgmap v5815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:54 smithi116 ceph-mon[37097]: pgmap v5815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:56 smithi171 ceph-mon[37789]: pgmap v5816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:56 smithi044 ceph-mon[32404]: pgmap v5816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:56 smithi116 ceph-mon[37097]: pgmap v5816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:28:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10080.849381 secs 2024-07-26T11:28:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10080.849381 secs 2024-07-26T11:28:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10080.849381 secs 2024-07-26T11:28:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:28:58 smithi171 ceph-mon[37789]: pgmap v5817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:28:58 smithi044 ceph-mon[32404]: pgmap v5817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:28:58 smithi116 ceph-mon[37097]: pgmap v5817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:28:58.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:58.817+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:28:58.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:58.817+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:58.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:28:58.817+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:29:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:00 smithi171 ceph-mon[37789]: pgmap v5818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:00 smithi044 ceph-mon[32404]: pgmap v5818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:00 smithi116 ceph-mon[37097]: pgmap v5818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10085.849523 secs 2024-07-26T11:29:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:02 smithi044 ceph-mon[32404]: pgmap v5819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10085.849523 secs 2024-07-26T11:29:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:02 smithi116 ceph-mon[37097]: pgmap v5819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10085.849523 secs 2024-07-26T11:29:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:02 smithi171 ceph-mon[37789]: pgmap v5819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:04 smithi044 ceph-mon[32404]: pgmap v5820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:04 smithi116 ceph-mon[37097]: pgmap v5820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:04 smithi171 ceph-mon[37789]: pgmap v5820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:06 smithi044 ceph-mon[32404]: pgmap v5821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:06 smithi116 ceph-mon[37097]: pgmap v5821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:06 smithi171 ceph-mon[37789]: pgmap v5821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10090.849675 secs 2024-07-26T11:29:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10090.849675 secs 2024-07-26T11:29:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10090.849675 secs 2024-07-26T11:29:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:08 smithi044 ceph-mon[32404]: pgmap v5822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:08 smithi116 ceph-mon[37097]: pgmap v5822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:08 smithi171 ceph-mon[37789]: pgmap v5822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:10 smithi044 ceph-mon[32404]: pgmap v5823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:10 smithi116 ceph-mon[37097]: pgmap v5823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:10 smithi171 ceph-mon[37789]: pgmap v5823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10095.849767 secs 2024-07-26T11:29:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:12 smithi044 ceph-mon[32404]: pgmap v5824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10095.849767 secs 2024-07-26T11:29:12.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:12 smithi116 ceph-mon[37097]: pgmap v5824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10095.849767 secs 2024-07-26T11:29:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:12 smithi171 ceph-mon[37789]: pgmap v5824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:13.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:13.821+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:29:13.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:13.822+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:29:13.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:13.822+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:14 smithi044 ceph-mon[32404]: pgmap v5825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:14 smithi116 ceph-mon[37097]: pgmap v5825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:14 smithi171 ceph-mon[37789]: pgmap v5825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:16.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:16.484+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:29:16.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:16.484+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:29:16.487 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:16 smithi044 ceph-mon[32404]: pgmap v5826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:16.484+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:29:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:16.485+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:29:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:16 smithi116 ceph-mon[37097]: pgmap v5826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:16 smithi171 ceph-mon[37789]: pgmap v5826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:17.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10100.849920 secs 2024-07-26T11:29:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10100.849920 secs 2024-07-26T11:29:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10100.849920 secs 2024-07-26T11:29:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:18 smithi044 ceph-mon[32404]: pgmap v5827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:18 smithi116 ceph-mon[37097]: pgmap v5827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:18 smithi171 ceph-mon[37789]: pgmap v5827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:20 smithi044 ceph-mon[32404]: pgmap v5828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:20 smithi116 ceph-mon[37097]: pgmap v5828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:20 smithi171 ceph-mon[37789]: pgmap v5828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10105.850056 secs 2024-07-26T11:29:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:22 smithi044 ceph-mon[32404]: pgmap v5829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10105.850056 secs 2024-07-26T11:29:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:22 smithi116 ceph-mon[37097]: pgmap v5829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10105.850056 secs 2024-07-26T11:29:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:22 smithi171 ceph-mon[37789]: pgmap v5829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:24 smithi044 ceph-mon[32404]: pgmap v5830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:24 smithi116 ceph-mon[37097]: pgmap v5830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:24 smithi171 ceph-mon[37789]: pgmap v5830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:26 smithi044 ceph-mon[32404]: pgmap v5831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:26 smithi116 ceph-mon[37097]: pgmap v5831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:26 smithi171 ceph-mon[37789]: pgmap v5831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10110.850216 secs 2024-07-26T11:29:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10110.850216 secs 2024-07-26T11:29:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10110.850216 secs 2024-07-26T11:29:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:28 smithi044 ceph-mon[32404]: pgmap v5832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:28 smithi116 ceph-mon[37097]: pgmap v5832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:28 smithi171 ceph-mon[37789]: pgmap v5832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:28.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:28.831+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:29:28.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:28.831+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:29:28.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:28.831+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:30 smithi044 ceph-mon[32404]: pgmap v5833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:30 smithi116 ceph-mon[37097]: pgmap v5833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:30 smithi171 ceph-mon[37789]: pgmap v5833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10115.850325 secs 2024-07-26T11:29:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:32 smithi044 ceph-mon[32404]: pgmap v5834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10115.850325 secs 2024-07-26T11:29:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:32 smithi116 ceph-mon[37097]: pgmap v5834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10115.850325 secs 2024-07-26T11:29:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:32 smithi171 ceph-mon[37789]: pgmap v5834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:33.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:29:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:34 smithi044 ceph-mon[32404]: pgmap v5835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:34 smithi044 ceph-mon[32404]: pgmap v5836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:34 smithi116 ceph-mon[37097]: pgmap v5835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:34 smithi116 ceph-mon[37097]: pgmap v5836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:34 smithi171 ceph-mon[37789]: pgmap v5835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:34 smithi171 ceph-mon[37789]: pgmap v5836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:35 smithi044 ceph-mon[32404]: pgmap v5837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:35 smithi116 ceph-mon[37097]: pgmap v5837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:35 smithi171 ceph-mon[37789]: pgmap v5837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10120.850467 secs 2024-07-26T11:29:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10120.850467 secs 2024-07-26T11:29:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10120.850467 secs 2024-07-26T11:29:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:37 smithi044 ceph-mon[32404]: pgmap v5838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:37 smithi116 ceph-mon[37097]: pgmap v5838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:37 smithi171 ceph-mon[37789]: pgmap v5838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:39 smithi044 ceph-mon[32404]: pgmap v5839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:39 smithi116 ceph-mon[37097]: pgmap v5839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:39 smithi171 ceph-mon[37789]: pgmap v5839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10125.850578 secs 2024-07-26T11:29:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:41 smithi044 ceph-mon[32404]: pgmap v5840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10125.850578 secs 2024-07-26T11:29:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:41 smithi116 ceph-mon[37097]: pgmap v5840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10125.850578 secs 2024-07-26T11:29:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:41 smithi171 ceph-mon[37789]: pgmap v5840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:29:43.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:43.836+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:29:43.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:43.836+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:29:43.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:43.836+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:43 smithi044 ceph-mon[32404]: pgmap v5841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:29:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:43 smithi116 ceph-mon[37097]: pgmap v5841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:29:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:43 smithi171 ceph-mon[37789]: pgmap v5841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:29:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:45 smithi044 ceph-mon[32404]: pgmap v5842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:45 smithi116 ceph-mon[37097]: pgmap v5842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:45 smithi171 ceph-mon[37789]: pgmap v5842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:29:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10130.850722 secs 2024-07-26T11:29:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10130.850722 secs 2024-07-26T11:29:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10130.850722 secs 2024-07-26T11:29:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:47 smithi044 ceph-mon[32404]: pgmap v5843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:47 smithi116 ceph-mon[37097]: pgmap v5843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:47 smithi171 ceph-mon[37789]: pgmap v5843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:49 smithi044 ceph-mon[32404]: pgmap v5844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:50 smithi116 ceph-mon[37097]: pgmap v5844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:50 smithi171 ceph-mon[37789]: pgmap v5844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:29:52.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10135.850875 secs 2024-07-26T11:29:52.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:52 smithi044 ceph-mon[32404]: pgmap v5845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10135.850875 secs 2024-07-26T11:29:52.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:52 smithi116 ceph-mon[37097]: pgmap v5845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10135.850875 secs 2024-07-26T11:29:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:52 smithi171 ceph-mon[37789]: pgmap v5845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:29:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:54 smithi044 ceph-mon[32404]: pgmap v5846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:29:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:54 smithi116 ceph-mon[37097]: pgmap v5846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:29:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:54 smithi171 ceph-mon[37789]: pgmap v5846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:29:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:56 smithi116 ceph-mon[37097]: pgmap v5847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:29:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:56 smithi171 ceph-mon[37789]: pgmap v5847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:29:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:56 smithi044 ceph-mon[32404]: pgmap v5847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:29:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10140.851030 secs 2024-07-26T11:29:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10140.851030 secs 2024-07-26T11:29:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10140.851030 secs 2024-07-26T11:29:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:29:58 smithi171 ceph-mon[37789]: pgmap v5848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:29:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:29:58 smithi044 ceph-mon[32404]: pgmap v5848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:29:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:29:58 smithi116 ceph-mon[37097]: pgmap v5848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:29:58.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:58.846+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:29:58.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:58.846+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:58.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:29:58.846+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:30:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:00 smithi171 ceph-mon[37789]: pgmap v5849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:00 smithi044 ceph-mon[32404]: pgmap v5849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:00.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:00 smithi116 ceph-mon[37097]: pgmap v5849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:30:02.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10145.851203 secs 2024-07-26T11:30:02.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:02 smithi116 ceph-mon[37097]: pgmap v5850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10145.851203 secs 2024-07-26T11:30:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:02 smithi171 ceph-mon[37789]: pgmap v5850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10145.851203 secs 2024-07-26T11:30:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:02 smithi044 ceph-mon[32404]: pgmap v5850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:04 smithi171 ceph-mon[37789]: pgmap v5851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:04 smithi044 ceph-mon[32404]: pgmap v5851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:04 smithi116 ceph-mon[37097]: pgmap v5851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:06 smithi171 ceph-mon[37789]: pgmap v5852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:06 smithi044 ceph-mon[32404]: pgmap v5852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:06 smithi116 ceph-mon[37097]: pgmap v5852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10150.851347 secs 2024-07-26T11:30:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10150.851347 secs 2024-07-26T11:30:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10150.851347 secs 2024-07-26T11:30:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:08 smithi171 ceph-mon[37789]: pgmap v5853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:08 smithi044 ceph-mon[32404]: pgmap v5853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:08 smithi116 ceph-mon[37097]: pgmap v5853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:10 smithi044 ceph-mon[32404]: pgmap v5854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:10 smithi171 ceph-mon[37789]: pgmap v5854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:10 smithi116 ceph-mon[37097]: pgmap v5854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10155.851508 secs 2024-07-26T11:30:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:12 smithi171 ceph-mon[37789]: pgmap v5855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10155.851508 secs 2024-07-26T11:30:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:12 smithi044 ceph-mon[32404]: pgmap v5855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10155.851508 secs 2024-07-26T11:30:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:12 smithi116 ceph-mon[37097]: pgmap v5855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:13.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:13.851+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:30:13.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:13.851+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:30:13.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:13.851+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:14 smithi044 ceph-mon[32404]: pgmap v5856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:14 smithi116 ceph-mon[37097]: pgmap v5856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:14 smithi171 ceph-mon[37789]: pgmap v5856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:16 smithi044 ceph-mon[32404]: pgmap v5857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:16 smithi116 ceph-mon[37097]: pgmap v5857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:16 smithi171 ceph-mon[37789]: pgmap v5857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:17.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10160.851682 secs 2024-07-26T11:30:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10160.851682 secs 2024-07-26T11:30:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10160.851682 secs 2024-07-26T11:30:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:18 smithi044 ceph-mon[32404]: pgmap v5858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:18 smithi116 ceph-mon[37097]: pgmap v5858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:18 smithi171 ceph-mon[37789]: pgmap v5858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:20 smithi044 ceph-mon[32404]: pgmap v5859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:20 smithi116 ceph-mon[37097]: pgmap v5859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:20 smithi171 ceph-mon[37789]: pgmap v5859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10165.851816 secs 2024-07-26T11:30:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:22 smithi044 ceph-mon[32404]: pgmap v5860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10165.851816 secs 2024-07-26T11:30:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:22 smithi116 ceph-mon[37097]: pgmap v5860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10165.851816 secs 2024-07-26T11:30:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:22 smithi171 ceph-mon[37789]: pgmap v5860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:24 smithi044 ceph-mon[32404]: pgmap v5861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:24 smithi116 ceph-mon[37097]: pgmap v5861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:24 smithi171 ceph-mon[37789]: pgmap v5861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:26 smithi044 ceph-mon[32404]: pgmap v5862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:26 smithi116 ceph-mon[37097]: pgmap v5862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:26 smithi171 ceph-mon[37789]: pgmap v5862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:30:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10170.851948 secs 2024-07-26T11:30:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10170.851948 secs 2024-07-26T11:30:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10170.851948 secs 2024-07-26T11:30:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:28 smithi044 ceph-mon[32404]: pgmap v5863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:28 smithi116 ceph-mon[37097]: pgmap v5863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:28 smithi171 ceph-mon[37789]: pgmap v5863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:28.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:28.862+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:30:28.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:28.862+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:28.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:28.862+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:30:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:30 smithi044 ceph-mon[32404]: pgmap v5864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:30 smithi116 ceph-mon[37097]: pgmap v5864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:30 smithi171 ceph-mon[37789]: pgmap v5864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10175.852113 secs 2024-07-26T11:30:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:32 smithi044 ceph-mon[32404]: pgmap v5865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10175.852113 secs 2024-07-26T11:30:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:32 smithi116 ceph-mon[37097]: pgmap v5865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10175.852113 secs 2024-07-26T11:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:32 smithi171 ceph-mon[37789]: pgmap v5865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:30:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:34 smithi044 ceph-mon[32404]: pgmap v5866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:34 smithi116 ceph-mon[37097]: pgmap v5866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:34 smithi171 ceph-mon[37789]: pgmap v5866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:30:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: pgmap v5867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: pgmap v5867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: pgmap v5867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:36.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10180.852242 secs 2024-07-26T11:30:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10180.852242 secs 2024-07-26T11:30:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10180.852242 secs 2024-07-26T11:30:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:38 smithi171 ceph-mon[37789]: pgmap v5868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:39.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:38 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:39.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:38 smithi044 ceph-mon[32404]: pgmap v5868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:38 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:38 smithi116 ceph-mon[37097]: pgmap v5868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:38 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:30:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:39 smithi171 ceph-mon[37789]: pgmap v5869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:30:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:39 smithi171 ceph-mon[37789]: pgmap v5870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:39 smithi044 ceph-mon[32404]: pgmap v5869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:30:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:39 smithi044 ceph-mon[32404]: pgmap v5870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:39 smithi116 ceph-mon[37097]: pgmap v5869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:30:40.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:39 smithi116 ceph-mon[37097]: pgmap v5870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:40 smithi044 ceph-mon[32404]: pgmap v5871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:40 smithi116 ceph-mon[37097]: pgmap v5871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:40 smithi171 ceph-mon[37789]: pgmap v5871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10185.852397 secs 2024-07-26T11:30:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10185.852397 secs 2024-07-26T11:30:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10185.852397 secs 2024-07-26T11:30:43.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:42 smithi044 ceph-mon[32404]: pgmap v5872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:42 smithi116 ceph-mon[37097]: pgmap v5872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:42 smithi171 ceph-mon[37789]: pgmap v5872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:43.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:43.867+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:30:43.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:43.867+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:30:43.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:43.867+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:45 smithi171 ceph-mon[37789]: pgmap v5873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:45 smithi044 ceph-mon[32404]: pgmap v5873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:45 smithi116 ceph-mon[37097]: pgmap v5873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10190.852576 secs 2024-07-26T11:30:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10190.852576 secs 2024-07-26T11:30:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10190.852576 secs 2024-07-26T11:30:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:47 smithi171 ceph-mon[37789]: pgmap v5874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:30:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:47 smithi044 ceph-mon[32404]: pgmap v5874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:30:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:47 smithi116 ceph-mon[37097]: pgmap v5874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:30:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:49 smithi171 ceph-mon[37789]: pgmap v5875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:49 smithi044 ceph-mon[32404]: pgmap v5875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:49 smithi116 ceph-mon[37097]: pgmap v5875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:30:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:50 smithi044 ceph-mon[32404]: pgmap v5876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:50 smithi116 ceph-mon[37097]: pgmap v5876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:50 smithi171 ceph-mon[37789]: pgmap v5876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10195.852762 secs 2024-07-26T11:30:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10195.852762 secs 2024-07-26T11:30:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10195.852762 secs 2024-07-26T11:30:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:52 smithi044 ceph-mon[32404]: pgmap v5877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:52 smithi116 ceph-mon[37097]: pgmap v5877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:52 smithi171 ceph-mon[37789]: pgmap v5877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:30:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:55 smithi171 ceph-mon[37789]: pgmap v5878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:55 smithi044 ceph-mon[32404]: pgmap v5878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:55 smithi116 ceph-mon[37097]: pgmap v5878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:30:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10200.852930 secs 2024-07-26T11:30:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10200.852930 secs 2024-07-26T11:30:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10200.852930 secs 2024-07-26T11:30:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:57 smithi171 ceph-mon[37789]: pgmap v5879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:57 smithi044 ceph-mon[32404]: pgmap v5879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:57 smithi116 ceph-mon[37097]: pgmap v5879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:30:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:58.877+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:30:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:58.877+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:30:58.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:30:58.877+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:30:59 smithi171 ceph-mon[37789]: pgmap v5880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:30:59 smithi044 ceph-mon[32404]: pgmap v5880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:30:59 smithi116 ceph-mon[37097]: pgmap v5880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:01 smithi171 ceph-mon[37789]: pgmap v5881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10205.853089 secs 2024-07-26T11:31:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:01 smithi044 ceph-mon[32404]: pgmap v5881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:02.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10205.853089 secs 2024-07-26T11:31:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:01 smithi116 ceph-mon[37097]: pgmap v5881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10205.853089 secs 2024-07-26T11:31:03.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:03 smithi116 ceph-mon[37097]: pgmap v5882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:03 smithi171 ceph-mon[37789]: pgmap v5882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:03 smithi044 ceph-mon[32404]: pgmap v5882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:05 smithi171 ceph-mon[37789]: pgmap v5883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:05 smithi044 ceph-mon[32404]: pgmap v5883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:05 smithi116 ceph-mon[37097]: pgmap v5883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10210.853230 secs 2024-07-26T11:31:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10210.853230 secs 2024-07-26T11:31:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10210.853230 secs 2024-07-26T11:31:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:07 smithi044 ceph-mon[32404]: pgmap v5884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:07 smithi116 ceph-mon[37097]: pgmap v5884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:07 smithi171 ceph-mon[37789]: pgmap v5884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:09 smithi044 ceph-mon[32404]: pgmap v5885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:09 smithi116 ceph-mon[37097]: pgmap v5885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:09 smithi171 ceph-mon[37789]: pgmap v5885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:11 smithi044 ceph-mon[32404]: pgmap v5886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10215.853410 secs 2024-07-26T11:31:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:11 smithi116 ceph-mon[37097]: pgmap v5886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10215.853410 secs 2024-07-26T11:31:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:11 smithi171 ceph-mon[37789]: pgmap v5886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10215.853410 secs 2024-07-26T11:31:13.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:13.882+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:31:13.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:13.882+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:31:13.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:13.882+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:13.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:13 smithi044 ceph-mon[32404]: pgmap v5887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:13 smithi116 ceph-mon[37097]: pgmap v5887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:13 smithi171 ceph-mon[37789]: pgmap v5887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:15 smithi044 ceph-mon[32404]: pgmap v5888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:15 smithi116 ceph-mon[37097]: pgmap v5888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:15 smithi171 ceph-mon[37789]: pgmap v5888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10220.853548 secs 2024-07-26T11:31:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10220.853548 secs 2024-07-26T11:31:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10220.853548 secs 2024-07-26T11:31:17.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:17 smithi171 ceph-mon[37789]: pgmap v5889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:17 smithi044 ceph-mon[32404]: pgmap v5889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:17 smithi116 ceph-mon[37097]: pgmap v5889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:19 smithi044 ceph-mon[32404]: pgmap v5890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:19 smithi116 ceph-mon[37097]: pgmap v5890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:19 smithi171 ceph-mon[37789]: pgmap v5890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:21 smithi044 ceph-mon[32404]: pgmap v5891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10225.853714 secs 2024-07-26T11:31:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:21 smithi116 ceph-mon[37097]: pgmap v5891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10225.853714 secs 2024-07-26T11:31:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:21 smithi171 ceph-mon[37789]: pgmap v5891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10225.853714 secs 2024-07-26T11:31:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:23 smithi044 ceph-mon[32404]: pgmap v5892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:23 smithi116 ceph-mon[37097]: pgmap v5892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:23 smithi171 ceph-mon[37789]: pgmap v5892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:25 smithi044 ceph-mon[32404]: pgmap v5893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:25 smithi116 ceph-mon[37097]: pgmap v5893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:25 smithi171 ceph-mon[37789]: pgmap v5893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10230.853856 secs 2024-07-26T11:31:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10230.853856 secs 2024-07-26T11:31:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10230.853856 secs 2024-07-26T11:31:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:27 smithi044 ceph-mon[32404]: pgmap v5894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:27 smithi116 ceph-mon[37097]: pgmap v5894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:27 smithi171 ceph-mon[37789]: pgmap v5894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:28.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:28.892+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:31:28.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:28.892+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:28.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:28.892+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:31:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:29 smithi044 ceph-mon[32404]: pgmap v5895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:29 smithi116 ceph-mon[37097]: pgmap v5895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:29 smithi171 ceph-mon[37789]: pgmap v5895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:31 smithi044 ceph-mon[32404]: pgmap v5896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10235.853961 secs 2024-07-26T11:31:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:31 smithi116 ceph-mon[37097]: pgmap v5896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10235.853961 secs 2024-07-26T11:31:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:31 smithi171 ceph-mon[37789]: pgmap v5896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10235.853961 secs 2024-07-26T11:31:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:33 smithi044 ceph-mon[32404]: pgmap v5897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:33 smithi116 ceph-mon[37097]: pgmap v5897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:33 smithi171 ceph-mon[37789]: pgmap v5897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:31:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:35 smithi044 ceph-mon[32404]: pgmap v5898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:35 smithi116 ceph-mon[37097]: pgmap v5898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:35 smithi171 ceph-mon[37789]: pgmap v5898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10240.854125 secs 2024-07-26T11:31:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10240.854125 secs 2024-07-26T11:31:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10240.854125 secs 2024-07-26T11:31:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:37 smithi044 ceph-mon[32404]: pgmap v5899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:37 smithi116 ceph-mon[37097]: pgmap v5899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:37 smithi171 ceph-mon[37789]: pgmap v5899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:39 smithi044 ceph-mon[32404]: pgmap v5900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:39 smithi116 ceph-mon[37097]: pgmap v5900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:39 smithi171 ceph-mon[37789]: pgmap v5900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:40 smithi044 ceph-mon[32404]: pgmap v5901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:31:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:40 smithi044 ceph-mon[32404]: pgmap v5902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:41.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:31:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:40 smithi116 ceph-mon[37097]: pgmap v5901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:31:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:40 smithi116 ceph-mon[37097]: pgmap v5902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:41.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:31:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:40 smithi171 ceph-mon[37789]: pgmap v5901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:31:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:40 smithi171 ceph-mon[37789]: pgmap v5902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:31:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10245.854291 secs 2024-07-26T11:31:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10245.854291 secs 2024-07-26T11:31:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10245.854291 secs 2024-07-26T11:31:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:42 smithi044 ceph-mon[32404]: pgmap v5903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:42 smithi116 ceph-mon[37097]: pgmap v5903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:42 smithi171 ceph-mon[37789]: pgmap v5903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:43.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:43.897+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:31:43.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:43.897+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:43.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:43.897+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:31:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:44 smithi171 ceph-mon[37789]: pgmap v5904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:44 smithi044 ceph-mon[32404]: pgmap v5904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:44 smithi116 ceph-mon[37097]: pgmap v5904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10250.854415 secs 2024-07-26T11:31:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:46 smithi171 ceph-mon[37789]: pgmap v5905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10250.854415 secs 2024-07-26T11:31:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:46 smithi044 ceph-mon[32404]: pgmap v5905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10250.854415 secs 2024-07-26T11:31:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:46 smithi116 ceph-mon[37097]: pgmap v5905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:31:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:48 smithi171 ceph-mon[37789]: pgmap v5906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:48 smithi044 ceph-mon[32404]: pgmap v5906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:48 smithi116 ceph-mon[37097]: pgmap v5906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:31:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:50 smithi171 ceph-mon[37789]: pgmap v5907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:50 smithi044 ceph-mon[32404]: pgmap v5907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:50 smithi116 ceph-mon[37097]: pgmap v5907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:31:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10255.854576 secs 2024-07-26T11:31:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10255.854576 secs 2024-07-26T11:31:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10255.854576 secs 2024-07-26T11:31:53.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:52 smithi044 ceph-mon[32404]: pgmap v5908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:52 smithi116 ceph-mon[37097]: pgmap v5908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:53.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:52 smithi171 ceph-mon[37789]: pgmap v5908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:54 smithi044 ceph-mon[32404]: pgmap v5909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:54 smithi116 ceph-mon[37097]: pgmap v5909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:54 smithi171 ceph-mon[37789]: pgmap v5909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10260.854717 secs 2024-07-26T11:31:57.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:56 smithi044 ceph-mon[32404]: pgmap v5910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10260.854717 secs 2024-07-26T11:31:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:56 smithi116 ceph-mon[37097]: pgmap v5910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10260.854717 secs 2024-07-26T11:31:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:56 smithi171 ceph-mon[37789]: pgmap v5910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:31:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:58.907+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:31:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:58.907+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:58.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:31:58.907+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:31:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:31:58 smithi044 ceph-mon[32404]: pgmap v5911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:31:58 smithi116 ceph-mon[37097]: pgmap v5911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:31:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:31:58 smithi171 ceph-mon[37789]: pgmap v5911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:00 smithi044 ceph-mon[32404]: pgmap v5912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:00 smithi116 ceph-mon[37097]: pgmap v5912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:00 smithi171 ceph-mon[37789]: pgmap v5912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10265.854859 secs 2024-07-26T11:32:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10265.854859 secs 2024-07-26T11:32:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10265.854859 secs 2024-07-26T11:32:03.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:02 smithi116 ceph-mon[37097]: pgmap v5913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:02 smithi044 ceph-mon[32404]: pgmap v5913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:02 smithi171 ceph-mon[37789]: pgmap v5913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:04.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:03 smithi044 ceph-mon[32404]: pgmap v5914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:03 smithi116 ceph-mon[37097]: pgmap v5914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:03 smithi171 ceph-mon[37789]: pgmap v5914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10270.855000 secs 2024-07-26T11:32:07.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:06 smithi171 ceph-mon[37789]: pgmap v5915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10270.855000 secs 2024-07-26T11:32:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:06 smithi044 ceph-mon[32404]: pgmap v5915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10270.855000 secs 2024-07-26T11:32:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:06 smithi116 ceph-mon[37097]: pgmap v5915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:08 smithi171 ceph-mon[37789]: pgmap v5916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:08 smithi044 ceph-mon[32404]: pgmap v5916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:08 smithi116 ceph-mon[37097]: pgmap v5916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:10 smithi171 ceph-mon[37789]: pgmap v5917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:10 smithi044 ceph-mon[32404]: pgmap v5917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:10 smithi116 ceph-mon[37097]: pgmap v5917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10275.855136 secs 2024-07-26T11:32:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10275.855136 secs 2024-07-26T11:32:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10275.855136 secs 2024-07-26T11:32:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:12 smithi044 ceph-mon[32404]: pgmap v5918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:12 smithi116 ceph-mon[37097]: pgmap v5918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:12 smithi171 ceph-mon[37789]: pgmap v5918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:13.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:13.912+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:32:13.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:13.912+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:13.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:13.912+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:32:15.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:14 smithi044 ceph-mon[32404]: pgmap v5919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:14 smithi116 ceph-mon[37097]: pgmap v5919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:14 smithi171 ceph-mon[37789]: pgmap v5919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:16.516+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:32:16.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:16.516+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:32:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:16.517+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:32:16.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:16.517+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:32:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10280.855290 secs 2024-07-26T11:32:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:16 smithi044 ceph-mon[32404]: pgmap v5920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10280.855290 secs 2024-07-26T11:32:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:16 smithi116 ceph-mon[37097]: pgmap v5920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10280.855290 secs 2024-07-26T11:32:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:16 smithi171 ceph-mon[37789]: pgmap v5920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:19.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:18 smithi171 ceph-mon[37789]: pgmap v5921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:18 smithi044 ceph-mon[32404]: pgmap v5921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:18 smithi116 ceph-mon[37097]: pgmap v5921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:19 smithi044 ceph-mon[32404]: pgmap v5922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:19 smithi116 ceph-mon[37097]: pgmap v5922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:19 smithi171 ceph-mon[37789]: pgmap v5922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10285.855396 secs 2024-07-26T11:32:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10285.855396 secs 2024-07-26T11:32:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10285.855396 secs 2024-07-26T11:32:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:22 smithi044 ceph-mon[32404]: pgmap v5923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:22 smithi116 ceph-mon[37097]: pgmap v5923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:22 smithi171 ceph-mon[37789]: pgmap v5923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:24 smithi171 ceph-mon[37789]: pgmap v5924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:24 smithi044 ceph-mon[32404]: pgmap v5924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:24 smithi116 ceph-mon[37097]: pgmap v5924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10290.855557 secs 2024-07-26T11:32:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:26 smithi171 ceph-mon[37789]: pgmap v5925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10290.855557 secs 2024-07-26T11:32:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:26 smithi044 ceph-mon[32404]: pgmap v5925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10290.855557 secs 2024-07-26T11:32:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:26 smithi116 ceph-mon[37097]: pgmap v5925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:28.922+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:32:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:28.922+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:28.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:28.922+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:32:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:28 smithi044 ceph-mon[32404]: pgmap v5926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:28 smithi116 ceph-mon[37097]: pgmap v5926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:28 smithi171 ceph-mon[37789]: pgmap v5926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:30 smithi044 ceph-mon[32404]: pgmap v5927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:30 smithi116 ceph-mon[37097]: pgmap v5927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:30 smithi171 ceph-mon[37789]: pgmap v5927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10295.855718 secs 2024-07-26T11:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10295.855718 secs 2024-07-26T11:32:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10295.855718 secs 2024-07-26T11:32:33.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:32 smithi116 ceph-mon[37097]: pgmap v5928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:32 smithi044 ceph-mon[32404]: pgmap v5928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:32 smithi171 ceph-mon[37789]: pgmap v5928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:33 smithi044 ceph-mon[32404]: pgmap v5929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:33 smithi116 ceph-mon[37097]: pgmap v5929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:33 smithi171 ceph-mon[37789]: pgmap v5929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10300.855875 secs 2024-07-26T11:32:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:36 smithi171 ceph-mon[37789]: pgmap v5930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10300.855875 secs 2024-07-26T11:32:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:36 smithi044 ceph-mon[32404]: pgmap v5930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10300.855875 secs 2024-07-26T11:32:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:36 smithi116 ceph-mon[37097]: pgmap v5930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:38 smithi044 ceph-mon[32404]: pgmap v5931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:38 smithi116 ceph-mon[37097]: pgmap v5931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:38 smithi171 ceph-mon[37789]: pgmap v5931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:40 smithi044 ceph-mon[32404]: pgmap v5932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:40 smithi116 ceph-mon[37097]: pgmap v5932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:40 smithi171 ceph-mon[37789]: pgmap v5932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:32:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10305.856000 secs 2024-07-26T11:32:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:32:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10305.856000 secs 2024-07-26T11:32:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:42.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:32:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10305.856000 secs 2024-07-26T11:32:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:42.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:32:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:42 smithi044 ceph-mon[32404]: pgmap v5933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:42 smithi116 ceph-mon[37097]: pgmap v5933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:42 smithi171 ceph-mon[37789]: pgmap v5933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:43.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:43.926+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:32:43.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:43.926+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:43.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:43.926+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:32:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:44 smithi044 ceph-mon[32404]: pgmap v5934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:44 smithi116 ceph-mon[37097]: pgmap v5934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:44 smithi171 ceph-mon[37789]: pgmap v5934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:46 smithi044 ceph-mon[32404]: pgmap v5935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10310.856162 secs 2024-07-26T11:32:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:46 smithi116 ceph-mon[37097]: pgmap v5935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10310.856162 secs 2024-07-26T11:32:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:46 smithi171 ceph-mon[37789]: pgmap v5935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10310.856162 secs 2024-07-26T11:32:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:47 smithi044 ceph-mon[32404]: pgmap v5936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:47 smithi116 ceph-mon[37097]: pgmap v5936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:47 smithi171 ceph-mon[37789]: pgmap v5936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:50 smithi171 ceph-mon[37789]: pgmap v5937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:50 smithi044 ceph-mon[32404]: pgmap v5937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:50 smithi116 ceph-mon[37097]: pgmap v5937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10315.856326 secs 2024-07-26T11:32:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:52 smithi171 ceph-mon[37789]: pgmap v5938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10315.856326 secs 2024-07-26T11:32:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:52 smithi044 ceph-mon[32404]: pgmap v5938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10315.856326 secs 2024-07-26T11:32:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:52 smithi116 ceph-mon[37097]: pgmap v5938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:32:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:54 smithi171 ceph-mon[37789]: pgmap v5939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:54 smithi044 ceph-mon[32404]: pgmap v5939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:54 smithi116 ceph-mon[37097]: pgmap v5939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:56 smithi171 ceph-mon[37789]: pgmap v5940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:56 smithi044 ceph-mon[32404]: pgmap v5940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:56 smithi116 ceph-mon[37097]: pgmap v5940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:32:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10320.856462 secs 2024-07-26T11:32:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10320.856462 secs 2024-07-26T11:32:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10320.856462 secs 2024-07-26T11:32:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:32:58 smithi044 ceph-mon[32404]: pgmap v5941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:32:58 smithi116 ceph-mon[37097]: pgmap v5941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:32:58 smithi171 ceph-mon[37789]: pgmap v5941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:32:58.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:58.937+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:32:58.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:58.937+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:58.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:32:58.937+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:33:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:00 smithi044 ceph-mon[32404]: pgmap v5942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:00 smithi116 ceph-mon[37097]: pgmap v5942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:00 smithi171 ceph-mon[37789]: pgmap v5942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10325.856667 secs 2024-07-26T11:33:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:02 smithi044 ceph-mon[32404]: pgmap v5943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10325.856667 secs 2024-07-26T11:33:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:02 smithi116 ceph-mon[37097]: pgmap v5943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10325.856667 secs 2024-07-26T11:33:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:02 smithi171 ceph-mon[37789]: pgmap v5943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:04 smithi044 ceph-mon[32404]: pgmap v5944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:04 smithi116 ceph-mon[37097]: pgmap v5944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:04 smithi171 ceph-mon[37789]: pgmap v5944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:06 smithi044 ceph-mon[32404]: pgmap v5945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:06 smithi116 ceph-mon[37097]: pgmap v5945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:06 smithi171 ceph-mon[37789]: pgmap v5945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10330.856825 secs 2024-07-26T11:33:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10330.856825 secs 2024-07-26T11:33:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10330.856825 secs 2024-07-26T11:33:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:08 smithi044 ceph-mon[32404]: pgmap v5946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:08 smithi116 ceph-mon[37097]: pgmap v5946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:08 smithi171 ceph-mon[37789]: pgmap v5946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:10 smithi044 ceph-mon[32404]: pgmap v5947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:10 smithi116 ceph-mon[37097]: pgmap v5947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:10 smithi171 ceph-mon[37789]: pgmap v5947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10335.856989 secs 2024-07-26T11:33:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:12 smithi044 ceph-mon[32404]: pgmap v5948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10335.856989 secs 2024-07-26T11:33:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:12 smithi116 ceph-mon[37097]: pgmap v5948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10335.856989 secs 2024-07-26T11:33:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:12 smithi171 ceph-mon[37789]: pgmap v5948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:13.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:13.942+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:33:13.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:13.942+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:13.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:13.942+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:33:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:14 smithi044 ceph-mon[32404]: pgmap v5949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:14 smithi116 ceph-mon[37097]: pgmap v5949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:14 smithi171 ceph-mon[37789]: pgmap v5949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:16 smithi044 ceph-mon[32404]: pgmap v5950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:16 smithi116 ceph-mon[37097]: pgmap v5950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:16 smithi171 ceph-mon[37789]: pgmap v5950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10340.857155 secs 2024-07-26T11:33:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10340.857155 secs 2024-07-26T11:33:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10340.857155 secs 2024-07-26T11:33:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:18 smithi044 ceph-mon[32404]: pgmap v5951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:18 smithi116 ceph-mon[37097]: pgmap v5951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:18 smithi171 ceph-mon[37789]: pgmap v5951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:20 smithi044 ceph-mon[32404]: pgmap v5952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:20 smithi116 ceph-mon[37097]: pgmap v5952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:20 smithi171 ceph-mon[37789]: pgmap v5952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10345.857321 secs 2024-07-26T11:33:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:22 smithi044 ceph-mon[32404]: pgmap v5953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10345.857321 secs 2024-07-26T11:33:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:22 smithi116 ceph-mon[37097]: pgmap v5953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10345.857321 secs 2024-07-26T11:33:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:22 smithi171 ceph-mon[37789]: pgmap v5953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:24 smithi044 ceph-mon[32404]: pgmap v5954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:24 smithi116 ceph-mon[37097]: pgmap v5954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:24 smithi171 ceph-mon[37789]: pgmap v5954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:26 smithi044 ceph-mon[32404]: pgmap v5955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:26 smithi116 ceph-mon[37097]: pgmap v5955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:26 smithi171 ceph-mon[37789]: pgmap v5955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10350.857442 secs 2024-07-26T11:33:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10350.857442 secs 2024-07-26T11:33:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10350.857442 secs 2024-07-26T11:33:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:28 smithi044 ceph-mon[32404]: pgmap v5956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:28 smithi116 ceph-mon[37097]: pgmap v5956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:28 smithi171 ceph-mon[37789]: pgmap v5956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:28.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:28.952+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:33:28.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:28.952+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:28.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:28.952+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:33:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:30 smithi044 ceph-mon[32404]: pgmap v5957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:30 smithi116 ceph-mon[37097]: pgmap v5957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:30 smithi171 ceph-mon[37789]: pgmap v5957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10355.857610 secs 2024-07-26T11:33:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:32 smithi044 ceph-mon[32404]: pgmap v5958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:33:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10355.857610 secs 2024-07-26T11:33:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:32 smithi116 ceph-mon[37097]: pgmap v5958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:33:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10355.857610 secs 2024-07-26T11:33:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:32 smithi171 ceph-mon[37789]: pgmap v5958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:33:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:34 smithi044 ceph-mon[32404]: pgmap v5959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:33:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:34 smithi116 ceph-mon[37097]: pgmap v5959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:33:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:34 smithi171 ceph-mon[37789]: pgmap v5959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:33:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:36 smithi044 ceph-mon[32404]: pgmap v5960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:33:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:36 smithi116 ceph-mon[37097]: pgmap v5960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:33:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:36 smithi171 ceph-mon[37789]: pgmap v5960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:33:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10360.857755 secs 2024-07-26T11:33:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10360.857755 secs 2024-07-26T11:33:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10360.857755 secs 2024-07-26T11:33:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:38 smithi044 ceph-mon[32404]: pgmap v5961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:38 smithi116 ceph-mon[37097]: pgmap v5961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:38 smithi171 ceph-mon[37789]: pgmap v5961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:40 smithi044 ceph-mon[32404]: pgmap v5962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:40 smithi116 ceph-mon[37097]: pgmap v5962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:40 smithi171 ceph-mon[37789]: pgmap v5962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:33:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10365.857916 secs 2024-07-26T11:33:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:42 smithi171 ceph-mon[37789]: pgmap v5963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:33:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10365.857916 secs 2024-07-26T11:33:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:42 smithi044 ceph-mon[32404]: pgmap v5963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:42.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:33:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10365.857916 secs 2024-07-26T11:33:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:42 smithi116 ceph-mon[37097]: pgmap v5963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:33:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:43 smithi171 ceph-mon[37789]: pgmap v5964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:33:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:43 smithi171 ceph-mon[37789]: pgmap v5965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:43 smithi044 ceph-mon[32404]: pgmap v5964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:33:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:43 smithi044 ceph-mon[32404]: pgmap v5965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:43 smithi116 ceph-mon[37097]: pgmap v5964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:33:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:43 smithi116 ceph-mon[37097]: pgmap v5965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:43.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:43.956+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:33:43.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:43.956+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:43.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:43.956+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:33:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:45 smithi171 ceph-mon[37789]: pgmap v5966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:45 smithi044 ceph-mon[32404]: pgmap v5966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:45 smithi116 ceph-mon[37097]: pgmap v5966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10370.858050 secs 2024-07-26T11:33:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:47 smithi171 ceph-mon[37789]: pgmap v5967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10370.858050 secs 2024-07-26T11:33:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:47 smithi044 ceph-mon[32404]: pgmap v5967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10370.858050 secs 2024-07-26T11:33:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:47 smithi116 ceph-mon[37097]: pgmap v5967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:49 smithi171 ceph-mon[37789]: pgmap v5968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:49 smithi044 ceph-mon[32404]: pgmap v5968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:49 smithi116 ceph-mon[37097]: pgmap v5968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:51 smithi171 ceph-mon[37789]: pgmap v5969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:51 smithi044 ceph-mon[32404]: pgmap v5969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:51 smithi116 ceph-mon[37097]: pgmap v5969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:33:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10375.858194 secs 2024-07-26T11:33:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10375.858194 secs 2024-07-26T11:33:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10375.858194 secs 2024-07-26T11:33:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:53 smithi171 ceph-mon[37789]: pgmap v5970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:33:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:53 smithi044 ceph-mon[32404]: pgmap v5970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:33:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:53 smithi116 ceph-mon[37097]: pgmap v5970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:33:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:55 smithi171 ceph-mon[37789]: pgmap v5971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:55 smithi044 ceph-mon[32404]: pgmap v5971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:55 smithi116 ceph-mon[37097]: pgmap v5971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10380.858357 secs 2024-07-26T11:33:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:57 smithi171 ceph-mon[37789]: pgmap v5972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10380.858357 secs 2024-07-26T11:33:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:57 smithi044 ceph-mon[32404]: pgmap v5972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10380.858357 secs 2024-07-26T11:33:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:57 smithi116 ceph-mon[37097]: pgmap v5972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:58.966+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:33:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:58.966+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:33:58.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:33:58.966+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:33:59 smithi171 ceph-mon[37789]: pgmap v5973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:33:59 smithi044 ceph-mon[32404]: pgmap v5973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:33:59 smithi116 ceph-mon[37097]: pgmap v5973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:01 smithi171 ceph-mon[37789]: pgmap v5974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:01 smithi044 ceph-mon[32404]: pgmap v5974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:01 smithi116 ceph-mon[37097]: pgmap v5974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10385.858530 secs 2024-07-26T11:34:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10385.858530 secs 2024-07-26T11:34:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10385.858530 secs 2024-07-26T11:34:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:03 smithi044 ceph-mon[32404]: pgmap v5975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:03 smithi116 ceph-mon[37097]: pgmap v5975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:03 smithi171 ceph-mon[37789]: pgmap v5975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:05 smithi044 ceph-mon[32404]: pgmap v5976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:05 smithi116 ceph-mon[37097]: pgmap v5976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:05 smithi171 ceph-mon[37789]: pgmap v5976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10390.858697 secs 2024-07-26T11:34:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10390.858697 secs 2024-07-26T11:34:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10390.858697 secs 2024-07-26T11:34:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:07 smithi044 ceph-mon[32404]: pgmap v5977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:07 smithi116 ceph-mon[37097]: pgmap v5977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:07 smithi171 ceph-mon[37789]: pgmap v5977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:09 smithi044 ceph-mon[32404]: pgmap v5978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:09 smithi116 ceph-mon[37097]: pgmap v5978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:09 smithi171 ceph-mon[37789]: pgmap v5978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:11 smithi044 ceph-mon[32404]: pgmap v5979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10395.858846 secs 2024-07-26T11:34:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:11 smithi116 ceph-mon[37097]: pgmap v5979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10395.858846 secs 2024-07-26T11:34:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:11 smithi171 ceph-mon[37789]: pgmap v5979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10395.858846 secs 2024-07-26T11:34:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:13 smithi044 ceph-mon[32404]: pgmap v5980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:13 smithi116 ceph-mon[37097]: pgmap v5980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:13 smithi171 ceph-mon[37789]: pgmap v5980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:13.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:13.971+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:34:13.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:13.971+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:34:13.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:13.972+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:15 smithi044 ceph-mon[32404]: pgmap v5981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:15 smithi116 ceph-mon[37097]: pgmap v5981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:15 smithi171 ceph-mon[37789]: pgmap v5981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10400.859015 secs 2024-07-26T11:34:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10400.859015 secs 2024-07-26T11:34:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10400.859015 secs 2024-07-26T11:34:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:17 smithi044 ceph-mon[32404]: pgmap v5982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:17 smithi116 ceph-mon[37097]: pgmap v5982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:17 smithi171 ceph-mon[37789]: pgmap v5982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:19 smithi044 ceph-mon[32404]: pgmap v5983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:19 smithi116 ceph-mon[37097]: pgmap v5983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:19 smithi171 ceph-mon[37789]: pgmap v5983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:21 smithi044 ceph-mon[32404]: pgmap v5984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10405.859206 secs 2024-07-26T11:34:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:21 smithi116 ceph-mon[37097]: pgmap v5984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10405.859206 secs 2024-07-26T11:34:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:21 smithi171 ceph-mon[37789]: pgmap v5984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10405.859206 secs 2024-07-26T11:34:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:23 smithi044 ceph-mon[32404]: pgmap v5985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:23 smithi116 ceph-mon[37097]: pgmap v5985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:23 smithi171 ceph-mon[37789]: pgmap v5985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:25 smithi044 ceph-mon[32404]: pgmap v5986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:25 smithi116 ceph-mon[37097]: pgmap v5986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:25 smithi171 ceph-mon[37789]: pgmap v5986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10410.859388 secs 2024-07-26T11:34:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10410.859388 secs 2024-07-26T11:34:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10410.859388 secs 2024-07-26T11:34:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:27 smithi044 ceph-mon[32404]: pgmap v5987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:27 smithi116 ceph-mon[37097]: pgmap v5987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:27 smithi171 ceph-mon[37789]: pgmap v5987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:28.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:28.982+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:34:28.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:28.982+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:34:28.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:28.982+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:29 smithi044 ceph-mon[32404]: pgmap v5988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:29 smithi116 ceph-mon[37097]: pgmap v5988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:29 smithi171 ceph-mon[37789]: pgmap v5988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:31.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:31.540+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:34:31.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:31.540+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:34:31.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:31.541+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:34:31.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:31.541+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:34:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:31 smithi044 ceph-mon[32404]: pgmap v5989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10415.859551 secs 2024-07-26T11:34:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:31 smithi116 ceph-mon[37097]: pgmap v5989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10415.859551 secs 2024-07-26T11:34:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:31 smithi171 ceph-mon[37789]: pgmap v5989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10415.859551 secs 2024-07-26T11:34:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:33 smithi044 ceph-mon[32404]: pgmap v5990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:33 smithi116 ceph-mon[37097]: pgmap v5990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:33 smithi171 ceph-mon[37789]: pgmap v5990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:35 smithi044 ceph-mon[32404]: pgmap v5991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:35 smithi116 ceph-mon[37097]: pgmap v5991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:35 smithi171 ceph-mon[37789]: pgmap v5991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:34:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10420.859716 secs 2024-07-26T11:34:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10420.859716 secs 2024-07-26T11:34:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10420.859716 secs 2024-07-26T11:34:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:37 smithi044 ceph-mon[32404]: pgmap v5992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:37 smithi116 ceph-mon[37097]: pgmap v5992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:37 smithi171 ceph-mon[37789]: pgmap v5992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:39 smithi044 ceph-mon[32404]: pgmap v5993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:39 smithi116 ceph-mon[37097]: pgmap v5993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:39 smithi171 ceph-mon[37789]: pgmap v5993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:41 smithi044 ceph-mon[32404]: pgmap v5994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10425.859882 secs 2024-07-26T11:34:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:41 smithi116 ceph-mon[37097]: pgmap v5994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10425.859882 secs 2024-07-26T11:34:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:41 smithi171 ceph-mon[37789]: pgmap v5994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10425.859882 secs 2024-07-26T11:34:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:43 smithi116 ceph-mon[37097]: pgmap v5995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:43.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:34:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:43 smithi171 ceph-mon[37789]: pgmap v5995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:34:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:43.987+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:34:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:43.987+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:34:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:43.987+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:43.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:43 smithi044 ceph-mon[32404]: pgmap v5995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:43.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:43.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:43.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:34:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:44 smithi116 ceph-mon[37097]: pgmap v5996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:34:44.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:44 smithi116 ceph-mon[37097]: pgmap v5997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:44 smithi171 ceph-mon[37789]: pgmap v5996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:34:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:44 smithi171 ceph-mon[37789]: pgmap v5997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:44 smithi044 ceph-mon[32404]: pgmap v5996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:34:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:44 smithi044 ceph-mon[32404]: pgmap v5997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:46 smithi171 ceph-mon[37789]: pgmap v5998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10430.860045 secs 2024-07-26T11:34:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:46 smithi044 ceph-mon[32404]: pgmap v5998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10430.860045 secs 2024-07-26T11:34:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:46 smithi116 ceph-mon[37097]: pgmap v5998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10430.860045 secs 2024-07-26T11:34:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:48 smithi171 ceph-mon[37789]: pgmap v5999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:48 smithi044 ceph-mon[32404]: pgmap v5999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:48 smithi116 ceph-mon[37097]: pgmap v5999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:50 smithi171 ceph-mon[37789]: pgmap v6000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:50 smithi044 ceph-mon[32404]: pgmap v6000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:50 smithi116 ceph-mon[37097]: pgmap v6000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10435.860206 secs 2024-07-26T11:34:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10435.860206 secs 2024-07-26T11:34:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10435.860206 secs 2024-07-26T11:34:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:52 smithi171 ceph-mon[37789]: pgmap v6001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:52 smithi044 ceph-mon[32404]: pgmap v6001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:52 smithi116 ceph-mon[37097]: pgmap v6001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:54 smithi171 ceph-mon[37789]: pgmap v6002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:54 smithi044 ceph-mon[32404]: pgmap v6002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:54 smithi116 ceph-mon[37097]: pgmap v6002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:56 smithi171 ceph-mon[37789]: pgmap v6003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:56 smithi044 ceph-mon[32404]: pgmap v6003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:56 smithi116 ceph-mon[37097]: pgmap v6003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10440.860347 secs 2024-07-26T11:34:57.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10440.860347 secs 2024-07-26T11:34:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10440.860347 secs 2024-07-26T11:34:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:34:58 smithi171 ceph-mon[37789]: pgmap v6004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:34:58 smithi044 ceph-mon[32404]: pgmap v6004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:34:58 smithi116 ceph-mon[37097]: pgmap v6004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:34:58.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:58.990+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:34:58.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:58.990+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:34:58.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:34:58.990+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:00 smithi171 ceph-mon[37789]: pgmap v6005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:01.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:00 smithi044 ceph-mon[32404]: pgmap v6005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:00 smithi116 ceph-mon[37097]: pgmap v6005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:01.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10445.860485 secs 2024-07-26T11:35:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10445.860485 secs 2024-07-26T11:35:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10445.860485 secs 2024-07-26T11:35:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:02 smithi171 ceph-mon[37789]: pgmap v6006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:02 smithi044 ceph-mon[32404]: pgmap v6006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:02 smithi116 ceph-mon[37097]: pgmap v6006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:04 smithi171 ceph-mon[37789]: pgmap v6007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:04 smithi044 ceph-mon[32404]: pgmap v6007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:04 smithi116 ceph-mon[37097]: pgmap v6007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:06.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:06 smithi116 ceph-mon[37097]: pgmap v6008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:06.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10450.860679 secs 2024-07-26T11:35:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:06 smithi171 ceph-mon[37789]: pgmap v6008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10450.860679 secs 2024-07-26T11:35:06.914 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:06 smithi044 ceph-mon[32404]: pgmap v6008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:06.914 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10450.860679 secs 2024-07-26T11:35:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:08 smithi171 ceph-mon[37789]: pgmap v6009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:08 smithi044 ceph-mon[32404]: pgmap v6009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:08 smithi116 ceph-mon[37097]: pgmap v6009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:10 smithi171 ceph-mon[37789]: pgmap v6010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:10 smithi044 ceph-mon[32404]: pgmap v6010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:10 smithi116 ceph-mon[37097]: pgmap v6010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10455.860780 secs 2024-07-26T11:35:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10455.860780 secs 2024-07-26T11:35:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10455.860780 secs 2024-07-26T11:35:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:12 smithi171 ceph-mon[37789]: pgmap v6011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:12 smithi044 ceph-mon[32404]: pgmap v6011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:12 smithi116 ceph-mon[37097]: pgmap v6011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:13.995+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:35:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:13.996+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:13.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:13.996+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:35:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:14 smithi044 ceph-mon[32404]: pgmap v6012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:14 smithi116 ceph-mon[37097]: pgmap v6012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:14 smithi171 ceph-mon[37789]: pgmap v6012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:16.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:16.548+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:35:16.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:16.548+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:35:16.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:16.549+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:35:16.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:16.549+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:35:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:16 smithi044 ceph-mon[32404]: pgmap v6013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10460.860937 secs 2024-07-26T11:35:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:16 smithi116 ceph-mon[37097]: pgmap v6013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10460.860937 secs 2024-07-26T11:35:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:16 smithi171 ceph-mon[37789]: pgmap v6013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10460.860937 secs 2024-07-26T11:35:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:18 smithi044 ceph-mon[32404]: pgmap v6014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:18 smithi116 ceph-mon[37097]: pgmap v6014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:18 smithi171 ceph-mon[37789]: pgmap v6014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:20 smithi044 ceph-mon[32404]: pgmap v6015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:20 smithi116 ceph-mon[37097]: pgmap v6015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:20 smithi171 ceph-mon[37789]: pgmap v6015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:21.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10465.861106 secs 2024-07-26T11:35:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10465.861106 secs 2024-07-26T11:35:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10465.861106 secs 2024-07-26T11:35:22.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:22 smithi044 ceph-mon[32404]: pgmap v6016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:22 smithi116 ceph-mon[37097]: pgmap v6016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:22 smithi171 ceph-mon[37789]: pgmap v6016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:24 smithi044 ceph-mon[32404]: pgmap v6017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:24 smithi116 ceph-mon[37097]: pgmap v6017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:24 smithi171 ceph-mon[37789]: pgmap v6017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:26 smithi044 ceph-mon[32404]: pgmap v6018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10470.861248 secs 2024-07-26T11:35:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:26 smithi116 ceph-mon[37097]: pgmap v6018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10470.861248 secs 2024-07-26T11:35:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:26 smithi171 ceph-mon[37789]: pgmap v6018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10470.861248 secs 2024-07-26T11:35:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:29.006+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:35:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:29.006+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:29.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:29.006+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:35:29.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:28 smithi044 ceph-mon[32404]: pgmap v6019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:28 smithi116 ceph-mon[37097]: pgmap v6019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:28 smithi171 ceph-mon[37789]: pgmap v6019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:30 smithi044 ceph-mon[32404]: pgmap v6020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:30 smithi116 ceph-mon[37097]: pgmap v6020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:30 smithi171 ceph-mon[37789]: pgmap v6020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10475.861391 secs 2024-07-26T11:35:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10475.861391 secs 2024-07-26T11:35:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10475.861391 secs 2024-07-26T11:35:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:32 smithi044 ceph-mon[32404]: pgmap v6021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:32 smithi116 ceph-mon[37097]: pgmap v6021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:32 smithi171 ceph-mon[37789]: pgmap v6021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:34 smithi044 ceph-mon[32404]: pgmap v6022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:34 smithi116 ceph-mon[37097]: pgmap v6022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:34 smithi171 ceph-mon[37789]: pgmap v6022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:36 smithi044 ceph-mon[32404]: pgmap v6023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10480.861500 secs 2024-07-26T11:35:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:36 smithi116 ceph-mon[37097]: pgmap v6023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10480.861500 secs 2024-07-26T11:35:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:36 smithi171 ceph-mon[37789]: pgmap v6023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10480.861500 secs 2024-07-26T11:35:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:38 smithi044 ceph-mon[32404]: pgmap v6024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:38 smithi116 ceph-mon[37097]: pgmap v6024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:38 smithi171 ceph-mon[37789]: pgmap v6024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:35:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:40 smithi044 ceph-mon[32404]: pgmap v6025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:40 smithi116 ceph-mon[37097]: pgmap v6025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:40 smithi171 ceph-mon[37789]: pgmap v6025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10485.861667 secs 2024-07-26T11:35:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10485.861667 secs 2024-07-26T11:35:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10485.861667 secs 2024-07-26T11:35:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:42 smithi044 ceph-mon[32404]: pgmap v6026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:42 smithi116 ceph-mon[37097]: pgmap v6026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:42 smithi171 ceph-mon[37789]: pgmap v6026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:44.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:44.011+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:35:44.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:44.011+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:44.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:44.011+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:35:44.014 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:44 smithi044 ceph-mon[32404]: pgmap v6027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:35:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:44 smithi116 ceph-mon[37097]: pgmap v6027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:35:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:44 smithi171 ceph-mon[37789]: pgmap v6027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:35:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:45 smithi044 ceph-mon[32404]: pgmap v6028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:46.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:35:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:45 smithi116 ceph-mon[37097]: pgmap v6028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:35:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:45 smithi171 ceph-mon[37789]: pgmap v6028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:35:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10490.861776 secs 2024-07-26T11:35:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10490.861776 secs 2024-07-26T11:35:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10490.861776 secs 2024-07-26T11:35:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:47 smithi044 ceph-mon[32404]: pgmap v6029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:47 smithi116 ceph-mon[37097]: pgmap v6029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:47 smithi171 ceph-mon[37789]: pgmap v6029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:49 smithi044 ceph-mon[32404]: pgmap v6030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:49 smithi116 ceph-mon[37097]: pgmap v6030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:49 smithi171 ceph-mon[37789]: pgmap v6030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10495.861916 secs 2024-07-26T11:35:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:51 smithi044 ceph-mon[32404]: pgmap v6031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10495.861916 secs 2024-07-26T11:35:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:51 smithi116 ceph-mon[37097]: pgmap v6031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10495.861916 secs 2024-07-26T11:35:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:51 smithi171 ceph-mon[37789]: pgmap v6031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:35:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:53 smithi044 ceph-mon[32404]: pgmap v6032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:53 smithi116 ceph-mon[37097]: pgmap v6032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:53 smithi171 ceph-mon[37789]: pgmap v6032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:55 smithi044 ceph-mon[32404]: pgmap v6033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:35:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:55 smithi116 ceph-mon[37097]: pgmap v6033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:35:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:55 smithi171 ceph-mon[37789]: pgmap v6033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:35:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10500.862049 secs 2024-07-26T11:35:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10500.862049 secs 2024-07-26T11:35:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10500.862049 secs 2024-07-26T11:35:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:57 smithi171 ceph-mon[37789]: pgmap v6034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:58.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:57 smithi044 ceph-mon[32404]: pgmap v6034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:57 smithi116 ceph-mon[37097]: pgmap v6034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:35:59.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:59.020+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:35:59.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:59.020+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:59.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:35:59.020+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:36:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:35:59 smithi171 ceph-mon[37789]: pgmap v6035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:35:59 smithi044 ceph-mon[32404]: pgmap v6035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:35:59 smithi116 ceph-mon[37097]: pgmap v6035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10505.862213 secs 2024-07-26T11:36:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:01 smithi171 ceph-mon[37789]: pgmap v6036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10505.862213 secs 2024-07-26T11:36:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:01 smithi044 ceph-mon[32404]: pgmap v6036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10505.862213 secs 2024-07-26T11:36:02.309 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:01 smithi116 ceph-mon[37097]: pgmap v6036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:03 smithi171 ceph-mon[37789]: pgmap v6037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:03 smithi044 ceph-mon[32404]: pgmap v6037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:03 smithi116 ceph-mon[37097]: pgmap v6037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:06.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:05 smithi116 ceph-mon[37097]: pgmap v6038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:05 smithi171 ceph-mon[37789]: pgmap v6038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:05 smithi044 ceph-mon[32404]: pgmap v6038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10510.862356 secs 2024-07-26T11:36:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10510.862356 secs 2024-07-26T11:36:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10510.862356 secs 2024-07-26T11:36:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:07 smithi171 ceph-mon[37789]: pgmap v6039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:07 smithi044 ceph-mon[32404]: pgmap v6039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:07 smithi116 ceph-mon[37097]: pgmap v6039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:09 smithi171 ceph-mon[37789]: pgmap v6040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:09 smithi044 ceph-mon[32404]: pgmap v6040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:09 smithi116 ceph-mon[37097]: pgmap v6040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10515.862521 secs 2024-07-26T11:36:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:11 smithi171 ceph-mon[37789]: pgmap v6041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10515.862521 secs 2024-07-26T11:36:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:11 smithi044 ceph-mon[32404]: pgmap v6041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10515.862521 secs 2024-07-26T11:36:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:11 smithi116 ceph-mon[37097]: pgmap v6041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:14.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:14.025+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:36:14.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:14.025+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:14.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:14.025+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:36:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:13 smithi171 ceph-mon[37789]: pgmap v6042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:13 smithi044 ceph-mon[32404]: pgmap v6042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:13 smithi116 ceph-mon[37097]: pgmap v6042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:15 smithi171 ceph-mon[37789]: pgmap v6043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:15 smithi044 ceph-mon[32404]: pgmap v6043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:15 smithi116 ceph-mon[37097]: pgmap v6043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10520.862691 secs 2024-07-26T11:36:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10520.862691 secs 2024-07-26T11:36:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10520.862691 secs 2024-07-26T11:36:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:17 smithi044 ceph-mon[32404]: pgmap v6044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:17 smithi116 ceph-mon[37097]: pgmap v6044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:17 smithi171 ceph-mon[37789]: pgmap v6044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:19 smithi044 ceph-mon[32404]: pgmap v6045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:19 smithi116 ceph-mon[37097]: pgmap v6045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:19 smithi171 ceph-mon[37789]: pgmap v6045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:21 smithi044 ceph-mon[32404]: pgmap v6046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:21 smithi116 ceph-mon[37097]: pgmap v6046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:21 smithi171 ceph-mon[37789]: pgmap v6046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10525.862856 secs 2024-07-26T11:36:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10525.862856 secs 2024-07-26T11:36:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10525.862856 secs 2024-07-26T11:36:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:23 smithi044 ceph-mon[32404]: pgmap v6047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:23 smithi116 ceph-mon[37097]: pgmap v6047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:23 smithi171 ceph-mon[37789]: pgmap v6047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:25 smithi171 ceph-mon[37789]: pgmap v6048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:25 smithi044 ceph-mon[32404]: pgmap v6048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:25 smithi116 ceph-mon[37097]: pgmap v6048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10530.862974 secs 2024-07-26T11:36:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10530.862974 secs 2024-07-26T11:36:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10530.862974 secs 2024-07-26T11:36:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:27 smithi171 ceph-mon[37789]: pgmap v6049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:28.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:27 smithi044 ceph-mon[32404]: pgmap v6049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:27 smithi116 ceph-mon[37097]: pgmap v6049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:29.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:29.033+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:36:29.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:29.033+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:29.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:29.033+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:36:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:29 smithi171 ceph-mon[37789]: pgmap v6050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:29 smithi044 ceph-mon[32404]: pgmap v6050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:29 smithi116 ceph-mon[37097]: pgmap v6050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10535.863135 secs 2024-07-26T11:36:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:31 smithi044 ceph-mon[32404]: pgmap v6051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10535.863135 secs 2024-07-26T11:36:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:31 smithi116 ceph-mon[37097]: pgmap v6051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10535.863135 secs 2024-07-26T11:36:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:31 smithi171 ceph-mon[37789]: pgmap v6051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:33 smithi044 ceph-mon[32404]: pgmap v6052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:33 smithi116 ceph-mon[37097]: pgmap v6052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:33 smithi171 ceph-mon[37789]: pgmap v6052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:35 smithi044 ceph-mon[32404]: pgmap v6053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:35 smithi116 ceph-mon[37097]: pgmap v6053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:35 smithi171 ceph-mon[37789]: pgmap v6053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:37.264 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10540.863302 secs 2024-07-26T11:36:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10540.863302 secs 2024-07-26T11:36:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10540.863302 secs 2024-07-26T11:36:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:37 smithi044 ceph-mon[32404]: pgmap v6054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:37 smithi116 ceph-mon[37097]: pgmap v6054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:37 smithi171 ceph-mon[37789]: pgmap v6054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:39 smithi044 ceph-mon[32404]: pgmap v6055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:39 smithi116 ceph-mon[37097]: pgmap v6055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:39 smithi171 ceph-mon[37789]: pgmap v6055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10545.863453 secs 2024-07-26T11:36:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:41 smithi044 ceph-mon[32404]: pgmap v6056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:36:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10545.863453 secs 2024-07-26T11:36:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:41 smithi116 ceph-mon[37097]: pgmap v6056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:36:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10545.863453 secs 2024-07-26T11:36:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:41 smithi171 ceph-mon[37789]: pgmap v6056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:36:44.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:44.038+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:36:44.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:44.038+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:44.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:44.038+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:36:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:43 smithi044 ceph-mon[32404]: pgmap v6057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:43 smithi116 ceph-mon[37097]: pgmap v6057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:43 smithi171 ceph-mon[37789]: pgmap v6057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:46.130 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:45 smithi116 ceph-mon[37097]: pgmap v6058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:46.130 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:45 smithi044 ceph-mon[32404]: pgmap v6058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:45 smithi171 ceph-mon[37789]: pgmap v6058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:36:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10550.863566 secs 2024-07-26T11:36:47.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10550.863566 secs 2024-07-26T11:36:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10550.863566 secs 2024-07-26T11:36:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:47 smithi044 ceph-mon[32404]: pgmap v6059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:47 smithi116 ceph-mon[37097]: pgmap v6059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:47 smithi171 ceph-mon[37789]: pgmap v6059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:49 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:50.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:49 smithi044 ceph-mon[32404]: pgmap v6060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:49 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:50.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:49 smithi116 ceph-mon[37097]: pgmap v6060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:49 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:49 smithi171 ceph-mon[37789]: pgmap v6060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:36:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10555.863673 secs 2024-07-26T11:36:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:51 smithi044 ceph-mon[32404]: pgmap v6061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:36:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10555.863673 secs 2024-07-26T11:36:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:51 smithi171 ceph-mon[37789]: pgmap v6061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:36:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10555.863673 secs 2024-07-26T11:36:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:51 smithi116 ceph-mon[37097]: pgmap v6061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:36:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: pgmap v6062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:36:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: pgmap v6063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:53.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:36:53.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:36:53.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:53.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:53.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: pgmap v6062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:36:53.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: pgmap v6063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:53.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: pgmap v6062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: pgmap v6063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:53.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:36:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:54 smithi044 ceph-mon[32404]: pgmap v6064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:54 smithi116 ceph-mon[37097]: pgmap v6064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:54 smithi171 ceph-mon[37789]: pgmap v6064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:55 smithi044 ceph-mon[32404]: pgmap v6065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:55 smithi116 ceph-mon[37097]: pgmap v6065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:55 smithi171 ceph-mon[37789]: pgmap v6065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10560.863843 secs 2024-07-26T11:36:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10560.863843 secs 2024-07-26T11:36:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10560.863843 secs 2024-07-26T11:36:58.169 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:57 smithi044 ceph-mon[32404]: pgmap v6066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:57 smithi116 ceph-mon[37097]: pgmap v6066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:57 smithi171 ceph-mon[37789]: pgmap v6066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:59.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:59.048+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:36:59.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:59.048+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:59.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:36:59.048+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:37:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:36:59 smithi044 ceph-mon[32404]: pgmap v6067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:36:59 smithi116 ceph-mon[37097]: pgmap v6067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:36:59 smithi171 ceph-mon[37789]: pgmap v6067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10565.863948 secs 2024-07-26T11:37:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10565.863948 secs 2024-07-26T11:37:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10565.863948 secs 2024-07-26T11:37:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:02 smithi044 ceph-mon[32404]: pgmap v6068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:02 smithi116 ceph-mon[37097]: pgmap v6068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:02 smithi171 ceph-mon[37789]: pgmap v6068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:03 smithi044 ceph-mon[32404]: pgmap v6069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:03 smithi116 ceph-mon[37097]: pgmap v6069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:03 smithi171 ceph-mon[37789]: pgmap v6069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:05 smithi044 ceph-mon[32404]: pgmap v6070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:05 smithi116 ceph-mon[37097]: pgmap v6070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:05 smithi171 ceph-mon[37789]: pgmap v6070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10570.864120 secs 2024-07-26T11:37:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10570.864120 secs 2024-07-26T11:37:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10570.864120 secs 2024-07-26T11:37:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:07 smithi044 ceph-mon[32404]: pgmap v6071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:07 smithi116 ceph-mon[37097]: pgmap v6071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:07 smithi171 ceph-mon[37789]: pgmap v6071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:09 smithi044 ceph-mon[32404]: pgmap v6072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:09 smithi116 ceph-mon[37097]: pgmap v6072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:09 smithi171 ceph-mon[37789]: pgmap v6072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10575.864255 secs 2024-07-26T11:37:11.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10575.864255 secs 2024-07-26T11:37:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10575.864255 secs 2024-07-26T11:37:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:12 smithi044 ceph-mon[32404]: pgmap v6073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:12 smithi116 ceph-mon[37097]: pgmap v6073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:12 smithi171 ceph-mon[37789]: pgmap v6073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:14.053+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:37:14.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:14.053+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:14.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:14.053+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:37:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:13 smithi044 ceph-mon[32404]: pgmap v6074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:13 smithi116 ceph-mon[37097]: pgmap v6074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:13 smithi171 ceph-mon[37789]: pgmap v6074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:15 smithi044 ceph-mon[32404]: pgmap v6075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:15 smithi116 ceph-mon[37097]: pgmap v6075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:15 smithi171 ceph-mon[37789]: pgmap v6075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10580.864432 secs 2024-07-26T11:37:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10580.864432 secs 2024-07-26T11:37:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10580.864432 secs 2024-07-26T11:37:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:17 smithi044 ceph-mon[32404]: pgmap v6076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:17 smithi116 ceph-mon[37097]: pgmap v6076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:17 smithi171 ceph-mon[37789]: pgmap v6076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:19 smithi044 ceph-mon[32404]: pgmap v6077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:19 smithi116 ceph-mon[37097]: pgmap v6077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:19 smithi171 ceph-mon[37789]: pgmap v6077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10585.864584 secs 2024-07-26T11:37:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10585.864584 secs 2024-07-26T11:37:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10585.864584 secs 2024-07-26T11:37:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:22 smithi044 ceph-mon[32404]: pgmap v6078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:22 smithi116 ceph-mon[37097]: pgmap v6078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:22 smithi171 ceph-mon[37789]: pgmap v6078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:23 smithi044 ceph-mon[32404]: pgmap v6079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:23 smithi116 ceph-mon[37097]: pgmap v6079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:23 smithi171 ceph-mon[37789]: pgmap v6079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:25 smithi044 ceph-mon[32404]: pgmap v6080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:25 smithi116 ceph-mon[37097]: pgmap v6080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:25 smithi171 ceph-mon[37789]: pgmap v6080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10590.864764 secs 2024-07-26T11:37:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10590.864764 secs 2024-07-26T11:37:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10590.864764 secs 2024-07-26T11:37:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:27 smithi044 ceph-mon[32404]: pgmap v6081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:27 smithi116 ceph-mon[37097]: pgmap v6081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:27 smithi171 ceph-mon[37789]: pgmap v6081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:29.063+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:37:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:29.063+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:29.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:29.063+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:37:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:29 smithi044 ceph-mon[32404]: pgmap v6082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:29 smithi116 ceph-mon[37097]: pgmap v6082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:29 smithi171 ceph-mon[37789]: pgmap v6082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:37:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10595.864857 secs 2024-07-26T11:37:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10595.864857 secs 2024-07-26T11:37:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10595.864857 secs 2024-07-26T11:37:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:32 smithi044 ceph-mon[32404]: pgmap v6083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:32 smithi116 ceph-mon[37097]: pgmap v6083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:32 smithi171 ceph-mon[37789]: pgmap v6083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:33 smithi044 ceph-mon[32404]: pgmap v6084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:34.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:33 smithi116 ceph-mon[37097]: pgmap v6084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:33 smithi171 ceph-mon[37789]: pgmap v6084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:35 smithi044 ceph-mon[32404]: pgmap v6085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:35 smithi116 ceph-mon[37097]: pgmap v6085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:35 smithi171 ceph-mon[37789]: pgmap v6085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10600.864965 secs 2024-07-26T11:37:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10600.864965 secs 2024-07-26T11:37:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10600.864965 secs 2024-07-26T11:37:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:38 smithi044 ceph-mon[32404]: pgmap v6086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:38 smithi116 ceph-mon[37097]: pgmap v6086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:38 smithi171 ceph-mon[37789]: pgmap v6086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:39 smithi044 ceph-mon[32404]: pgmap v6087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:39 smithi116 ceph-mon[37097]: pgmap v6087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:39 smithi171 ceph-mon[37789]: pgmap v6087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10605.865124 secs 2024-07-26T11:37:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10605.865124 secs 2024-07-26T11:37:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10605.865124 secs 2024-07-26T11:37:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:42 smithi044 ceph-mon[32404]: pgmap v6088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:42 smithi116 ceph-mon[37097]: pgmap v6088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:42 smithi171 ceph-mon[37789]: pgmap v6088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:44.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:44.068+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:37:44.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:44.068+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:37:44.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:44.068+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:43 smithi044 ceph-mon[32404]: pgmap v6089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:43 smithi116 ceph-mon[37097]: pgmap v6089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:43 smithi171 ceph-mon[37789]: pgmap v6089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:37:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:45 smithi044 ceph-mon[32404]: pgmap v6090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:45 smithi116 ceph-mon[37097]: pgmap v6090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:45 smithi171 ceph-mon[37789]: pgmap v6090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10610.865276 secs 2024-07-26T11:37:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10610.865276 secs 2024-07-26T11:37:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10610.865276 secs 2024-07-26T11:37:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:48 smithi044 ceph-mon[32404]: pgmap v6091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:48 smithi116 ceph-mon[37097]: pgmap v6091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:48 smithi171 ceph-mon[37789]: pgmap v6091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:49 smithi044 ceph-mon[32404]: pgmap v6092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:49 smithi116 ceph-mon[37097]: pgmap v6092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:49 smithi171 ceph-mon[37789]: pgmap v6092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10615.865436 secs 2024-07-26T11:37:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10615.865436 secs 2024-07-26T11:37:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10615.865436 secs 2024-07-26T11:37:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:52 smithi044 ceph-mon[32404]: pgmap v6093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:52 smithi116 ceph-mon[37097]: pgmap v6093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:52 smithi171 ceph-mon[37789]: pgmap v6093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:37:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:53 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:37:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:37:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:53 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:37:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:37:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:53 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:37:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:54 smithi044 ceph-mon[32404]: pgmap v6094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:54 smithi116 ceph-mon[37097]: pgmap v6094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:54 smithi171 ceph-mon[37789]: pgmap v6094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10620.865613 secs 2024-07-26T11:37:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: pgmap v6095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10620.865613 secs 2024-07-26T11:37:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:37:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:57.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:57.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: pgmap v6095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:57.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10620.865613 secs 2024-07-26T11:37:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: pgmap v6095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:37:59.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:37:58 smithi044 ceph-mon[32404]: pgmap v6096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:59.078+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:37:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:59.078+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:37:59.078+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:37:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:37:58 smithi116 ceph-mon[37097]: pgmap v6096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:37:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:37:58 smithi171 ceph-mon[37789]: pgmap v6096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:38:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:00 smithi044 ceph-mon[32404]: pgmap v6097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:00 smithi116 ceph-mon[37097]: pgmap v6097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:00 smithi171 ceph-mon[37789]: pgmap v6097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10625.865767 secs 2024-07-26T11:38:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10625.865767 secs 2024-07-26T11:38:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10625.865767 secs 2024-07-26T11:38:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:02 smithi044 ceph-mon[32404]: pgmap v6098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:38:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:02 smithi116 ceph-mon[37097]: pgmap v6098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:38:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:02 smithi171 ceph-mon[37789]: pgmap v6098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:38:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:04 smithi044 ceph-mon[32404]: pgmap v6099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:04 smithi116 ceph-mon[37097]: pgmap v6099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:04 smithi171 ceph-mon[37789]: pgmap v6099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10630.865926 secs 2024-07-26T11:38:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:06 smithi044 ceph-mon[32404]: pgmap v6100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:38:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10630.865926 secs 2024-07-26T11:38:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:06 smithi116 ceph-mon[37097]: pgmap v6100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:38:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10630.865926 secs 2024-07-26T11:38:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:06 smithi171 ceph-mon[37789]: pgmap v6100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:38:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:08 smithi044 ceph-mon[32404]: pgmap v6101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:08 smithi116 ceph-mon[37097]: pgmap v6101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:08 smithi171 ceph-mon[37789]: pgmap v6101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:10 smithi044 ceph-mon[32404]: pgmap v6102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:10 smithi116 ceph-mon[37097]: pgmap v6102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:10 smithi171 ceph-mon[37789]: pgmap v6102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10635.866110 secs 2024-07-26T11:38:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10635.866110 secs 2024-07-26T11:38:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10635.866110 secs 2024-07-26T11:38:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:12 smithi044 ceph-mon[32404]: pgmap v6103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:12 smithi116 ceph-mon[37097]: pgmap v6103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:12 smithi171 ceph-mon[37789]: pgmap v6103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:14.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:14.082+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:38:14.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:14.082+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:14.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:14.082+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:38:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:14 smithi044 ceph-mon[32404]: pgmap v6104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:14 smithi116 ceph-mon[37097]: pgmap v6104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:14 smithi171 ceph-mon[37789]: pgmap v6104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:15 smithi044 ceph-mon[32404]: pgmap v6105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:15 smithi116 ceph-mon[37097]: pgmap v6105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:15 smithi171 ceph-mon[37789]: pgmap v6105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:16.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:16.580+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:38:16.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:16.580+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:38:16.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:16.581+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:38:16.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:16.581+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:38:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10640.866250 secs 2024-07-26T11:38:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10640.866250 secs 2024-07-26T11:38:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10640.866250 secs 2024-07-26T11:38:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:18 smithi044 ceph-mon[32404]: pgmap v6106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:18 smithi116 ceph-mon[37097]: pgmap v6106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:18 smithi171 ceph-mon[37789]: pgmap v6106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:20 smithi044 ceph-mon[32404]: pgmap v6107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:20 smithi116 ceph-mon[37097]: pgmap v6107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:20 smithi171 ceph-mon[37789]: pgmap v6107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10645.866415 secs 2024-07-26T11:38:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10645.866415 secs 2024-07-26T11:38:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10645.866415 secs 2024-07-26T11:38:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:22 smithi044 ceph-mon[32404]: pgmap v6108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:22 smithi116 ceph-mon[37097]: pgmap v6108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:22 smithi171 ceph-mon[37789]: pgmap v6108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:24 smithi044 ceph-mon[32404]: pgmap v6109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:24 smithi116 ceph-mon[37097]: pgmap v6109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:24 smithi171 ceph-mon[37789]: pgmap v6109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10650.866574 secs 2024-07-26T11:38:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:26 smithi044 ceph-mon[32404]: pgmap v6110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:27.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10650.866574 secs 2024-07-26T11:38:27.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:26 smithi116 ceph-mon[37097]: pgmap v6110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10650.866574 secs 2024-07-26T11:38:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:26 smithi171 ceph-mon[37789]: pgmap v6110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:29.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:29.092+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:38:29.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:29.092+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:29.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:29.092+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:38:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:28 smithi044 ceph-mon[32404]: pgmap v6111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:28 smithi116 ceph-mon[37097]: pgmap v6111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:28 smithi171 ceph-mon[37789]: pgmap v6111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:30 smithi044 ceph-mon[32404]: pgmap v6112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:30 smithi116 ceph-mon[37097]: pgmap v6112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:30 smithi171 ceph-mon[37789]: pgmap v6112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10655.866742 secs 2024-07-26T11:38:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10655.866742 secs 2024-07-26T11:38:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10655.866742 secs 2024-07-26T11:38:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:32 smithi044 ceph-mon[32404]: pgmap v6113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:32 smithi116 ceph-mon[37097]: pgmap v6113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:32 smithi171 ceph-mon[37789]: pgmap v6113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:33 smithi044 ceph-mon[32404]: pgmap v6114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:33 smithi116 ceph-mon[37097]: pgmap v6114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:33 smithi171 ceph-mon[37789]: pgmap v6114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10660.866907 secs 2024-07-26T11:38:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:36 smithi044 ceph-mon[32404]: pgmap v6115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10660.866907 secs 2024-07-26T11:38:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:36 smithi116 ceph-mon[37097]: pgmap v6115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10660.866907 secs 2024-07-26T11:38:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:36 smithi171 ceph-mon[37789]: pgmap v6115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:38 smithi044 ceph-mon[32404]: pgmap v6116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:38 smithi116 ceph-mon[37097]: pgmap v6116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:38 smithi171 ceph-mon[37789]: pgmap v6116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:40 smithi044 ceph-mon[32404]: pgmap v6117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:40 smithi116 ceph-mon[37097]: pgmap v6117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:40 smithi171 ceph-mon[37789]: pgmap v6117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10665.867072 secs 2024-07-26T11:38:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10665.867072 secs 2024-07-26T11:38:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10665.867072 secs 2024-07-26T11:38:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:42 smithi044 ceph-mon[32404]: pgmap v6118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:42 smithi116 ceph-mon[37097]: pgmap v6118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:42 smithi171 ceph-mon[37789]: pgmap v6118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:44.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:44.097+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:38:44.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:44.098+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:44.100 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:44.098+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:38:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:44 smithi044 ceph-mon[32404]: pgmap v6119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:44 smithi116 ceph-mon[37097]: pgmap v6119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:44 smithi171 ceph-mon[37789]: pgmap v6119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10670.867206 secs 2024-07-26T11:38:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:46 smithi044 ceph-mon[32404]: pgmap v6120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10670.867206 secs 2024-07-26T11:38:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:46 smithi116 ceph-mon[37097]: pgmap v6120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10670.867206 secs 2024-07-26T11:38:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:46 smithi171 ceph-mon[37789]: pgmap v6120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:48 smithi044 ceph-mon[32404]: pgmap v6121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:48 smithi116 ceph-mon[37097]: pgmap v6121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:48 smithi171 ceph-mon[37789]: pgmap v6121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:50 smithi044 ceph-mon[32404]: pgmap v6122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:50 smithi116 ceph-mon[37097]: pgmap v6122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:50 smithi171 ceph-mon[37789]: pgmap v6122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:38:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10675.867358 secs 2024-07-26T11:38:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10675.867358 secs 2024-07-26T11:38:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10675.867358 secs 2024-07-26T11:38:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:52 smithi044 ceph-mon[32404]: pgmap v6123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:52 smithi116 ceph-mon[37097]: pgmap v6123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:52 smithi171 ceph-mon[37789]: pgmap v6123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:53 smithi044 ceph-mon[32404]: pgmap v6124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:53 smithi116 ceph-mon[37097]: pgmap v6124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:53 smithi171 ceph-mon[37789]: pgmap v6124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:38:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10680.867525 secs 2024-07-26T11:38:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:56 smithi044 ceph-mon[32404]: pgmap v6125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10680.867525 secs 2024-07-26T11:38:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:56 smithi116 ceph-mon[37097]: pgmap v6125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10680.867525 secs 2024-07-26T11:38:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:56 smithi171 ceph-mon[37789]: pgmap v6125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:38:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:38:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:38:59.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:59.108+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:38:59.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:59.108+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:59.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:38:59.108+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:38:59.111 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:58 smithi044 ceph-mon[32404]: pgmap v6126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:38:59.111 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:38:58 smithi044 ceph-mon[32404]: pgmap v6127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:38:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:58 smithi116 ceph-mon[37097]: pgmap v6126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:38:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:38:58 smithi116 ceph-mon[37097]: pgmap v6127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:38:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:58 smithi171 ceph-mon[37789]: pgmap v6126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:38:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:38:58 smithi171 ceph-mon[37789]: pgmap v6127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:00 smithi044 ceph-mon[32404]: pgmap v6128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:00 smithi116 ceph-mon[37097]: pgmap v6128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:00 smithi171 ceph-mon[37789]: pgmap v6128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10685.867670 secs 2024-07-26T11:39:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10685.867670 secs 2024-07-26T11:39:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10685.867670 secs 2024-07-26T11:39:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:02 smithi044 ceph-mon[32404]: pgmap v6129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:02 smithi116 ceph-mon[37097]: pgmap v6129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:02 smithi171 ceph-mon[37789]: pgmap v6129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:04 smithi044 ceph-mon[32404]: pgmap v6130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:04 smithi116 ceph-mon[37097]: pgmap v6130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:04 smithi171 ceph-mon[37789]: pgmap v6130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:05 smithi044 ceph-mon[32404]: pgmap v6131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:05 smithi116 ceph-mon[37097]: pgmap v6131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:05 smithi171 ceph-mon[37789]: pgmap v6131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:39:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10690.867830 secs 2024-07-26T11:39:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10690.867830 secs 2024-07-26T11:39:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10690.867830 secs 2024-07-26T11:39:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:07 smithi044 ceph-mon[32404]: pgmap v6132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:07 smithi116 ceph-mon[37097]: pgmap v6132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:07 smithi171 ceph-mon[37789]: pgmap v6132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:39:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:10 smithi044 ceph-mon[32404]: pgmap v6133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:10 smithi116 ceph-mon[37097]: pgmap v6133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:10 smithi171 ceph-mon[37789]: pgmap v6133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10695.867999 secs 2024-07-26T11:39:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:12 smithi044 ceph-mon[32404]: pgmap v6134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10695.867999 secs 2024-07-26T11:39:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:12 smithi116 ceph-mon[37097]: pgmap v6134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10695.867999 secs 2024-07-26T11:39:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:12 smithi171 ceph-mon[37789]: pgmap v6134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:14.108+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:39:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:14.108+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:39:14.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:14.108+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:14 smithi044 ceph-mon[32404]: pgmap v6135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:14 smithi116 ceph-mon[37097]: pgmap v6135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:14 smithi171 ceph-mon[37789]: pgmap v6135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:16 smithi044 ceph-mon[32404]: pgmap v6136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:16 smithi116 ceph-mon[37097]: pgmap v6136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:16 smithi171 ceph-mon[37789]: pgmap v6136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10700.868154 secs 2024-07-26T11:39:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10700.868154 secs 2024-07-26T11:39:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10700.868154 secs 2024-07-26T11:39:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:18 smithi044 ceph-mon[32404]: pgmap v6137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:18 smithi116 ceph-mon[37097]: pgmap v6137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:18 smithi171 ceph-mon[37789]: pgmap v6137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:20 smithi044 ceph-mon[32404]: pgmap v6138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:20 smithi116 ceph-mon[37097]: pgmap v6138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:20 smithi171 ceph-mon[37789]: pgmap v6138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10705.868302 secs 2024-07-26T11:39:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:22 smithi044 ceph-mon[32404]: pgmap v6139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10705.868302 secs 2024-07-26T11:39:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:22 smithi116 ceph-mon[37097]: pgmap v6139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10705.868302 secs 2024-07-26T11:39:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:22 smithi171 ceph-mon[37789]: pgmap v6139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:24 smithi044 ceph-mon[32404]: pgmap v6140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:24 smithi116 ceph-mon[37097]: pgmap v6140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:24 smithi171 ceph-mon[37789]: pgmap v6140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:26 smithi044 ceph-mon[32404]: pgmap v6141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:26 smithi116 ceph-mon[37097]: pgmap v6141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:26 smithi171 ceph-mon[37789]: pgmap v6141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10710.868466 secs 2024-07-26T11:39:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10710.868466 secs 2024-07-26T11:39:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10710.868466 secs 2024-07-26T11:39:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:28 smithi044 ceph-mon[32404]: pgmap v6142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:28 smithi116 ceph-mon[37097]: pgmap v6142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:28 smithi171 ceph-mon[37789]: pgmap v6142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:29.118+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:39:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:29.118+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:29.118+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:39:30.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:30 smithi044 ceph-mon[32404]: pgmap v6143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:30 smithi116 ceph-mon[37097]: pgmap v6143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:30 smithi171 ceph-mon[37789]: pgmap v6143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10715.868658 secs 2024-07-26T11:39:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:32 smithi044 ceph-mon[32404]: pgmap v6144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10715.868658 secs 2024-07-26T11:39:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:32 smithi116 ceph-mon[37097]: pgmap v6144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10715.868658 secs 2024-07-26T11:39:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:32 smithi171 ceph-mon[37789]: pgmap v6144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:34 smithi044 ceph-mon[32404]: pgmap v6145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:34 smithi116 ceph-mon[37097]: pgmap v6145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:34 smithi171 ceph-mon[37789]: pgmap v6145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:36 smithi044 ceph-mon[32404]: pgmap v6146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:36 smithi116 ceph-mon[37097]: pgmap v6146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:36 smithi171 ceph-mon[37789]: pgmap v6146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10720.868801 secs 2024-07-26T11:39:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10720.868801 secs 2024-07-26T11:39:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10720.868801 secs 2024-07-26T11:39:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:38 smithi044 ceph-mon[32404]: pgmap v6147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:38 smithi116 ceph-mon[37097]: pgmap v6147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:38 smithi171 ceph-mon[37789]: pgmap v6147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:40 smithi044 ceph-mon[32404]: pgmap v6148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:40 smithi116 ceph-mon[37097]: pgmap v6148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:40 smithi171 ceph-mon[37789]: pgmap v6148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10725.868896 secs 2024-07-26T11:39:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:42 smithi044 ceph-mon[32404]: pgmap v6149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10725.868896 secs 2024-07-26T11:39:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:42 smithi116 ceph-mon[37097]: pgmap v6149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10725.868896 secs 2024-07-26T11:39:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:42 smithi171 ceph-mon[37789]: pgmap v6149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:44.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:44.119+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:39:44.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:44.119+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:44.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:44.119+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:39:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:44 smithi044 ceph-mon[32404]: pgmap v6150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:44 smithi116 ceph-mon[37097]: pgmap v6150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:44 smithi171 ceph-mon[37789]: pgmap v6150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:46 smithi044 ceph-mon[32404]: pgmap v6151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:46 smithi116 ceph-mon[37097]: pgmap v6151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:46 smithi171 ceph-mon[37789]: pgmap v6151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10730.868998 secs 2024-07-26T11:39:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10730.868998 secs 2024-07-26T11:39:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10730.868998 secs 2024-07-26T11:39:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:48 smithi171 ceph-mon[37789]: pgmap v6152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:48 smithi044 ceph-mon[32404]: pgmap v6152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:48 smithi116 ceph-mon[37097]: pgmap v6152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:50 smithi171 ceph-mon[37789]: pgmap v6153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:50 smithi044 ceph-mon[32404]: pgmap v6153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:50 smithi116 ceph-mon[37097]: pgmap v6153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10735.869162 secs 2024-07-26T11:39:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:52 smithi171 ceph-mon[37789]: pgmap v6154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10735.869162 secs 2024-07-26T11:39:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:52 smithi044 ceph-mon[32404]: pgmap v6154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10735.869162 secs 2024-07-26T11:39:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:52 smithi116 ceph-mon[37097]: pgmap v6154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:54 smithi171 ceph-mon[37789]: pgmap v6155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:54 smithi044 ceph-mon[32404]: pgmap v6155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:54 smithi116 ceph-mon[37097]: pgmap v6155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:56 smithi171 ceph-mon[37789]: pgmap v6156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:56 smithi044 ceph-mon[32404]: pgmap v6156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:56 smithi116 ceph-mon[37097]: pgmap v6156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10740.869322 secs 2024-07-26T11:39:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10740.869322 secs 2024-07-26T11:39:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10740.869322 secs 2024-07-26T11:39:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:59.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:59.129+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:39:59.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:59.129+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:59.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:39:59.129+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:39:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:58 smithi044 ceph-mon[32404]: pgmap v6157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:39:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:39:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:58 smithi116 ceph-mon[37097]: pgmap v6157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:39:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:39:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:58 smithi171 ceph-mon[37789]: pgmap v6157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:39:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:40:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:39:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:40:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:39:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:40:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:39:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:40:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:00 smithi044 ceph-mon[32404]: pgmap v6158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:00 smithi044 ceph-mon[32404]: pgmap v6159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:00 smithi116 ceph-mon[37097]: pgmap v6158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:00 smithi116 ceph-mon[37097]: pgmap v6159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:00 smithi171 ceph-mon[37789]: pgmap v6158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:00 smithi171 ceph-mon[37789]: pgmap v6159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:40:02.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10745.869487 secs 2024-07-26T11:40:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10745.869487 secs 2024-07-26T11:40:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10745.869487 secs 2024-07-26T11:40:02.700 INFO:tasks.workunit.client.0.smithi044.stderr:Terminated 2024-07-26T11:40:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-07-26T11:40:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:02 smithi044 ceph-mon[32404]: pgmap v6160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:02 smithi116 ceph-mon[37097]: pgmap v6160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:02 smithi171 ceph-mon[37789]: pgmap v6160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:03 smithi044 ceph-mon[32404]: pgmap v6161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:03 smithi116 ceph-mon[37097]: pgmap v6161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:03 smithi171 ceph-mon[37789]: pgmap v6161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:06 smithi044 ceph-mon[32404]: pgmap v6162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:06 smithi116 ceph-mon[37097]: pgmap v6162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:06 smithi171 ceph-mon[37789]: pgmap v6162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10750.869671 secs 2024-07-26T11:40:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10750.869671 secs 2024-07-26T11:40:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10750.869671 secs 2024-07-26T11:40:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:08 smithi044 ceph-mon[32404]: pgmap v6163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:08 smithi116 ceph-mon[37097]: pgmap v6163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:08 smithi171 ceph-mon[37789]: pgmap v6163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:40:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:10 smithi044 ceph-mon[32404]: pgmap v6164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:10 smithi116 ceph-mon[37097]: pgmap v6164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:10 smithi171 ceph-mon[37789]: pgmap v6164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10755.869811 secs 2024-07-26T11:40:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:12 smithi044 ceph-mon[32404]: pgmap v6165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10755.869811 secs 2024-07-26T11:40:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:12 smithi116 ceph-mon[37097]: pgmap v6165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10755.869811 secs 2024-07-26T11:40:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:12 smithi171 ceph-mon[37789]: pgmap v6165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:14.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:14.134+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:40:14.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:14.134+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:14.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:14.134+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:40:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:14 smithi044 ceph-mon[32404]: pgmap v6166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:14 smithi116 ceph-mon[37097]: pgmap v6166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:14 smithi171 ceph-mon[37789]: pgmap v6166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:16 smithi044 ceph-mon[32404]: pgmap v6167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:16 smithi116 ceph-mon[37097]: pgmap v6167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:16 smithi171 ceph-mon[37789]: pgmap v6167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10760.869974 secs 2024-07-26T11:40:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10760.869974 secs 2024-07-26T11:40:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10760.869974 secs 2024-07-26T11:40:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:18 smithi044 ceph-mon[32404]: pgmap v6168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:18 smithi116 ceph-mon[37097]: pgmap v6168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:18 smithi171 ceph-mon[37789]: pgmap v6168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:20 smithi044 ceph-mon[32404]: pgmap v6169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:20 smithi116 ceph-mon[37097]: pgmap v6169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:20 smithi171 ceph-mon[37789]: pgmap v6169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10765.870094 secs 2024-07-26T11:40:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:22 smithi044 ceph-mon[32404]: pgmap v6170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10765.870094 secs 2024-07-26T11:40:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:22 smithi116 ceph-mon[37097]: pgmap v6170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10765.870094 secs 2024-07-26T11:40:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:22 smithi171 ceph-mon[37789]: pgmap v6170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:24 smithi044 ceph-mon[32404]: pgmap v6171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:24 smithi116 ceph-mon[37097]: pgmap v6171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:24 smithi171 ceph-mon[37789]: pgmap v6171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:26 smithi044 ceph-mon[32404]: pgmap v6172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:26 smithi116 ceph-mon[37097]: pgmap v6172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:26 smithi171 ceph-mon[37789]: pgmap v6172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10770.870241 secs 2024-07-26T11:40:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10770.870241 secs 2024-07-26T11:40:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10770.870241 secs 2024-07-26T11:40:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:28 smithi044 ceph-mon[32404]: pgmap v6173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:28.569 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:28 smithi116 ceph-mon[37097]: pgmap v6173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:28 smithi171 ceph-mon[37789]: pgmap v6173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:29.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:29.144+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:40:29.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:29.144+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:29.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:29.144+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:40:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:30 smithi044 ceph-mon[32404]: pgmap v6174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:30 smithi116 ceph-mon[37097]: pgmap v6174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:30 smithi171 ceph-mon[37789]: pgmap v6174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:32.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10775.870386 secs 2024-07-26T11:40:32.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:32 smithi044 ceph-mon[32404]: pgmap v6175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10775.870386 secs 2024-07-26T11:40:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:32 smithi116 ceph-mon[37097]: pgmap v6175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10775.870386 secs 2024-07-26T11:40:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:32 smithi171 ceph-mon[37789]: pgmap v6175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:34 smithi044 ceph-mon[32404]: pgmap v6176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:34 smithi116 ceph-mon[37097]: pgmap v6176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:34 smithi171 ceph-mon[37789]: pgmap v6176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:36 smithi044 ceph-mon[32404]: pgmap v6177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:36 smithi116 ceph-mon[37097]: pgmap v6177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:36 smithi171 ceph-mon[37789]: pgmap v6177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10780.870532 secs 2024-07-26T11:40:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10780.870532 secs 2024-07-26T11:40:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10780.870532 secs 2024-07-26T11:40:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:38 smithi044 ceph-mon[32404]: pgmap v6178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:38 smithi116 ceph-mon[37097]: pgmap v6178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:38 smithi171 ceph-mon[37789]: pgmap v6178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:40 smithi044 ceph-mon[32404]: pgmap v6179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:40 smithi116 ceph-mon[37097]: pgmap v6179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:40 smithi171 ceph-mon[37789]: pgmap v6179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10785.870696 secs 2024-07-26T11:40:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:42 smithi044 ceph-mon[32404]: pgmap v6180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10785.870696 secs 2024-07-26T11:40:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:42 smithi116 ceph-mon[37097]: pgmap v6180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10785.870696 secs 2024-07-26T11:40:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:42 smithi171 ceph-mon[37789]: pgmap v6180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:44.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:44.149+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:40:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:44.149+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:44.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:44.149+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:40:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:44 smithi044 ceph-mon[32404]: pgmap v6181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:44 smithi116 ceph-mon[37097]: pgmap v6181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:44 smithi171 ceph-mon[37789]: pgmap v6181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:46 smithi116 ceph-mon[37097]: pgmap v6182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:40:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:46 smithi171 ceph-mon[37789]: pgmap v6182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:40:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:46 smithi044 ceph-mon[32404]: pgmap v6182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:40:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10790.870835 secs 2024-07-26T11:40:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10790.870835 secs 2024-07-26T11:40:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10790.870835 secs 2024-07-26T11:40:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:48 smithi171 ceph-mon[37789]: pgmap v6183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:48 smithi044 ceph-mon[32404]: pgmap v6183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:48 smithi116 ceph-mon[37097]: pgmap v6183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:50 smithi171 ceph-mon[37789]: pgmap v6184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:50 smithi044 ceph-mon[32404]: pgmap v6184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:50 smithi116 ceph-mon[37097]: pgmap v6184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10795.870978 secs 2024-07-26T11:40:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:52 smithi171 ceph-mon[37789]: pgmap v6185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:40:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10795.870978 secs 2024-07-26T11:40:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:52 smithi044 ceph-mon[32404]: pgmap v6185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:40:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10795.870978 secs 2024-07-26T11:40:52.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:52 smithi116 ceph-mon[37097]: pgmap v6185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:40:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:54 smithi171 ceph-mon[37789]: pgmap v6186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:54 smithi044 ceph-mon[32404]: pgmap v6186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:54 smithi116 ceph-mon[37097]: pgmap v6186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:56 smithi171 ceph-mon[37789]: pgmap v6187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:56 smithi044 ceph-mon[32404]: pgmap v6187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:56 smithi116 ceph-mon[37097]: pgmap v6187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10800.871124 secs 2024-07-26T11:40:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10800.871124 secs 2024-07-26T11:40:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10800.871124 secs 2024-07-26T11:40:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:58 smithi171 ceph-mon[37789]: pgmap v6188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:58 smithi044 ceph-mon[32404]: pgmap v6188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:58 smithi116 ceph-mon[37097]: pgmap v6188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:40:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:59.159+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:40:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:59.159+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:59.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:40:59.159+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:40:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:40:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:40:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:40:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:00 smithi044 ceph-mon[32404]: pgmap v6189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:00 smithi116 ceph-mon[37097]: pgmap v6189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:00 smithi171 ceph-mon[37789]: pgmap v6189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10805.871220 secs 2024-07-26T11:41:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:02 smithi044 ceph-mon[32404]: pgmap v6190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10805.871220 secs 2024-07-26T11:41:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:02 smithi116 ceph-mon[37097]: pgmap v6190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10805.871220 secs 2024-07-26T11:41:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:02 smithi171 ceph-mon[37789]: pgmap v6190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:02.702 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-07-26T11:41:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:04 smithi044 ceph-mon[32404]: pgmap v6191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:05.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:04 smithi116 ceph-mon[37097]: pgmap v6191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:04 smithi171 ceph-mon[37789]: pgmap v6191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:41:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:05 smithi044 ceph-mon[32404]: pgmap v6192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:41:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:05 smithi044 ceph-mon[32404]: pgmap v6193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:05 smithi116 ceph-mon[37097]: pgmap v6192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:41:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:05 smithi116 ceph-mon[37097]: pgmap v6193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:05 smithi171 ceph-mon[37789]: pgmap v6192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:41:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:05 smithi171 ceph-mon[37789]: pgmap v6193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10810.871367 secs 2024-07-26T11:41:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:06 smithi044 ceph-mon[32404]: pgmap v6194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10810.871367 secs 2024-07-26T11:41:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:06 smithi116 ceph-mon[37097]: pgmap v6194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10810.871367 secs 2024-07-26T11:41:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:06 smithi171 ceph-mon[37789]: pgmap v6194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:09 smithi044 ceph-mon[32404]: pgmap v6195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:09 smithi116 ceph-mon[37097]: pgmap v6195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:09 smithi171 ceph-mon[37789]: pgmap v6195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:11 smithi044 ceph-mon[32404]: pgmap v6196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:11 smithi116 ceph-mon[37097]: pgmap v6196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:11 smithi171 ceph-mon[37789]: pgmap v6196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:41:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10815.871528 secs 2024-07-26T11:41:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10815.871528 secs 2024-07-26T11:41:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10815.871528 secs 2024-07-26T11:41:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:13 smithi044 ceph-mon[32404]: pgmap v6197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:13 smithi116 ceph-mon[37097]: pgmap v6197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:13 smithi171 ceph-mon[37789]: pgmap v6197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:14.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.164+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:41:14.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.164+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:14.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.164+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf5807c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf5807c000 comp rx=0 tx=0).stop 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:41:14.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.720+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:41:14.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.721+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:41:14.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:14.723+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf5807e070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:41:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:15 smithi044 ceph-mon[32404]: pgmap v6198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:15 smithi116 ceph-mon[37097]: pgmap v6198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:15 smithi171 ceph-mon[37789]: pgmap v6198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:16.615+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:41:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:16.615+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:41:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:16.615+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:41:16.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:16.616+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:41:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10820.871688 secs 2024-07-26T11:41:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:17 smithi044 ceph-mon[32404]: pgmap v6199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10820.871688 secs 2024-07-26T11:41:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:17 smithi116 ceph-mon[37097]: pgmap v6199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10820.871688 secs 2024-07-26T11:41:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:17 smithi171 ceph-mon[37789]: pgmap v6199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:19 smithi044 ceph-mon[32404]: pgmap v6200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:19 smithi116 ceph-mon[37097]: pgmap v6200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:19 smithi171 ceph-mon[37789]: pgmap v6200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:21 smithi044 ceph-mon[32404]: pgmap v6201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:21 smithi116 ceph-mon[37097]: pgmap v6201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:21 smithi171 ceph-mon[37789]: pgmap v6201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:22.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10825.871854 secs 2024-07-26T11:41:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10825.871854 secs 2024-07-26T11:41:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10825.871854 secs 2024-07-26T11:41:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:23 smithi044 ceph-mon[32404]: pgmap v6202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:23 smithi116 ceph-mon[37097]: pgmap v6202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:23 smithi171 ceph-mon[37789]: pgmap v6202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:25 smithi044 ceph-mon[32404]: pgmap v6203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:25 smithi116 ceph-mon[37097]: pgmap v6203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:25 smithi171 ceph-mon[37789]: pgmap v6203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10830.872024 secs 2024-07-26T11:41:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:27 smithi044 ceph-mon[32404]: pgmap v6204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10830.872024 secs 2024-07-26T11:41:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:27 smithi116 ceph-mon[37097]: pgmap v6204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10830.872024 secs 2024-07-26T11:41:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:27 smithi171 ceph-mon[37789]: pgmap v6204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:29.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:29.173+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:41:29.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:29.173+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:29.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:29.173+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:41:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:29 smithi044 ceph-mon[32404]: pgmap v6205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:29 smithi116 ceph-mon[37097]: pgmap v6205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:29 smithi171 ceph-mon[37789]: pgmap v6205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:31 smithi044 ceph-mon[32404]: pgmap v6206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:31 smithi116 ceph-mon[37097]: pgmap v6206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:31 smithi171 ceph-mon[37789]: pgmap v6206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10835.872158 secs 2024-07-26T11:41:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10835.872158 secs 2024-07-26T11:41:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10835.872158 secs 2024-07-26T11:41:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:33 smithi044 ceph-mon[32404]: pgmap v6207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:33 smithi116 ceph-mon[37097]: pgmap v6207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:33 smithi171 ceph-mon[37789]: pgmap v6207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:35 smithi044 ceph-mon[32404]: pgmap v6208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:35 smithi116 ceph-mon[37097]: pgmap v6208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:35 smithi171 ceph-mon[37789]: pgmap v6208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10840.872311 secs 2024-07-26T11:41:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:37 smithi044 ceph-mon[32404]: pgmap v6209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:41:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10840.872311 secs 2024-07-26T11:41:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:37 smithi116 ceph-mon[37097]: pgmap v6209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:41:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10840.872311 secs 2024-07-26T11:41:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:37 smithi171 ceph-mon[37789]: pgmap v6209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:41:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:39 smithi044 ceph-mon[32404]: pgmap v6210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:41:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:39 smithi116 ceph-mon[37097]: pgmap v6210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:41:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:39 smithi171 ceph-mon[37789]: pgmap v6210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:41:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:41 smithi044 ceph-mon[32404]: pgmap v6211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:41:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:41 smithi116 ceph-mon[37097]: pgmap v6211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:41:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:41 smithi171 ceph-mon[37789]: pgmap v6211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:41:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10845.872471 secs 2024-07-26T11:41:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10845.872471 secs 2024-07-26T11:41:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10845.872471 secs 2024-07-26T11:41:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:43 smithi116 ceph-mon[37097]: pgmap v6212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:43 smithi171 ceph-mon[37789]: pgmap v6212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:43 smithi044 ceph-mon[32404]: pgmap v6212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:41:44.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:44.178+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:41:44.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:44.178+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:41:44.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:44.178+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:45 smithi171 ceph-mon[37789]: pgmap v6213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:45 smithi044 ceph-mon[32404]: pgmap v6213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:45 smithi116 ceph-mon[37097]: pgmap v6213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10850.872661 secs 2024-07-26T11:41:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:47 smithi171 ceph-mon[37789]: pgmap v6214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10850.872661 secs 2024-07-26T11:41:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:47 smithi044 ceph-mon[32404]: pgmap v6214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10850.872661 secs 2024-07-26T11:41:47.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:47 smithi116 ceph-mon[37097]: pgmap v6214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:49 smithi171 ceph-mon[37789]: pgmap v6215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:49 smithi044 ceph-mon[32404]: pgmap v6215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:49 smithi116 ceph-mon[37097]: pgmap v6215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:51 smithi171 ceph-mon[37789]: pgmap v6216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:51 smithi044 ceph-mon[32404]: pgmap v6216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:51 smithi116 ceph-mon[37097]: pgmap v6216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10855.872833 secs 2024-07-26T11:41:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10855.872833 secs 2024-07-26T11:41:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10855.872833 secs 2024-07-26T11:41:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:53 smithi171 ceph-mon[37789]: pgmap v6217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:53 smithi044 ceph-mon[32404]: pgmap v6217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:53 smithi116 ceph-mon[37097]: pgmap v6217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:55 smithi171 ceph-mon[37789]: pgmap v6218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:55 smithi044 ceph-mon[32404]: pgmap v6218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:55 smithi116 ceph-mon[37097]: pgmap v6218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:41:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10860.872993 secs 2024-07-26T11:41:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:57 smithi171 ceph-mon[37789]: pgmap v6219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10860.872993 secs 2024-07-26T11:41:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:57 smithi044 ceph-mon[32404]: pgmap v6219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10860.872993 secs 2024-07-26T11:41:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:57 smithi116 ceph-mon[37097]: pgmap v6219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:59.188+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:41:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:59.188+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:59.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:41:59.188+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:41:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:41:59 smithi044 ceph-mon[32404]: pgmap v6220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:41:59 smithi171 ceph-mon[37789]: pgmap v6220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:41:59 smithi116 ceph-mon[37097]: pgmap v6220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:01 smithi171 ceph-mon[37789]: pgmap v6221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:01 smithi044 ceph-mon[32404]: pgmap v6221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:01 smithi116 ceph-mon[37097]: pgmap v6221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:02.703 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-07-26T11:42:02.703 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-07-26T11:42:02.704 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-07-26T11:42:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10865.873133 secs 2024-07-26T11:42:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10865.873133 secs 2024-07-26T11:42:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10865.873133 secs 2024-07-26T11:42:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:03 smithi044 ceph-mon[32404]: pgmap v6222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:03 smithi116 ceph-mon[37097]: pgmap v6222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:03.827 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi044 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-07-26T11:42:04.055 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d7254ea96e094deebb68ca12bc1ee78c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi044 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-07-26T11:42:04.058 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-07-26T11:42:04.068 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-07-26T11:42:04.068 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:42:04.068 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-07-26T11:42:04.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-07-26T11:42:04.073 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:03 smithi171 ceph-mon[37789]: pgmap v6222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:04.167 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-07-26T11:42:04.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-26T11:42:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:04.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 -- 172.21.15.44:0/2850175578 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d4109b10 msgr2=0x7f76d4109f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:42:04.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 --2- 172.21.15.44:0/2850175578 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d4109b10 0x7f76d4109f20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009900 tx=0x7f76c402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:42:04.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 -- 172.21.15.44:0/2850175578 shutdown_connections 2024-07-26T11:42:04.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 --2- 172.21.15.44:0/2850175578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d410fd90 0x7f76d41165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:04.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 --2- 172.21.15.44:0/2850175578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d410f440 0x7f76d410f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:04.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 --2- 172.21.15.44:0/2850175578 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d4109b10 0x7f76d4109f20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:04.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 -- 172.21.15.44:0/2850175578 >> 172.21.15.44:0/2850175578 conn(0x7f76d4079060 msgr2=0x7f76d4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:42:04.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 -- 172.21.15.44:0/2850175578 shutdown_connections 2024-07-26T11:42:04.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.544+0000 7f76da600640 1 -- 172.21.15.44:0/2850175578 wait complete. 2024-07-26T11:42:04.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.545+0000 7f76da600640 1 Processor -- start 2024-07-26T11:42:04.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.545+0000 7f76da600640 1 -- start start 2024-07-26T11:42:04.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76da600640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d4109b10 0x7f76d41a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:42:04.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76da600640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d410f440 0x7f76d41a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:42:04.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76da600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d410fd90 0x7f76d41a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76da600640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4119aa0 con 0x7f76d4109b10 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76da600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4119920 con 0x7f76d410fd90 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76da600640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4119c20 con 0x7f76d410f440 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76d3400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d4109b10 0x7f76d41a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76caa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d410f440 0x7f76d41a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76d3400640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d4109b10 0x7f76d41a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60030/0 (socket says 172.21.15.44:60030) 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76caa00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d410f440 0x7f76d41a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.44:42254/0 (socket says 172.21.15.44:42254) 2024-07-26T11:42:04.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76d3400640 1 -- 172.21.15.44:0/1321661336 learned_addr learned my addr 172.21.15.44:0/1321661336 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.546+0000 7f76d3e00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d410fd90 0x7f76d41a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76caa00640 1 -- 172.21.15.44:0/1321661336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d410fd90 msgr2=0x7f76d41a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76caa00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d410fd90 0x7f76d41a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76caa00640 1 -- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d4109b10 msgr2=0x7f76d41a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76caa00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d4109b10 0x7f76d41a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76caa00640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d41a63b0 con 0x7f76d410f440 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76d3400640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f76d4109b10 0x7f76d41a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76caa00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f76d410f440 0x7f76d41a0a20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00e960 tx=0x7f76bc00ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:42:04.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc00cc60 con 0x7f76d410f440 2024-07-26T11:42:04.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76bc00ce00 con 0x7f76d410f440 2024-07-26T11:42:04.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76da600640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d41a66a0 con 0x7f76d410f440 2024-07-26T11:42:04.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc01a780 con 0x7f76d410f440 2024-07-26T11:42:04.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.547+0000 7f76da600640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d41ace30 con 0x7f76d410f440 2024-07-26T11:42:04.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.549+0000 7f76da600640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d4109f20 con 0x7f76d410f440 2024-07-26T11:42:04.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.551+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f76bc011470 con 0x7f76d410f440 2024-07-26T11:42:04.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.552+0000 7f76d0c00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 0x7f76b80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:42:04.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.553+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 11237+0+0 (secure 0 0 0) 0x7f76bc0e4eb0 con 0x7f76d410f440 2024-07-26T11:42:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.553+0000 7f76d3400640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 0x7f76b80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:04.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.554+0000 7f76d3400640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 0x7f76b80b62d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f76c40096c0 tx=0x7f76c4005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:42:04.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.554+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f76bc016030 con 0x7f76d410f440 2024-07-26T11:42:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:04.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:04.815+0000 7f76da600640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f76d4063230 con 0x7f76b80b3dc0 2024-07-26T11:42:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:05.730 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:05 smithi044 ceph-mon[32404]: pgmap v6223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:05.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:05 smithi044 ceph-mon[32404]: from='client.25912 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:42:05.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:05.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:05.731 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:42:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:05 smithi116 ceph-mon[37097]: pgmap v6223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:05 smithi116 ceph-mon[37097]: from='client.25912 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:42:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:05.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:42:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:05 smithi171 ceph-mon[37789]: pgmap v6223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:05 smithi171 ceph-mon[37789]: from='client.25912 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:42:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:05.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:05.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:42:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:06 smithi044 ceph-mon[32404]: pgmap v6224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:42:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:06 smithi044 ceph-mon[32404]: pgmap v6225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:06 smithi116 ceph-mon[37097]: pgmap v6224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:42:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:06 smithi116 ceph-mon[37097]: pgmap v6225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:06 smithi171 ceph-mon[37789]: pgmap v6224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:42:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:06 smithi171 ceph-mon[37789]: pgmap v6225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10870.873301 secs 2024-07-26T11:42:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10870.873301 secs 2024-07-26T11:42:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10870.873301 secs 2024-07-26T11:42:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:08 smithi044 ceph-mon[32404]: pgmap v6226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:08 smithi116 ceph-mon[37097]: pgmap v6226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:08 smithi171 ceph-mon[37789]: pgmap v6226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:10 smithi044 ceph-mon[32404]: pgmap v6227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:10 smithi116 ceph-mon[37097]: pgmap v6227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:10 smithi171 ceph-mon[37789]: pgmap v6227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10875.873454 secs 2024-07-26T11:42:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:12 smithi044 ceph-mon[32404]: pgmap v6228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10875.873454 secs 2024-07-26T11:42:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:12 smithi116 ceph-mon[37097]: pgmap v6228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10875.873454 secs 2024-07-26T11:42:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:12 smithi171 ceph-mon[37789]: pgmap v6228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:14.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:14.193+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:42:14.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:14.193+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:14.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:14.193+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:42:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:14 smithi044 ceph-mon[32404]: pgmap v6229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:14 smithi116 ceph-mon[37097]: pgmap v6229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:14 smithi171 ceph-mon[37789]: pgmap v6229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:16 smithi044 ceph-mon[32404]: pgmap v6230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:16 smithi116 ceph-mon[37097]: pgmap v6230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:16 smithi171 ceph-mon[37789]: pgmap v6230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10880.873610 secs 2024-07-26T11:42:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10880.873610 secs 2024-07-26T11:42:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10880.873610 secs 2024-07-26T11:42:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:18 smithi044 ceph-mon[32404]: pgmap v6231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:18 smithi116 ceph-mon[37097]: pgmap v6231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:18 smithi171 ceph-mon[37789]: pgmap v6231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:20 smithi044 ceph-mon[32404]: pgmap v6232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:20 smithi116 ceph-mon[37097]: pgmap v6232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:20 smithi171 ceph-mon[37789]: pgmap v6232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10885.873773 secs 2024-07-26T11:42:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:22 smithi044 ceph-mon[32404]: pgmap v6233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10885.873773 secs 2024-07-26T11:42:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:22 smithi116 ceph-mon[37097]: pgmap v6233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10885.873773 secs 2024-07-26T11:42:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:22 smithi171 ceph-mon[37789]: pgmap v6233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:24 smithi044 ceph-mon[32404]: pgmap v6234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:24 smithi116 ceph-mon[37097]: pgmap v6234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:24 smithi171 ceph-mon[37789]: pgmap v6234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:26 smithi044 ceph-mon[32404]: pgmap v6235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10890.873939 secs 2024-07-26T11:42:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:26 smithi116 ceph-mon[37097]: pgmap v6235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10890.873939 secs 2024-07-26T11:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:26 smithi171 ceph-mon[37789]: pgmap v6235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10890.873939 secs 2024-07-26T11:42:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:28 smithi044 ceph-mon[32404]: pgmap v6236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:28 smithi116 ceph-mon[37097]: pgmap v6236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:28 smithi171 ceph-mon[37789]: pgmap v6236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:29.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:29.203+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:42:29.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:29.203+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:29.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:29.203+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:42:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:30 smithi044 ceph-mon[32404]: pgmap v6237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:30 smithi116 ceph-mon[37097]: pgmap v6237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:30 smithi171 ceph-mon[37789]: pgmap v6237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10895.874108 secs 2024-07-26T11:42:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10895.874108 secs 2024-07-26T11:42:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10895.874108 secs 2024-07-26T11:42:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:32 smithi044 ceph-mon[32404]: pgmap v6238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:32 smithi116 ceph-mon[37097]: pgmap v6238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:32 smithi171 ceph-mon[37789]: pgmap v6238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:34 smithi044 ceph-mon[32404]: pgmap v6239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:34 smithi116 ceph-mon[37097]: pgmap v6239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:34 smithi171 ceph-mon[37789]: pgmap v6239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:36 smithi044 ceph-mon[32404]: pgmap v6240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10900.874257 secs 2024-07-26T11:42:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:36 smithi116 ceph-mon[37097]: pgmap v6240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10900.874257 secs 2024-07-26T11:42:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:36 smithi171 ceph-mon[37789]: pgmap v6240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10900.874257 secs 2024-07-26T11:42:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:38 smithi044 ceph-mon[32404]: pgmap v6241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:38 smithi116 ceph-mon[37097]: pgmap v6241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:38 smithi171 ceph-mon[37789]: pgmap v6241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:40 smithi044 ceph-mon[32404]: pgmap v6242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:40 smithi116 ceph-mon[37097]: pgmap v6242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:40 smithi171 ceph-mon[37789]: pgmap v6242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10905.874440 secs 2024-07-26T11:42:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10905.874440 secs 2024-07-26T11:42:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10905.874440 secs 2024-07-26T11:42:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:42 smithi044 ceph-mon[32404]: pgmap v6243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:42 smithi116 ceph-mon[37097]: pgmap v6243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:42 smithi171 ceph-mon[37789]: pgmap v6243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:44.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:44.208+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:42:44.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:44.208+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:44.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:44.208+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:42:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:44 smithi044 ceph-mon[32404]: pgmap v6244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:44 smithi116 ceph-mon[37097]: pgmap v6244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:44 smithi171 ceph-mon[37789]: pgmap v6244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:46.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:46.630+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:42:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:46.630+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:42:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:46.631+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:42:46.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:46.631+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:42:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:46 smithi116 ceph-mon[37097]: pgmap v6245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10910.874599 secs 2024-07-26T11:42:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:46 smithi171 ceph-mon[37789]: pgmap v6245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10910.874599 secs 2024-07-26T11:42:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:46 smithi044 ceph-mon[32404]: pgmap v6245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10910.874599 secs 2024-07-26T11:42:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:48 smithi171 ceph-mon[37789]: pgmap v6246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:48 smithi044 ceph-mon[32404]: pgmap v6246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:48 smithi116 ceph-mon[37097]: pgmap v6246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:50 smithi171 ceph-mon[37789]: pgmap v6247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:50 smithi044 ceph-mon[32404]: pgmap v6247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:50 smithi116 ceph-mon[37097]: pgmap v6247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:42:51.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10915.874775 secs 2024-07-26T11:42:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10915.874775 secs 2024-07-26T11:42:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10915.874775 secs 2024-07-26T11:42:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:52 smithi171 ceph-mon[37789]: pgmap v6248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:52 smithi044 ceph-mon[32404]: pgmap v6248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:52 smithi116 ceph-mon[37097]: pgmap v6248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:54 smithi171 ceph-mon[37789]: pgmap v6249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:54 smithi044 ceph-mon[32404]: pgmap v6249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:54 smithi116 ceph-mon[37097]: pgmap v6249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:56 smithi171 ceph-mon[37789]: pgmap v6250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:56 smithi044 ceph-mon[32404]: pgmap v6250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:56 smithi116 ceph-mon[37097]: pgmap v6250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10920.874920 secs 2024-07-26T11:42:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10920.874920 secs 2024-07-26T11:42:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10920.874920 secs 2024-07-26T11:42:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:42:58 smithi171 ceph-mon[37789]: pgmap v6251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:42:58 smithi044 ceph-mon[32404]: pgmap v6251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:42:58 smithi116 ceph-mon[37097]: pgmap v6251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:42:59.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:59.209+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:42:59.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:59.209+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:42:59.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:42:59.209+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:00 smithi044 ceph-mon[32404]: pgmap v6252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:00 smithi116 ceph-mon[37097]: pgmap v6252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:00 smithi171 ceph-mon[37789]: pgmap v6252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10925.875066 secs 2024-07-26T11:43:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:02 smithi044 ceph-mon[32404]: pgmap v6253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10925.875066 secs 2024-07-26T11:43:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:02 smithi116 ceph-mon[37097]: pgmap v6253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10925.875066 secs 2024-07-26T11:43:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:02 smithi171 ceph-mon[37789]: pgmap v6253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:04 smithi044 ceph-mon[32404]: pgmap v6254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:04 smithi116 ceph-mon[37097]: pgmap v6254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:04 smithi171 ceph-mon[37789]: pgmap v6254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:06.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:06 smithi044 ceph-mon[32404]: pgmap v6255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:06.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:06 smithi116 ceph-mon[37097]: pgmap v6255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:06.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:06 smithi171 ceph-mon[37789]: pgmap v6255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10930.875156 secs 2024-07-26T11:43:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:07 smithi044 ceph-mon[32404]: pgmap v6256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:43:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:07 smithi044 ceph-mon[32404]: pgmap v6257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:07.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:43:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10930.875156 secs 2024-07-26T11:43:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:07 smithi116 ceph-mon[37097]: pgmap v6256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:43:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:07 smithi116 ceph-mon[37097]: pgmap v6257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:43:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10930.875156 secs 2024-07-26T11:43:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:07 smithi171 ceph-mon[37789]: pgmap v6256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:43:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:07 smithi171 ceph-mon[37789]: pgmap v6257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:07.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:43:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:09 smithi044 ceph-mon[32404]: pgmap v6258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:09 smithi116 ceph-mon[37097]: pgmap v6258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:09 smithi171 ceph-mon[37789]: pgmap v6258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:11 smithi171 ceph-mon[37789]: pgmap v6259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:11 smithi044 ceph-mon[32404]: pgmap v6259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:11 smithi116 ceph-mon[37097]: pgmap v6259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10935.875318 secs 2024-07-26T11:43:12.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10935.875318 secs 2024-07-26T11:43:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10935.875318 secs 2024-07-26T11:43:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:13 smithi171 ceph-mon[37789]: pgmap v6260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:13 smithi044 ceph-mon[32404]: pgmap v6260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:13 smithi116 ceph-mon[37097]: pgmap v6260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:14.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:14.214+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:43:14.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:14.214+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:43:14.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:14.214+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:15 smithi044 ceph-mon[32404]: pgmap v6261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:15 smithi116 ceph-mon[37097]: pgmap v6261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:15 smithi171 ceph-mon[37789]: pgmap v6261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:43:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10940.875477 secs 2024-07-26T11:43:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10940.875477 secs 2024-07-26T11:43:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10940.875477 secs 2024-07-26T11:43:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:17 smithi044 ceph-mon[32404]: pgmap v6262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:43:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:17 smithi116 ceph-mon[37097]: pgmap v6262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:43:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:17 smithi171 ceph-mon[37789]: pgmap v6262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:43:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:19 smithi044 ceph-mon[32404]: pgmap v6263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:19 smithi116 ceph-mon[37097]: pgmap v6263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:19 smithi171 ceph-mon[37789]: pgmap v6263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10945.875666 secs 2024-07-26T11:43:22.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:21 smithi044 ceph-mon[32404]: pgmap v6264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10945.875666 secs 2024-07-26T11:43:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:21 smithi116 ceph-mon[37097]: pgmap v6264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10945.875666 secs 2024-07-26T11:43:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:21 smithi171 ceph-mon[37789]: pgmap v6264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:23 smithi044 ceph-mon[32404]: pgmap v6265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:23 smithi116 ceph-mon[37097]: pgmap v6265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:23 smithi171 ceph-mon[37789]: pgmap v6265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:25 smithi044 ceph-mon[32404]: pgmap v6266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:25 smithi116 ceph-mon[37097]: pgmap v6266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:25 smithi171 ceph-mon[37789]: pgmap v6266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10950.875831 secs 2024-07-26T11:43:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10950.875831 secs 2024-07-26T11:43:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10950.875831 secs 2024-07-26T11:43:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:27 smithi044 ceph-mon[32404]: pgmap v6267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:27 smithi116 ceph-mon[37097]: pgmap v6267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:27 smithi171 ceph-mon[37789]: pgmap v6267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:29.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:29.217+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:43:29.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:29.217+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:43:29.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:29.217+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:30.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:29 smithi044 ceph-mon[32404]: pgmap v6268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:29 smithi116 ceph-mon[37097]: pgmap v6268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:29 smithi171 ceph-mon[37789]: pgmap v6268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10955.875918 secs 2024-07-26T11:43:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:31 smithi044 ceph-mon[32404]: pgmap v6269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10955.875918 secs 2024-07-26T11:43:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:31 smithi116 ceph-mon[37097]: pgmap v6269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10955.875918 secs 2024-07-26T11:43:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:31 smithi171 ceph-mon[37789]: pgmap v6269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:33 smithi044 ceph-mon[32404]: pgmap v6270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:33 smithi116 ceph-mon[37097]: pgmap v6270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:33 smithi171 ceph-mon[37789]: pgmap v6270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:35 smithi044 ceph-mon[32404]: pgmap v6271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:35 smithi116 ceph-mon[37097]: pgmap v6271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:35 smithi171 ceph-mon[37789]: pgmap v6271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10960.876081 secs 2024-07-26T11:43:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10960.876081 secs 2024-07-26T11:43:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10960.876081 secs 2024-07-26T11:43:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:37 smithi044 ceph-mon[32404]: pgmap v6272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:38.060 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:37 smithi116 ceph-mon[37097]: pgmap v6272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:37 smithi171 ceph-mon[37789]: pgmap v6272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:39 smithi044 ceph-mon[32404]: pgmap v6273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:39 smithi116 ceph-mon[37097]: pgmap v6273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:39 smithi171 ceph-mon[37789]: pgmap v6273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10965.876249 secs 2024-07-26T11:43:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:41 smithi044 ceph-mon[32404]: pgmap v6274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10965.876249 secs 2024-07-26T11:43:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:41 smithi116 ceph-mon[37097]: pgmap v6274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10965.876249 secs 2024-07-26T11:43:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:41 smithi171 ceph-mon[37789]: pgmap v6274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:43 smithi116 ceph-mon[37097]: pgmap v6275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:43 smithi171 ceph-mon[37789]: pgmap v6275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:44.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:44.222+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:43:44.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:44.222+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:43:44.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:44.222+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:44.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:43 smithi044 ceph-mon[32404]: pgmap v6275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:45 smithi171 ceph-mon[37789]: pgmap v6276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:45 smithi044 ceph-mon[32404]: pgmap v6276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:45 smithi116 ceph-mon[37097]: pgmap v6276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10970.876412 secs 2024-07-26T11:43:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10970.876412 secs 2024-07-26T11:43:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10970.876412 secs 2024-07-26T11:43:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:47 smithi171 ceph-mon[37789]: pgmap v6277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:47 smithi044 ceph-mon[32404]: pgmap v6277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:47 smithi116 ceph-mon[37097]: pgmap v6277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:49 smithi171 ceph-mon[37789]: pgmap v6278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:49 smithi044 ceph-mon[32404]: pgmap v6278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:49 smithi116 ceph-mon[37097]: pgmap v6278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10975.876548 secs 2024-07-26T11:43:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:51 smithi171 ceph-mon[37789]: pgmap v6279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10975.876548 secs 2024-07-26T11:43:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:51 smithi044 ceph-mon[32404]: pgmap v6279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10975.876548 secs 2024-07-26T11:43:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:51 smithi116 ceph-mon[37097]: pgmap v6279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:53 smithi044 ceph-mon[32404]: pgmap v6280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:53 smithi171 ceph-mon[37789]: pgmap v6280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:53 smithi116 ceph-mon[37097]: pgmap v6280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:43:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:55 smithi171 ceph-mon[37789]: pgmap v6281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:55 smithi044 ceph-mon[32404]: pgmap v6281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:55 smithi116 ceph-mon[37097]: pgmap v6281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:43:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10980.876688 secs 2024-07-26T11:43:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10980.876688 secs 2024-07-26T11:43:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10980.876688 secs 2024-07-26T11:43:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:57 smithi171 ceph-mon[37789]: pgmap v6282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:57 smithi044 ceph-mon[32404]: pgmap v6282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:57 smithi116 ceph-mon[37097]: pgmap v6282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:59.233+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:43:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:59.233+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:43:59.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:43:59.233+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:43:59 smithi171 ceph-mon[37789]: pgmap v6283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:44:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:43:59 smithi044 ceph-mon[32404]: pgmap v6283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:44:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:43:59 smithi116 ceph-mon[37097]: pgmap v6283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:44:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10985.876826 secs 2024-07-26T11:44:02.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:01 smithi044 ceph-mon[32404]: pgmap v6284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10985.876826 secs 2024-07-26T11:44:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:01 smithi116 ceph-mon[37097]: pgmap v6284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10985.876826 secs 2024-07-26T11:44:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:01 smithi171 ceph-mon[37789]: pgmap v6284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:03 smithi044 ceph-mon[32404]: pgmap v6285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:03 smithi116 ceph-mon[37097]: pgmap v6285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:03 smithi171 ceph-mon[37789]: pgmap v6285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:05 smithi044 ceph-mon[32404]: pgmap v6286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:05 smithi116 ceph-mon[37097]: pgmap v6286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:05 smithi171 ceph-mon[37789]: pgmap v6286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10990.876919 secs 2024-07-26T11:44:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10990.876919 secs 2024-07-26T11:44:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10990.876919 secs 2024-07-26T11:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:07 smithi044 ceph-mon[32404]: pgmap v6287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:08.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:44:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:07 smithi116 ceph-mon[37097]: pgmap v6287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:08.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:44:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:07 smithi171 ceph-mon[37789]: pgmap v6287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:44:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:08 smithi044 ceph-mon[32404]: pgmap v6288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:44:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:08 smithi044 ceph-mon[32404]: pgmap v6289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:08 smithi116 ceph-mon[37097]: pgmap v6288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:44:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:08 smithi116 ceph-mon[37097]: pgmap v6289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:08 smithi171 ceph-mon[37789]: pgmap v6288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:44:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:08 smithi171 ceph-mon[37789]: pgmap v6289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:10 smithi044 ceph-mon[32404]: pgmap v6290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:10 smithi116 ceph-mon[37097]: pgmap v6290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:10 smithi171 ceph-mon[37789]: pgmap v6290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 10995.877073 secs 2024-07-26T11:44:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 10995.877073 secs 2024-07-26T11:44:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 10995.877073 secs 2024-07-26T11:44:13.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:12 smithi116 ceph-mon[37097]: pgmap v6291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:12 smithi044 ceph-mon[32404]: pgmap v6291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:12 smithi171 ceph-mon[37789]: pgmap v6291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:14.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:14.238+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:44:14.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:14.238+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:44:14.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:14.239+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:14 smithi044 ceph-mon[32404]: pgmap v6292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:14 smithi116 ceph-mon[37097]: pgmap v6292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:14 smithi171 ceph-mon[37789]: pgmap v6292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:44:16.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:16.648+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:44:16.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:16.648+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:44:16.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:16.648+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:44:16.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:16.648+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:44:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11000.877220 secs 2024-07-26T11:44:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:16 smithi044 ceph-mon[32404]: pgmap v6293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11000.877220 secs 2024-07-26T11:44:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:16 smithi116 ceph-mon[37097]: pgmap v6293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11000.877220 secs 2024-07-26T11:44:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:16 smithi171 ceph-mon[37789]: pgmap v6293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:44:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:18 smithi044 ceph-mon[32404]: pgmap v6294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:44:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:18 smithi116 ceph-mon[37097]: pgmap v6294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:44:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:18 smithi171 ceph-mon[37789]: pgmap v6294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:44:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:20 smithi044 ceph-mon[32404]: pgmap v6295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:20 smithi116 ceph-mon[37097]: pgmap v6295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:20 smithi171 ceph-mon[37789]: pgmap v6295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11005.877358 secs 2024-07-26T11:44:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11005.877358 secs 2024-07-26T11:44:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11005.877358 secs 2024-07-26T11:44:23.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:22 smithi171 ceph-mon[37789]: pgmap v6296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:22 smithi044 ceph-mon[32404]: pgmap v6296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:22 smithi116 ceph-mon[37097]: pgmap v6296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:24 smithi044 ceph-mon[32404]: pgmap v6297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:24 smithi116 ceph-mon[37097]: pgmap v6297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:24 smithi171 ceph-mon[37789]: pgmap v6297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11010.877522 secs 2024-07-26T11:44:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:26 smithi044 ceph-mon[32404]: pgmap v6298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11010.877522 secs 2024-07-26T11:44:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:26 smithi116 ceph-mon[37097]: pgmap v6298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11010.877522 secs 2024-07-26T11:44:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:26 smithi171 ceph-mon[37789]: pgmap v6298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:29.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:29.249+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:44:29.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:29.249+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:44:29.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:29.249+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:29.252 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:28 smithi044 ceph-mon[32404]: pgmap v6299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:28 smithi116 ceph-mon[37097]: pgmap v6299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:28 smithi171 ceph-mon[37789]: pgmap v6299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:30 smithi044 ceph-mon[32404]: pgmap v6300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:30 smithi116 ceph-mon[37097]: pgmap v6300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:30 smithi171 ceph-mon[37789]: pgmap v6300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11015.877686 secs 2024-07-26T11:44:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11015.877686 secs 2024-07-26T11:44:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11015.877686 secs 2024-07-26T11:44:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:32 smithi044 ceph-mon[32404]: pgmap v6301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:32 smithi116 ceph-mon[37097]: pgmap v6301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:32 smithi171 ceph-mon[37789]: pgmap v6301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:34 smithi044 ceph-mon[32404]: pgmap v6302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:34 smithi116 ceph-mon[37097]: pgmap v6302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:34 smithi171 ceph-mon[37789]: pgmap v6302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11020.877770 secs 2024-07-26T11:44:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:36 smithi044 ceph-mon[32404]: pgmap v6303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11020.877770 secs 2024-07-26T11:44:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:36 smithi116 ceph-mon[37097]: pgmap v6303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11020.877770 secs 2024-07-26T11:44:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:36 smithi171 ceph-mon[37789]: pgmap v6303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:38 smithi044 ceph-mon[32404]: pgmap v6304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:38 smithi116 ceph-mon[37097]: pgmap v6304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:38 smithi171 ceph-mon[37789]: pgmap v6304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:40 smithi044 ceph-mon[32404]: pgmap v6305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:40 smithi116 ceph-mon[37097]: pgmap v6305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:40 smithi171 ceph-mon[37789]: pgmap v6305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11025.877912 secs 2024-07-26T11:44:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11025.877912 secs 2024-07-26T11:44:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11025.877912 secs 2024-07-26T11:44:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:43 smithi044 ceph-mon[32404]: pgmap v6306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:43 smithi116 ceph-mon[37097]: pgmap v6306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:43 smithi171 ceph-mon[37789]: pgmap v6306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:44.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:44.254+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:44:44.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:44.254+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:44:44.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:44.254+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:45 smithi044 ceph-mon[32404]: pgmap v6307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:45 smithi116 ceph-mon[37097]: pgmap v6307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:45 smithi171 ceph-mon[37789]: pgmap v6307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11030.878076 secs 2024-07-26T11:44:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:47 smithi116 ceph-mon[37097]: pgmap v6308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11030.878076 secs 2024-07-26T11:44:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:47 smithi171 ceph-mon[37789]: pgmap v6308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11030.878076 secs 2024-07-26T11:44:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:47 smithi044 ceph-mon[32404]: pgmap v6308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:49 smithi171 ceph-mon[37789]: pgmap v6309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:49 smithi044 ceph-mon[32404]: pgmap v6309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:49 smithi116 ceph-mon[37097]: pgmap v6309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:51 smithi171 ceph-mon[37789]: pgmap v6310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:51 smithi044 ceph-mon[32404]: pgmap v6310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:51 smithi116 ceph-mon[37097]: pgmap v6310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:44:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11035.878234 secs 2024-07-26T11:44:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:52 smithi171 ceph-mon[37789]: pgmap v6311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11035.878234 secs 2024-07-26T11:44:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:52 smithi044 ceph-mon[32404]: pgmap v6311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11035.878234 secs 2024-07-26T11:44:52.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:52 smithi116 ceph-mon[37097]: pgmap v6311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:54 smithi044 ceph-mon[32404]: pgmap v6312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:54 smithi116 ceph-mon[37097]: pgmap v6312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:54 smithi171 ceph-mon[37789]: pgmap v6312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11040.878371 secs 2024-07-26T11:44:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:56 smithi044 ceph-mon[32404]: pgmap v6313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11040.878371 secs 2024-07-26T11:44:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:56 smithi116 ceph-mon[37097]: pgmap v6313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11040.878371 secs 2024-07-26T11:44:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:56 smithi171 ceph-mon[37789]: pgmap v6313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:44:58 smithi044 ceph-mon[32404]: pgmap v6314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:59.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:59.264+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:44:59.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:59.264+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:44:59.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:44:59.264+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:44:58 smithi116 ceph-mon[37097]: pgmap v6314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:44:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:44:58 smithi171 ceph-mon[37789]: pgmap v6314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:00 smithi044 ceph-mon[32404]: pgmap v6315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:00 smithi116 ceph-mon[37097]: pgmap v6315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:00 smithi171 ceph-mon[37789]: pgmap v6315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:45:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11045.878521 secs 2024-07-26T11:45:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11045.878521 secs 2024-07-26T11:45:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11045.878521 secs 2024-07-26T11:45:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:02 smithi044 ceph-mon[32404]: pgmap v6316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:02 smithi116 ceph-mon[37097]: pgmap v6316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:02 smithi171 ceph-mon[37789]: pgmap v6316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:04.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:04.578+0000 7f76cb400640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f76ac0034c0 con 0x7f76d410f440 2024-07-26T11:45:04.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:04.579+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f76ac0034c0 con 0x7f76d410f440 2024-07-26T11:45:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:04 smithi044 ceph-mon[32404]: pgmap v6317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:04 smithi116 ceph-mon[37097]: pgmap v6317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:04 smithi171 ceph-mon[37789]: pgmap v6317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11050.878665 secs 2024-07-26T11:45:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:06 smithi044 ceph-mon[32404]: pgmap v6318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:45:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11050.878665 secs 2024-07-26T11:45:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:06 smithi116 ceph-mon[37097]: pgmap v6318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:45:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11050.878665 secs 2024-07-26T11:45:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:06 smithi171 ceph-mon[37789]: pgmap v6318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:45:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:08 smithi044 ceph-mon[32404]: pgmap v6319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:08 smithi116 ceph-mon[37097]: pgmap v6319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:08 smithi171 ceph-mon[37789]: pgmap v6319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:10.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:45:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:45:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:10.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:45:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:10 smithi044 ceph-mon[32404]: pgmap v6320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:45:11.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:10 smithi044 ceph-mon[32404]: pgmap v6321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:10 smithi116 ceph-mon[37097]: pgmap v6320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:45:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:10 smithi116 ceph-mon[37097]: pgmap v6321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:10 smithi171 ceph-mon[37789]: pgmap v6320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:45:12.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:10 smithi171 ceph-mon[37789]: pgmap v6321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11055.878806 secs 2024-07-26T11:45:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11055.878806 secs 2024-07-26T11:45:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11055.878806 secs 2024-07-26T11:45:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:13 smithi044 ceph-mon[32404]: pgmap v6322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:13 smithi116 ceph-mon[37097]: pgmap v6322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:13 smithi171 ceph-mon[37789]: pgmap v6322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:14.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:14.269+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:45:14.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:14.269+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:14.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:14.269+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:45:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:15 smithi116 ceph-mon[37097]: pgmap v6323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:15 smithi171 ceph-mon[37789]: pgmap v6323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:15 smithi044 ceph-mon[32404]: pgmap v6323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:17 smithi171 ceph-mon[37789]: pgmap v6324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11060.878942 secs 2024-07-26T11:45:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:17 smithi044 ceph-mon[32404]: pgmap v6324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11060.878942 secs 2024-07-26T11:45:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:17 smithi116 ceph-mon[37097]: pgmap v6324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11060.878942 secs 2024-07-26T11:45:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:19 smithi171 ceph-mon[37789]: pgmap v6325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:19 smithi044 ceph-mon[32404]: pgmap v6325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:19 smithi116 ceph-mon[37097]: pgmap v6325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:21 smithi171 ceph-mon[37789]: pgmap v6326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:21 smithi044 ceph-mon[32404]: pgmap v6326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:21 smithi116 ceph-mon[37097]: pgmap v6326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11065.879081 secs 2024-07-26T11:45:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11065.879081 secs 2024-07-26T11:45:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11065.879081 secs 2024-07-26T11:45:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:23 smithi171 ceph-mon[37789]: pgmap v6327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:23 smithi044 ceph-mon[32404]: pgmap v6327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:23 smithi116 ceph-mon[37097]: pgmap v6327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:24 smithi171 ceph-mon[37789]: pgmap v6328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:24 smithi044 ceph-mon[32404]: pgmap v6328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:24 smithi116 ceph-mon[37097]: pgmap v6328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:26 smithi044 ceph-mon[32404]: pgmap v6329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:26 smithi116 ceph-mon[37097]: pgmap v6329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:26 smithi171 ceph-mon[37789]: pgmap v6329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:27.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11070.879179 secs 2024-07-26T11:45:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11070.879179 secs 2024-07-26T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11070.879179 secs 2024-07-26T11:45:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:28 smithi044 ceph-mon[32404]: pgmap v6330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:28 smithi116 ceph-mon[37097]: pgmap v6330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:28 smithi171 ceph-mon[37789]: pgmap v6330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:29.279+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:45:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:29.279+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:29.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:29.279+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:45:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:30 smithi044 ceph-mon[32404]: pgmap v6331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:30 smithi116 ceph-mon[37097]: pgmap v6331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:30 smithi171 ceph-mon[37789]: pgmap v6331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:45:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11075.879338 secs 2024-07-26T11:45:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:32 smithi044 ceph-mon[32404]: pgmap v6332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11075.879338 secs 2024-07-26T11:45:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:32 smithi116 ceph-mon[37097]: pgmap v6332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11075.879338 secs 2024-07-26T11:45:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:32 smithi171 ceph-mon[37789]: pgmap v6332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:34 smithi044 ceph-mon[32404]: pgmap v6333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:34 smithi116 ceph-mon[37097]: pgmap v6333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:34 smithi171 ceph-mon[37789]: pgmap v6333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:36 smithi044 ceph-mon[32404]: pgmap v6334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:36 smithi116 ceph-mon[37097]: pgmap v6334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:36 smithi171 ceph-mon[37789]: pgmap v6334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11080.879497 secs 2024-07-26T11:45:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11080.879497 secs 2024-07-26T11:45:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11080.879497 secs 2024-07-26T11:45:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:38 smithi044 ceph-mon[32404]: pgmap v6335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:38 smithi116 ceph-mon[37097]: pgmap v6335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:38 smithi171 ceph-mon[37789]: pgmap v6335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:40 smithi044 ceph-mon[32404]: pgmap v6336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:40 smithi116 ceph-mon[37097]: pgmap v6336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:40 smithi171 ceph-mon[37789]: pgmap v6336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11085.879662 secs 2024-07-26T11:45:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:42 smithi044 ceph-mon[32404]: pgmap v6337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11085.879662 secs 2024-07-26T11:45:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:42 smithi116 ceph-mon[37097]: pgmap v6337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11085.879662 secs 2024-07-26T11:45:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:42 smithi171 ceph-mon[37789]: pgmap v6337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:44.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:44.281+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:45:44.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:44.282+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:45:44.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:44.282+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:44 smithi044 ceph-mon[32404]: pgmap v6338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:44 smithi116 ceph-mon[37097]: pgmap v6338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:44 smithi171 ceph-mon[37789]: pgmap v6338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:46 smithi044 ceph-mon[32404]: pgmap v6339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:46 smithi116 ceph-mon[37097]: pgmap v6339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:46 smithi171 ceph-mon[37789]: pgmap v6339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11090.879773 secs 2024-07-26T11:45:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11090.879773 secs 2024-07-26T11:45:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11090.879773 secs 2024-07-26T11:45:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:48 smithi116 ceph-mon[37097]: pgmap v6340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:48 smithi171 ceph-mon[37789]: pgmap v6340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:48 smithi044 ceph-mon[32404]: pgmap v6340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:50 smithi171 ceph-mon[37789]: pgmap v6341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:50 smithi044 ceph-mon[32404]: pgmap v6341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:50 smithi116 ceph-mon[37097]: pgmap v6341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11095.879933 secs 2024-07-26T11:45:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:52 smithi171 ceph-mon[37789]: pgmap v6342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11095.879933 secs 2024-07-26T11:45:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:52 smithi044 ceph-mon[32404]: pgmap v6342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11095.879933 secs 2024-07-26T11:45:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:52 smithi116 ceph-mon[37097]: pgmap v6342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:54 smithi171 ceph-mon[37789]: pgmap v6343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:54 smithi044 ceph-mon[32404]: pgmap v6343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:54 smithi116 ceph-mon[37097]: pgmap v6343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:56 smithi171 ceph-mon[37789]: pgmap v6344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:56 smithi044 ceph-mon[32404]: pgmap v6344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:56 smithi116 ceph-mon[37097]: pgmap v6344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11100.880058 secs 2024-07-26T11:45:57.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11100.880058 secs 2024-07-26T11:45:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11100.880058 secs 2024-07-26T11:45:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:45:58 smithi171 ceph-mon[37789]: pgmap v6345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:45:58 smithi044 ceph-mon[32404]: pgmap v6345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:45:58 smithi116 ceph-mon[37097]: pgmap v6345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:45:59.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:59.288+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:45:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:59.288+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:59.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:45:59.288+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:46:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:00 smithi171 ceph-mon[37789]: pgmap v6346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:00 smithi044 ceph-mon[32404]: pgmap v6346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:00 smithi116 ceph-mon[37097]: pgmap v6346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11105.880213 secs 2024-07-26T11:46:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:02 smithi171 ceph-mon[37789]: pgmap v6347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11105.880213 secs 2024-07-26T11:46:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:02 smithi044 ceph-mon[32404]: pgmap v6347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11105.880213 secs 2024-07-26T11:46:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:02 smithi116 ceph-mon[37097]: pgmap v6347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:04 smithi171 ceph-mon[37789]: pgmap v6348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:04 smithi044 ceph-mon[32404]: pgmap v6348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:04 smithi116 ceph-mon[37097]: pgmap v6348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:06 smithi171 ceph-mon[37789]: pgmap v6349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:06 smithi044 ceph-mon[32404]: pgmap v6349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:06 smithi116 ceph-mon[37097]: pgmap v6349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11110.880336 secs 2024-07-26T11:46:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11110.880336 secs 2024-07-26T11:46:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11110.880336 secs 2024-07-26T11:46:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:08 smithi044 ceph-mon[32404]: pgmap v6350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:08 smithi116 ceph-mon[37097]: pgmap v6350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:08 smithi171 ceph-mon[37789]: pgmap v6350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:10 smithi044 ceph-mon[32404]: pgmap v6351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:10 smithi116 ceph-mon[37097]: pgmap v6351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:10 smithi171 ceph-mon[37789]: pgmap v6351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:46:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:11 smithi044 ceph-mon[32404]: pgmap v6352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:46:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:11 smithi044 ceph-mon[32404]: pgmap v6353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:11 smithi116 ceph-mon[37097]: pgmap v6352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:46:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:11 smithi116 ceph-mon[37097]: pgmap v6353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:11 smithi171 ceph-mon[37789]: pgmap v6352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:46:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:11 smithi171 ceph-mon[37789]: pgmap v6353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11115.880471 secs 2024-07-26T11:46:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11115.880471 secs 2024-07-26T11:46:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11115.880471 secs 2024-07-26T11:46:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:13 smithi044 ceph-mon[32404]: pgmap v6354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:13 smithi116 ceph-mon[37097]: pgmap v6354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:13 smithi171 ceph-mon[37789]: pgmap v6354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:14.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:14.290+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:46:14.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:14.290+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:14.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:14.290+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:46:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:15 smithi044 ceph-mon[32404]: pgmap v6355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:15 smithi116 ceph-mon[37097]: pgmap v6355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:15 smithi171 ceph-mon[37789]: pgmap v6355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11120.880678 secs 2024-07-26T11:46:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:17 smithi044 ceph-mon[32404]: pgmap v6356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:46:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11120.880678 secs 2024-07-26T11:46:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:17 smithi116 ceph-mon[37097]: pgmap v6356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:46:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11120.880678 secs 2024-07-26T11:46:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:17 smithi171 ceph-mon[37789]: pgmap v6356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:46:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:19 smithi044 ceph-mon[32404]: pgmap v6357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:19 smithi116 ceph-mon[37097]: pgmap v6357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:19 smithi171 ceph-mon[37789]: pgmap v6357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:21 smithi044 ceph-mon[32404]: pgmap v6358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:46:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:21 smithi116 ceph-mon[37097]: pgmap v6358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:46:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:21 smithi171 ceph-mon[37789]: pgmap v6358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:46:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11125.880774 secs 2024-07-26T11:46:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11125.880774 secs 2024-07-26T11:46:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11125.880774 secs 2024-07-26T11:46:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:23 smithi044 ceph-mon[32404]: pgmap v6359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:46:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:23 smithi116 ceph-mon[37097]: pgmap v6359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:46:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:23 smithi171 ceph-mon[37789]: pgmap v6359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:46:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:25 smithi044 ceph-mon[32404]: pgmap v6360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:46:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:25 smithi116 ceph-mon[37097]: pgmap v6360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:46:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:25 smithi171 ceph-mon[37789]: pgmap v6360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:46:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11130.880945 secs 2024-07-26T11:46:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:27 smithi044 ceph-mon[32404]: pgmap v6361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11130.880945 secs 2024-07-26T11:46:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:27 smithi116 ceph-mon[37097]: pgmap v6361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11130.880945 secs 2024-07-26T11:46:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:27 smithi171 ceph-mon[37789]: pgmap v6361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:29.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:29.300+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:46:29.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:29.300+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:46:29.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:29.300+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:29 smithi044 ceph-mon[32404]: pgmap v6362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:46:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:29 smithi116 ceph-mon[37097]: pgmap v6362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:46:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:29 smithi171 ceph-mon[37789]: pgmap v6362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:46:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:31 smithi044 ceph-mon[32404]: pgmap v6363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:31 smithi116 ceph-mon[37097]: pgmap v6363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:31 smithi171 ceph-mon[37789]: pgmap v6363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11135.881091 secs 2024-07-26T11:46:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11135.881091 secs 2024-07-26T11:46:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11135.881091 secs 2024-07-26T11:46:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:33 smithi044 ceph-mon[32404]: pgmap v6364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:33 smithi116 ceph-mon[37097]: pgmap v6364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:33 smithi171 ceph-mon[37789]: pgmap v6364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:35 smithi044 ceph-mon[32404]: pgmap v6365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:35 smithi116 ceph-mon[37097]: pgmap v6365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:35 smithi171 ceph-mon[37789]: pgmap v6365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11140.881250 secs 2024-07-26T11:46:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:37 smithi044 ceph-mon[32404]: pgmap v6366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11140.881250 secs 2024-07-26T11:46:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:37 smithi116 ceph-mon[37097]: pgmap v6366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:37.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11140.881250 secs 2024-07-26T11:46:37.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:37 smithi171 ceph-mon[37789]: pgmap v6366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:39 smithi044 ceph-mon[32404]: pgmap v6367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:39 smithi116 ceph-mon[37097]: pgmap v6367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:39 smithi171 ceph-mon[37789]: pgmap v6367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:41 smithi044 ceph-mon[32404]: pgmap v6368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11145.881375 secs 2024-07-26T11:46:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:41 smithi116 ceph-mon[37097]: pgmap v6368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11145.881375 secs 2024-07-26T11:46:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:41 smithi171 ceph-mon[37789]: pgmap v6368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11145.881375 secs 2024-07-26T11:46:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:43 smithi044 ceph-mon[32404]: pgmap v6369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:43.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:43 smithi116 ceph-mon[37097]: pgmap v6369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:43 smithi171 ceph-mon[37789]: pgmap v6369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:44.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:44.305+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:46:44.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:44.305+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:46:44.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:44.305+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:45 smithi044 ceph-mon[32404]: pgmap v6370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:45 smithi116 ceph-mon[37097]: pgmap v6370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:45 smithi171 ceph-mon[37789]: pgmap v6370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:46.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11150.881544 secs 2024-07-26T11:46:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11150.881544 secs 2024-07-26T11:46:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11150.881544 secs 2024-07-26T11:46:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:47 smithi116 ceph-mon[37097]: pgmap v6371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:47 smithi171 ceph-mon[37789]: pgmap v6371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:47 smithi044 ceph-mon[32404]: pgmap v6371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:49 smithi171 ceph-mon[37789]: pgmap v6372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:49 smithi044 ceph-mon[32404]: pgmap v6372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:49 smithi116 ceph-mon[37097]: pgmap v6372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:51 smithi171 ceph-mon[37789]: pgmap v6373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11155.881679 secs 2024-07-26T11:46:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:51 smithi044 ceph-mon[32404]: pgmap v6373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11155.881679 secs 2024-07-26T11:46:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:51 smithi116 ceph-mon[37097]: pgmap v6373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11155.881679 secs 2024-07-26T11:46:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:53 smithi171 ceph-mon[37789]: pgmap v6374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:53 smithi044 ceph-mon[32404]: pgmap v6374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:53 smithi116 ceph-mon[37097]: pgmap v6374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:46:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:55 smithi171 ceph-mon[37789]: pgmap v6375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:55 smithi044 ceph-mon[32404]: pgmap v6375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:55 smithi116 ceph-mon[37097]: pgmap v6375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:46:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11160.881849 secs 2024-07-26T11:46:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11160.881849 secs 2024-07-26T11:46:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11160.881849 secs 2024-07-26T11:46:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:57 smithi171 ceph-mon[37789]: pgmap v6376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:57 smithi044 ceph-mon[32404]: pgmap v6376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:57 smithi116 ceph-mon[37097]: pgmap v6376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:59.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:59.315+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:46:59.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:59.315+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:46:59.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:46:59.315+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:46:59 smithi171 ceph-mon[37789]: pgmap v6377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:46:59 smithi044 ceph-mon[32404]: pgmap v6377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:46:59 smithi116 ceph-mon[37097]: pgmap v6377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:01 smithi171 ceph-mon[37789]: pgmap v6378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11165.881996 secs 2024-07-26T11:47:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:01 smithi044 ceph-mon[32404]: pgmap v6378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11165.881996 secs 2024-07-26T11:47:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:01 smithi116 ceph-mon[37097]: pgmap v6378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11165.881996 secs 2024-07-26T11:47:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:03 smithi044 ceph-mon[32404]: pgmap v6379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:03 smithi116 ceph-mon[37097]: pgmap v6379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:03 smithi171 ceph-mon[37789]: pgmap v6379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:05 smithi044 ceph-mon[32404]: pgmap v6380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:05 smithi116 ceph-mon[37097]: pgmap v6380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:05 smithi171 ceph-mon[37789]: pgmap v6380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11170.882138 secs 2024-07-26T11:47:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11170.882138 secs 2024-07-26T11:47:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11170.882138 secs 2024-07-26T11:47:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:07 smithi044 ceph-mon[32404]: pgmap v6381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:07 smithi116 ceph-mon[37097]: pgmap v6381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:07 smithi171 ceph-mon[37789]: pgmap v6381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:09 smithi044 ceph-mon[32404]: pgmap v6382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:09 smithi116 ceph-mon[37097]: pgmap v6382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:09 smithi171 ceph-mon[37789]: pgmap v6382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:11 smithi044 ceph-mon[32404]: pgmap v6383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11175.882295 secs 2024-07-26T11:47:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:11 smithi116 ceph-mon[37097]: pgmap v6383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11175.882295 secs 2024-07-26T11:47:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:11 smithi171 ceph-mon[37789]: pgmap v6383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11175.882295 secs 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: pgmap v6384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:13.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: pgmap v6385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:13.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:47:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:47:13.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:47:13.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:13.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:13.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: pgmap v6384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:13.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: pgmap v6385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:13.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:47:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: pgmap v6384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: pgmap v6385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:13.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:47:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:14.320+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:47:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:14.320+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:47:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:14.320+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:14.978 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:14 smithi044 ceph-mon[32404]: pgmap v6386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:14 smithi116 ceph-mon[37097]: pgmap v6386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:14 smithi171 ceph-mon[37789]: pgmap v6386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:16.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:16.681+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:47:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:16.682+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:47:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:16.683+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:47:16.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:16.684+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:47:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11180.882443 secs 2024-07-26T11:47:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:16 smithi044 ceph-mon[32404]: pgmap v6387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:47:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11180.882443 secs 2024-07-26T11:47:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:16 smithi116 ceph-mon[37097]: pgmap v6387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:47:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11180.882443 secs 2024-07-26T11:47:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:16 smithi171 ceph-mon[37789]: pgmap v6387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:47:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:18 smithi044 ceph-mon[32404]: pgmap v6388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:18 smithi116 ceph-mon[37097]: pgmap v6388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:18 smithi171 ceph-mon[37789]: pgmap v6388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:20 smithi044 ceph-mon[32404]: pgmap v6389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:20 smithi116 ceph-mon[37097]: pgmap v6389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:20 smithi171 ceph-mon[37789]: pgmap v6389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:47:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11185.882607 secs 2024-07-26T11:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11185.882607 secs 2024-07-26T11:47:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11185.882607 secs 2024-07-26T11:47:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:22 smithi044 ceph-mon[32404]: pgmap v6390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:22 smithi116 ceph-mon[37097]: pgmap v6390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:22 smithi171 ceph-mon[37789]: pgmap v6390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:24 smithi044 ceph-mon[32404]: pgmap v6391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:24 smithi116 ceph-mon[37097]: pgmap v6391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:24 smithi171 ceph-mon[37789]: pgmap v6391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11190.882772 secs 2024-07-26T11:47:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:26 smithi044 ceph-mon[32404]: pgmap v6392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11190.882772 secs 2024-07-26T11:47:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:26 smithi116 ceph-mon[37097]: pgmap v6392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11190.882772 secs 2024-07-26T11:47:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:26 smithi171 ceph-mon[37789]: pgmap v6392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:28 smithi044 ceph-mon[32404]: pgmap v6393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:28 smithi116 ceph-mon[37097]: pgmap v6393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:28 smithi171 ceph-mon[37789]: pgmap v6393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:29.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:29.325+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:47:29.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:29.325+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:47:29.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:29.325+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:30 smithi116 ceph-mon[37097]: pgmap v6394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:30 smithi171 ceph-mon[37789]: pgmap v6394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:30 smithi044 ceph-mon[32404]: pgmap v6394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11195.882919 secs 2024-07-26T11:47:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11195.882919 secs 2024-07-26T11:47:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11195.882919 secs 2024-07-26T11:47:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:32 smithi171 ceph-mon[37789]: pgmap v6395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:32 smithi044 ceph-mon[32404]: pgmap v6395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:32 smithi116 ceph-mon[37097]: pgmap v6395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:34 smithi171 ceph-mon[37789]: pgmap v6396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:34 smithi044 ceph-mon[32404]: pgmap v6396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:34 smithi116 ceph-mon[37097]: pgmap v6396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11200.883089 secs 2024-07-26T11:47:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:36 smithi171 ceph-mon[37789]: pgmap v6397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11200.883089 secs 2024-07-26T11:47:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:36 smithi044 ceph-mon[32404]: pgmap v6397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11200.883089 secs 2024-07-26T11:47:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:36 smithi116 ceph-mon[37097]: pgmap v6397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:38 smithi171 ceph-mon[37789]: pgmap v6398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:38 smithi044 ceph-mon[32404]: pgmap v6398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:38 smithi116 ceph-mon[37097]: pgmap v6398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:40 smithi171 ceph-mon[37789]: pgmap v6399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:40 smithi044 ceph-mon[32404]: pgmap v6399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:40 smithi116 ceph-mon[37097]: pgmap v6399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11205.883239 secs 2024-07-26T11:47:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11205.883239 secs 2024-07-26T11:47:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11205.883239 secs 2024-07-26T11:47:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:42 smithi171 ceph-mon[37789]: pgmap v6400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:42 smithi044 ceph-mon[32404]: pgmap v6400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:42 smithi116 ceph-mon[37097]: pgmap v6400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:44.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:44.330+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:47:44.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:44.330+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:47:44.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:44.330+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:44 smithi171 ceph-mon[37789]: pgmap v6401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:44 smithi044 ceph-mon[32404]: pgmap v6401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:44 smithi116 ceph-mon[37097]: pgmap v6401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11210.883404 secs 2024-07-26T11:47:47.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:46 smithi044 ceph-mon[32404]: pgmap v6402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11210.883404 secs 2024-07-26T11:47:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:46 smithi171 ceph-mon[37789]: pgmap v6402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11210.883404 secs 2024-07-26T11:47:47.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:46 smithi116 ceph-mon[37097]: pgmap v6402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:48 smithi044 ceph-mon[32404]: pgmap v6403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:48 smithi116 ceph-mon[37097]: pgmap v6403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:48 smithi171 ceph-mon[37789]: pgmap v6403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:50 smithi044 ceph-mon[32404]: pgmap v6404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:50 smithi116 ceph-mon[37097]: pgmap v6404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:50 smithi171 ceph-mon[37789]: pgmap v6404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11215.883569 secs 2024-07-26T11:47:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11215.883569 secs 2024-07-26T11:47:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11215.883569 secs 2024-07-26T11:47:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:52 smithi044 ceph-mon[32404]: pgmap v6405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:52 smithi116 ceph-mon[37097]: pgmap v6405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:52 smithi171 ceph-mon[37789]: pgmap v6405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:54 smithi044 ceph-mon[32404]: pgmap v6406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:54 smithi116 ceph-mon[37097]: pgmap v6406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:54 smithi171 ceph-mon[37789]: pgmap v6406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:47:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11220.883732 secs 2024-07-26T11:47:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:56 smithi044 ceph-mon[32404]: pgmap v6407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11220.883732 secs 2024-07-26T11:47:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:56 smithi116 ceph-mon[37097]: pgmap v6407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11220.883732 secs 2024-07-26T11:47:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:56 smithi171 ceph-mon[37789]: pgmap v6407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:59.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:47:58 smithi171 ceph-mon[37789]: pgmap v6408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:47:58 smithi044 ceph-mon[32404]: pgmap v6408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:47:58 smithi116 ceph-mon[37097]: pgmap v6408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:47:59.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:59.331+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:47:59.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:59.331+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:47:59.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:47:59.331+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:00 smithi044 ceph-mon[32404]: pgmap v6409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:00 smithi116 ceph-mon[37097]: pgmap v6409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:00 smithi171 ceph-mon[37789]: pgmap v6409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11225.883899 secs 2024-07-26T11:48:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11225.883899 secs 2024-07-26T11:48:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11225.883899 secs 2024-07-26T11:48:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:02 smithi044 ceph-mon[32404]: pgmap v6410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:02 smithi116 ceph-mon[37097]: pgmap v6410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:02 smithi171 ceph-mon[37789]: pgmap v6410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:04.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:04.612+0000 7f76cb400640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f76ac005850 con 0x7f76d410f440 2024-07-26T11:48:04.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:04.613+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f76ac005850 con 0x7f76d410f440 2024-07-26T11:48:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:04 smithi044 ceph-mon[32404]: pgmap v6411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:04 smithi116 ceph-mon[37097]: pgmap v6411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:04 smithi171 ceph-mon[37789]: pgmap v6411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11230.883997 secs 2024-07-26T11:48:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:06 smithi044 ceph-mon[32404]: pgmap v6412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11230.883997 secs 2024-07-26T11:48:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:06 smithi116 ceph-mon[37097]: pgmap v6412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11230.883997 secs 2024-07-26T11:48:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:06 smithi171 ceph-mon[37789]: pgmap v6412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:08 smithi044 ceph-mon[32404]: pgmap v6413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:08 smithi116 ceph-mon[37097]: pgmap v6413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:08 smithi171 ceph-mon[37789]: pgmap v6413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:10 smithi044 ceph-mon[32404]: pgmap v6414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:10 smithi116 ceph-mon[37097]: pgmap v6414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:10 smithi171 ceph-mon[37789]: pgmap v6414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11235.884149 secs 2024-07-26T11:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11235.884149 secs 2024-07-26T11:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11235.884149 secs 2024-07-26T11:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: pgmap v6415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: pgmap v6415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: pgmap v6415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:13.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:48:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:48:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:48:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:48:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:48:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: pgmap v6416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:48:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: pgmap v6417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:48:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:48:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: pgmap v6416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: pgmap v6417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:48:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: pgmap v6416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:48:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: pgmap v6417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:14.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:48:14.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:14.336+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:48:14.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:14.336+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:48:14.339 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:14.336+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:16 smithi044 ceph-mon[32404]: pgmap v6418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:16 smithi116 ceph-mon[37097]: pgmap v6418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:16 smithi171 ceph-mon[37789]: pgmap v6418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11240.884266 secs 2024-07-26T11:48:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11240.884266 secs 2024-07-26T11:48:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11240.884266 secs 2024-07-26T11:48:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:18 smithi044 ceph-mon[32404]: pgmap v6419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:18 smithi171 ceph-mon[37789]: pgmap v6419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:18 smithi116 ceph-mon[37097]: pgmap v6419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:20 smithi171 ceph-mon[37789]: pgmap v6420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:20 smithi044 ceph-mon[32404]: pgmap v6420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:20 smithi116 ceph-mon[37097]: pgmap v6420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:48:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11245.884404 secs 2024-07-26T11:48:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:22 smithi171 ceph-mon[37789]: pgmap v6421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11245.884404 secs 2024-07-26T11:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:22 smithi044 ceph-mon[32404]: pgmap v6421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11245.884404 secs 2024-07-26T11:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:22 smithi116 ceph-mon[37097]: pgmap v6421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:48:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:24 smithi171 ceph-mon[37789]: pgmap v6422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:24 smithi044 ceph-mon[32404]: pgmap v6422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:24 smithi116 ceph-mon[37097]: pgmap v6422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:26 smithi171 ceph-mon[37789]: pgmap v6423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:26 smithi044 ceph-mon[32404]: pgmap v6423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:26 smithi116 ceph-mon[37097]: pgmap v6423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11250.884495 secs 2024-07-26T11:48:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11250.884495 secs 2024-07-26T11:48:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11250.884495 secs 2024-07-26T11:48:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:28 smithi171 ceph-mon[37789]: pgmap v6424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:28 smithi044 ceph-mon[32404]: pgmap v6424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:28 smithi116 ceph-mon[37097]: pgmap v6424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:29.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:29.346+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:48:29.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:29.346+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:48:29.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:29.346+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:30 smithi171 ceph-mon[37789]: pgmap v6425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:30 smithi044 ceph-mon[32404]: pgmap v6425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:30 smithi116 ceph-mon[37097]: pgmap v6425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11255.884676 secs 2024-07-26T11:48:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:32 smithi171 ceph-mon[37789]: pgmap v6426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11255.884676 secs 2024-07-26T11:48:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:32 smithi044 ceph-mon[32404]: pgmap v6426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11255.884676 secs 2024-07-26T11:48:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:32 smithi116 ceph-mon[37097]: pgmap v6426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:34 smithi171 ceph-mon[37789]: pgmap v6427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:34 smithi044 ceph-mon[32404]: pgmap v6427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:34 smithi116 ceph-mon[37097]: pgmap v6427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:35 smithi044 ceph-mon[32404]: pgmap v6428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:35 smithi116 ceph-mon[37097]: pgmap v6428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:35 smithi171 ceph-mon[37789]: pgmap v6428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11260.884843 secs 2024-07-26T11:48:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11260.884843 secs 2024-07-26T11:48:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11260.884843 secs 2024-07-26T11:48:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:37 smithi171 ceph-mon[37789]: pgmap v6429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:37 smithi044 ceph-mon[32404]: pgmap v6429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:37 smithi116 ceph-mon[37097]: pgmap v6429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:40 smithi171 ceph-mon[37789]: pgmap v6430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:40 smithi044 ceph-mon[32404]: pgmap v6430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:40 smithi116 ceph-mon[37097]: pgmap v6430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11265.885002 secs 2024-07-26T11:48:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:42 smithi171 ceph-mon[37789]: pgmap v6431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11265.885002 secs 2024-07-26T11:48:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:42 smithi044 ceph-mon[32404]: pgmap v6431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11265.885002 secs 2024-07-26T11:48:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:42 smithi116 ceph-mon[37097]: pgmap v6431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:44 smithi171 ceph-mon[37789]: pgmap v6432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:44.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:44.351+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:48:44.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:44.351+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:48:44.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:44.351+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:44.354 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:44 smithi044 ceph-mon[32404]: pgmap v6432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:44 smithi116 ceph-mon[37097]: pgmap v6432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:46 smithi171 ceph-mon[37789]: pgmap v6433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:46 smithi044 ceph-mon[32404]: pgmap v6433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:46 smithi116 ceph-mon[37097]: pgmap v6433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:48:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11270.885160 secs 2024-07-26T11:48:47.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11270.885160 secs 2024-07-26T11:48:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11270.885160 secs 2024-07-26T11:48:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:48 smithi171 ceph-mon[37789]: pgmap v6434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:48 smithi044 ceph-mon[32404]: pgmap v6434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:48 smithi116 ceph-mon[37097]: pgmap v6434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:50 smithi171 ceph-mon[37789]: pgmap v6435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:50 smithi044 ceph-mon[32404]: pgmap v6435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:50 smithi116 ceph-mon[37097]: pgmap v6435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11275.885308 secs 2024-07-26T11:48:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:52 smithi171 ceph-mon[37789]: pgmap v6436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11275.885308 secs 2024-07-26T11:48:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:52 smithi044 ceph-mon[32404]: pgmap v6436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11275.885308 secs 2024-07-26T11:48:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:52 smithi116 ceph-mon[37097]: pgmap v6436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:54 smithi171 ceph-mon[37789]: pgmap v6437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:54 smithi044 ceph-mon[32404]: pgmap v6437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:54 smithi116 ceph-mon[37097]: pgmap v6437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:56 smithi171 ceph-mon[37789]: pgmap v6438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:56 smithi044 ceph-mon[32404]: pgmap v6438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:56 smithi116 ceph-mon[37097]: pgmap v6438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11280.885471 secs 2024-07-26T11:48:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11280.885471 secs 2024-07-26T11:48:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11280.885471 secs 2024-07-26T11:48:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:58 smithi044 ceph-mon[32404]: pgmap v6439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:58 smithi116 ceph-mon[37097]: pgmap v6439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:58 smithi171 ceph-mon[37789]: pgmap v6439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:59.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:59.361+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:48:59.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:59.361+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:48:59.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:48:59.361+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:59.364 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:48:59 smithi044 ceph-mon[32404]: pgmap v6440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:48:59 smithi116 ceph-mon[37097]: pgmap v6440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:48:59 smithi171 ceph-mon[37789]: pgmap v6440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11285.885608 secs 2024-07-26T11:49:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:02 smithi171 ceph-mon[37789]: pgmap v6441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11285.885608 secs 2024-07-26T11:49:02.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:02 smithi044 ceph-mon[32404]: pgmap v6441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11285.885608 secs 2024-07-26T11:49:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:02 smithi116 ceph-mon[37097]: pgmap v6441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:04 smithi171 ceph-mon[37789]: pgmap v6442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:04 smithi044 ceph-mon[32404]: pgmap v6442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:04 smithi116 ceph-mon[37097]: pgmap v6442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:06 smithi171 ceph-mon[37789]: pgmap v6443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:06 smithi044 ceph-mon[32404]: pgmap v6443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:06 smithi116 ceph-mon[37097]: pgmap v6443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11290.885747 secs 2024-07-26T11:49:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11290.885747 secs 2024-07-26T11:49:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11290.885747 secs 2024-07-26T11:49:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:08 smithi171 ceph-mon[37789]: pgmap v6444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:08 smithi044 ceph-mon[32404]: pgmap v6444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:08 smithi116 ceph-mon[37097]: pgmap v6444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:10 smithi171 ceph-mon[37789]: pgmap v6445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:10 smithi044 ceph-mon[32404]: pgmap v6445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:10 smithi116 ceph-mon[37097]: pgmap v6445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11295.885906 secs 2024-07-26T11:49:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:12 smithi171 ceph-mon[37789]: pgmap v6446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11295.885906 secs 2024-07-26T11:49:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:12 smithi044 ceph-mon[32404]: pgmap v6446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11295.885906 secs 2024-07-26T11:49:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:12 smithi116 ceph-mon[37097]: pgmap v6446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:14 smithi171 ceph-mon[37789]: pgmap v6447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:14.366+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:49:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:14.366+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:49:14.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:14.366+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:14.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:14 smithi044 ceph-mon[32404]: pgmap v6447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:14.369 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:14 smithi116 ceph-mon[37097]: pgmap v6447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:49:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:49:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:49:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:16 smithi044 ceph-mon[32404]: pgmap v6448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:49:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:16 smithi044 ceph-mon[32404]: pgmap v6449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:16 smithi116 ceph-mon[37097]: pgmap v6448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:49:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:16 smithi116 ceph-mon[37097]: pgmap v6449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:16 smithi171 ceph-mon[37789]: pgmap v6448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:49:16.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:16 smithi171 ceph-mon[37789]: pgmap v6449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:17.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11300.886067 secs 2024-07-26T11:49:17.424 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:17 smithi116 ceph-mon[37097]: pgmap v6450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11300.886067 secs 2024-07-26T11:49:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:17 smithi044 ceph-mon[32404]: pgmap v6450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11300.886067 secs 2024-07-26T11:49:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:17 smithi171 ceph-mon[37789]: pgmap v6450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:19 smithi171 ceph-mon[37789]: pgmap v6451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:19 smithi044 ceph-mon[32404]: pgmap v6451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:19 smithi116 ceph-mon[37097]: pgmap v6451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:21 smithi171 ceph-mon[37789]: pgmap v6452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:21 smithi044 ceph-mon[32404]: pgmap v6452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:21 smithi116 ceph-mon[37097]: pgmap v6452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:49:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11305.886227 secs 2024-07-26T11:49:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11305.886227 secs 2024-07-26T11:49:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11305.886227 secs 2024-07-26T11:49:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:23 smithi044 ceph-mon[32404]: pgmap v6453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:23 smithi116 ceph-mon[37097]: pgmap v6453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:23 smithi171 ceph-mon[37789]: pgmap v6453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:49:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:25 smithi044 ceph-mon[32404]: pgmap v6454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:25 smithi116 ceph-mon[37097]: pgmap v6454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:25 smithi171 ceph-mon[37789]: pgmap v6454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11310.886374 secs 2024-07-26T11:49:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:27 smithi044 ceph-mon[32404]: pgmap v6455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11310.886374 secs 2024-07-26T11:49:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:27 smithi116 ceph-mon[37097]: pgmap v6455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11310.886374 secs 2024-07-26T11:49:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:27 smithi171 ceph-mon[37789]: pgmap v6455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:29.376+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:49:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:29.377+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:49:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:29.377+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:29 smithi044 ceph-mon[32404]: pgmap v6456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:29 smithi116 ceph-mon[37097]: pgmap v6456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:29 smithi171 ceph-mon[37789]: pgmap v6456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:31 smithi044 ceph-mon[32404]: pgmap v6457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:31 smithi116 ceph-mon[37097]: pgmap v6457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:31 smithi171 ceph-mon[37789]: pgmap v6457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:32.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11315.886510 secs 2024-07-26T11:49:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11315.886510 secs 2024-07-26T11:49:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11315.886510 secs 2024-07-26T11:49:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:33 smithi044 ceph-mon[32404]: pgmap v6458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:33 smithi116 ceph-mon[37097]: pgmap v6458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:33 smithi171 ceph-mon[37789]: pgmap v6458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:35 smithi044 ceph-mon[32404]: pgmap v6459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:35 smithi116 ceph-mon[37097]: pgmap v6459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:35 smithi171 ceph-mon[37789]: pgmap v6459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:37.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11320.886691 secs 2024-07-26T11:49:37.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:37 smithi044 ceph-mon[32404]: pgmap v6460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11320.886691 secs 2024-07-26T11:49:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:37 smithi116 ceph-mon[37097]: pgmap v6460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11320.886691 secs 2024-07-26T11:49:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:37 smithi171 ceph-mon[37789]: pgmap v6460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:39 smithi044 ceph-mon[32404]: pgmap v6461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:39 smithi116 ceph-mon[37097]: pgmap v6461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:39 smithi171 ceph-mon[37789]: pgmap v6461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:41 smithi044 ceph-mon[32404]: pgmap v6462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:41 smithi116 ceph-mon[37097]: pgmap v6462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:41 smithi171 ceph-mon[37789]: pgmap v6462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11325.886836 secs 2024-07-26T11:49:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11325.886836 secs 2024-07-26T11:49:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11325.886836 secs 2024-07-26T11:49:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:43 smithi044 ceph-mon[32404]: pgmap v6463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:43 smithi116 ceph-mon[37097]: pgmap v6463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:43 smithi171 ceph-mon[37789]: pgmap v6463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:44.382+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:49:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:44.382+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:49:44.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:44.382+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:45 smithi044 ceph-mon[32404]: pgmap v6464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:45 smithi116 ceph-mon[37097]: pgmap v6464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:45 smithi171 ceph-mon[37789]: pgmap v6464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11330.886976 secs 2024-07-26T11:49:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:47 smithi044 ceph-mon[32404]: pgmap v6465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11330.886976 secs 2024-07-26T11:49:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:47 smithi116 ceph-mon[37097]: pgmap v6465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11330.886976 secs 2024-07-26T11:49:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:47 smithi171 ceph-mon[37789]: pgmap v6465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:49 smithi044 ceph-mon[32404]: pgmap v6466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:49 smithi116 ceph-mon[37097]: pgmap v6466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:49 smithi171 ceph-mon[37789]: pgmap v6466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:51 smithi044 ceph-mon[32404]: pgmap v6467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:51 smithi116 ceph-mon[37097]: pgmap v6467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:51 smithi171 ceph-mon[37789]: pgmap v6467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11335.887131 secs 2024-07-26T11:49:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11335.887131 secs 2024-07-26T11:49:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11335.887131 secs 2024-07-26T11:49:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:53 smithi044 ceph-mon[32404]: pgmap v6468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:53 smithi116 ceph-mon[37097]: pgmap v6468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:53 smithi171 ceph-mon[37789]: pgmap v6468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:55 smithi044 ceph-mon[32404]: pgmap v6469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:55 smithi116 ceph-mon[37097]: pgmap v6469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:55 smithi171 ceph-mon[37789]: pgmap v6469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:49:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11340.887290 secs 2024-07-26T11:49:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:57 smithi044 ceph-mon[32404]: pgmap v6470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11340.887290 secs 2024-07-26T11:49:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:57 smithi116 ceph-mon[37097]: pgmap v6470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11340.887290 secs 2024-07-26T11:49:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:57 smithi171 ceph-mon[37789]: pgmap v6470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:59.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:59.391+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:49:59.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:59.391+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:49:59.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:49:59.391+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:49:59 smithi044 ceph-mon[32404]: pgmap v6471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:49:59 smithi116 ceph-mon[37097]: pgmap v6471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:49:59 smithi171 ceph-mon[37789]: pgmap v6471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T11:50:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:01 smithi171 ceph-mon[37789]: pgmap v6472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:01 smithi044 ceph-mon[32404]: pgmap v6472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:01 smithi116 ceph-mon[37097]: pgmap v6472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11345.887432 secs 2024-07-26T11:50:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11345.887432 secs 2024-07-26T11:50:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11345.887432 secs 2024-07-26T11:50:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:03 smithi171 ceph-mon[37789]: pgmap v6473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:03 smithi044 ceph-mon[32404]: pgmap v6473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:03 smithi116 ceph-mon[37097]: pgmap v6473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:50:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:05 smithi171 ceph-mon[37789]: pgmap v6474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:05.986 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:05 smithi044 ceph-mon[32404]: pgmap v6474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:05 smithi116 ceph-mon[37097]: pgmap v6474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11350.887581 secs 2024-07-26T11:50:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11350.887581 secs 2024-07-26T11:50:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11350.887581 secs 2024-07-26T11:50:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:07 smithi171 ceph-mon[37789]: pgmap v6475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:07 smithi044 ceph-mon[32404]: pgmap v6475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:07 smithi116 ceph-mon[37097]: pgmap v6475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:09 smithi171 ceph-mon[37789]: pgmap v6476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:10.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:09 smithi044 ceph-mon[32404]: pgmap v6476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:09 smithi116 ceph-mon[37097]: pgmap v6476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:11 smithi171 ceph-mon[37789]: pgmap v6477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11355.887748 secs 2024-07-26T11:50:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:11 smithi044 ceph-mon[32404]: pgmap v6477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11355.887748 secs 2024-07-26T11:50:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:11 smithi116 ceph-mon[37097]: pgmap v6477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11355.887748 secs 2024-07-26T11:50:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:13 smithi171 ceph-mon[37789]: pgmap v6478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:50:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:13 smithi044 ceph-mon[32404]: pgmap v6478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:13 smithi116 ceph-mon[37097]: pgmap v6478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:50:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:14.397+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:50:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:14.397+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:14.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:14.397+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:50:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:15 smithi044 ceph-mon[32404]: pgmap v6479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:15 smithi116 ceph-mon[37097]: pgmap v6479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:15 smithi171 ceph-mon[37789]: pgmap v6479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:16.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:16.713+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:50:16.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:16.713+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:50:16.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:16.714+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:50:16.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:16.714+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:50:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11360.887829 secs 2024-07-26T11:50:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11360.887829 secs 2024-07-26T11:50:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:17.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11360.887829 secs 2024-07-26T11:50:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:50:17.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:17 smithi116 ceph-mon[37097]: pgmap v6480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:17.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:17 smithi116 ceph-mon[37097]: pgmap v6481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:17 smithi044 ceph-mon[32404]: pgmap v6480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:17 smithi044 ceph-mon[32404]: pgmap v6481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:17 smithi171 ceph-mon[37789]: pgmap v6480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:17 smithi171 ceph-mon[37789]: pgmap v6481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:19.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:19.637+0000 7f76cb400640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f76ac005b90 con 0x7f76d410f440 2024-07-26T11:50:19.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:19.638+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f76ac005b90 con 0x7f76d410f440 2024-07-26T11:50:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:19 smithi044 ceph-mon[32404]: pgmap v6482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:19 smithi116 ceph-mon[37097]: pgmap v6482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:19 smithi171 ceph-mon[37789]: pgmap v6482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:50:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:21 smithi044 ceph-mon[32404]: pgmap v6483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11365.887987 secs 2024-07-26T11:50:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:21 smithi116 ceph-mon[37097]: pgmap v6483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11365.887987 secs 2024-07-26T11:50:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:21 smithi171 ceph-mon[37789]: pgmap v6483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11365.887987 secs 2024-07-26T11:50:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:23 smithi044 ceph-mon[32404]: pgmap v6484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:23 smithi116 ceph-mon[37097]: pgmap v6484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:23 smithi171 ceph-mon[37789]: pgmap v6484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:25 smithi044 ceph-mon[32404]: pgmap v6485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:25 smithi116 ceph-mon[37097]: pgmap v6485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:25 smithi171 ceph-mon[37789]: pgmap v6485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:50:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11370.888133 secs 2024-07-26T11:50:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11370.888133 secs 2024-07-26T11:50:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11370.888133 secs 2024-07-26T11:50:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:27 smithi044 ceph-mon[32404]: pgmap v6486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:27 smithi116 ceph-mon[37097]: pgmap v6486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:27 smithi171 ceph-mon[37789]: pgmap v6486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:29.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:29.408+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:50:29.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:29.408+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:29.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:29.408+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:50:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:29 smithi044 ceph-mon[32404]: pgmap v6487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:29 smithi116 ceph-mon[37097]: pgmap v6487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:29 smithi171 ceph-mon[37789]: pgmap v6487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:31 smithi044 ceph-mon[32404]: pgmap v6488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11375.888287 secs 2024-07-26T11:50:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:31 smithi116 ceph-mon[37097]: pgmap v6488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11375.888287 secs 2024-07-26T11:50:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:31 smithi171 ceph-mon[37789]: pgmap v6488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11375.888287 secs 2024-07-26T11:50:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:33 smithi044 ceph-mon[32404]: pgmap v6489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:33 smithi116 ceph-mon[37097]: pgmap v6489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:33 smithi171 ceph-mon[37789]: pgmap v6489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:35 smithi044 ceph-mon[32404]: pgmap v6490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:35 smithi116 ceph-mon[37097]: pgmap v6490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:35 smithi171 ceph-mon[37789]: pgmap v6490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11380.888451 secs 2024-07-26T11:50:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11380.888451 secs 2024-07-26T11:50:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11380.888451 secs 2024-07-26T11:50:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:37 smithi044 ceph-mon[32404]: pgmap v6491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:37 smithi116 ceph-mon[37097]: pgmap v6491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:37 smithi171 ceph-mon[37789]: pgmap v6491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:39 smithi044 ceph-mon[32404]: pgmap v6492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:39 smithi116 ceph-mon[37097]: pgmap v6492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:39 smithi171 ceph-mon[37789]: pgmap v6492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:41 smithi044 ceph-mon[32404]: pgmap v6493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11385.888612 secs 2024-07-26T11:50:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:41 smithi116 ceph-mon[37097]: pgmap v6493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11385.888612 secs 2024-07-26T11:50:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:41 smithi171 ceph-mon[37789]: pgmap v6493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11385.888612 secs 2024-07-26T11:50:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:43 smithi044 ceph-mon[32404]: pgmap v6494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:43 smithi116 ceph-mon[37097]: pgmap v6494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:43 smithi171 ceph-mon[37789]: pgmap v6494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:44.413+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:50:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:44.413+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:44.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:44.413+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:50:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:45 smithi044 ceph-mon[32404]: pgmap v6495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:45 smithi116 ceph-mon[37097]: pgmap v6495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:45 smithi171 ceph-mon[37789]: pgmap v6495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11390.888753 secs 2024-07-26T11:50:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11390.888753 secs 2024-07-26T11:50:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11390.888753 secs 2024-07-26T11:50:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:47 smithi044 ceph-mon[32404]: pgmap v6496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:47 smithi116 ceph-mon[37097]: pgmap v6496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:47 smithi171 ceph-mon[37789]: pgmap v6496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:49 smithi044 ceph-mon[32404]: pgmap v6497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:49 smithi116 ceph-mon[37097]: pgmap v6497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:49 smithi171 ceph-mon[37789]: pgmap v6497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:51 smithi044 ceph-mon[32404]: pgmap v6498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11395.888849 secs 2024-07-26T11:50:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:51 smithi116 ceph-mon[37097]: pgmap v6498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11395.888849 secs 2024-07-26T11:50:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:51 smithi171 ceph-mon[37789]: pgmap v6498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:50:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11395.888849 secs 2024-07-26T11:50:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:53 smithi044 ceph-mon[32404]: pgmap v6499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:53 smithi116 ceph-mon[37097]: pgmap v6499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:53 smithi171 ceph-mon[37789]: pgmap v6499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:55 smithi044 ceph-mon[32404]: pgmap v6500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:55 smithi116 ceph-mon[37097]: pgmap v6500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:55 smithi171 ceph-mon[37789]: pgmap v6500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:50:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11400.889006 secs 2024-07-26T11:50:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11400.889006 secs 2024-07-26T11:50:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11400.889006 secs 2024-07-26T11:50:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:57 smithi171 ceph-mon[37789]: pgmap v6501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:57 smithi044 ceph-mon[32404]: pgmap v6501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:57 smithi116 ceph-mon[37097]: pgmap v6501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:59.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:59.423+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:50:59.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:59.423+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:59.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:50:59.423+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:51:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:50:59 smithi171 ceph-mon[37789]: pgmap v6502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:50:59 smithi044 ceph-mon[32404]: pgmap v6502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:50:59 smithi116 ceph-mon[37097]: pgmap v6502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:01.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:01.722+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:51:01.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:01.722+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:51:01.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:01.722+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:51:01.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:01.722+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:51:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:01 smithi044 ceph-mon[32404]: pgmap v6503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11405.889161 secs 2024-07-26T11:51:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:01 smithi171 ceph-mon[37789]: pgmap v6503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11405.889161 secs 2024-07-26T11:51:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:01 smithi116 ceph-mon[37097]: pgmap v6503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:02.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11405.889161 secs 2024-07-26T11:51:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:03 smithi171 ceph-mon[37789]: pgmap v6504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:03 smithi044 ceph-mon[32404]: pgmap v6504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:03 smithi116 ceph-mon[37097]: pgmap v6504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:04.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:04.644+0000 7f76cb400640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f76ac005ee0 con 0x7f76d410f440 2024-07-26T11:51:04.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:04.645+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f76ac005ee0 con 0x7f76d410f440 2024-07-26T11:51:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:05 smithi171 ceph-mon[37789]: pgmap v6505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:05 smithi044 ceph-mon[32404]: pgmap v6505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:05 smithi116 ceph-mon[37097]: pgmap v6505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11410.889279 secs 2024-07-26T11:51:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11410.889279 secs 2024-07-26T11:51:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11410.889279 secs 2024-07-26T11:51:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:07 smithi171 ceph-mon[37789]: pgmap v6506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:07 smithi044 ceph-mon[32404]: pgmap v6506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:07 smithi116 ceph-mon[37097]: pgmap v6506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:09 smithi171 ceph-mon[37789]: pgmap v6507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:09 smithi044 ceph-mon[32404]: pgmap v6507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:09 smithi116 ceph-mon[37097]: pgmap v6507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11415.889430 secs 2024-07-26T11:51:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11415.889430 secs 2024-07-26T11:51:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11415.889430 secs 2024-07-26T11:51:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:11 smithi044 ceph-mon[32404]: pgmap v6508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:11 smithi116 ceph-mon[37097]: pgmap v6508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:11 smithi171 ceph-mon[37789]: pgmap v6508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:13 smithi044 ceph-mon[32404]: pgmap v6509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:13 smithi116 ceph-mon[37097]: pgmap v6509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:13 smithi171 ceph-mon[37789]: pgmap v6509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:14.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:14.428+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:51:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:14.428+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:51:14.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:14.428+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:15 smithi044 ceph-mon[32404]: pgmap v6510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11420.889595 secs 2024-07-26T11:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:15 smithi116 ceph-mon[37097]: pgmap v6510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11420.889595 secs 2024-07-26T11:51:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:15 smithi171 ceph-mon[37789]: pgmap v6510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11420.889595 secs 2024-07-26T11:51:17.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:17.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:17 smithi116 ceph-mon[37097]: pgmap v6511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:17 smithi171 ceph-mon[37789]: pgmap v6511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:17 smithi044 ceph-mon[32404]: pgmap v6511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:19 smithi044 ceph-mon[32404]: pgmap v6512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:19 smithi116 ceph-mon[37097]: pgmap v6512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:19 smithi171 ceph-mon[37789]: pgmap v6512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:19.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11425.889709 secs 2024-07-26T11:51:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11425.889709 secs 2024-07-26T11:51:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11425.889709 secs 2024-07-26T11:51:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:22 smithi116 ceph-mon[37097]: pgmap v6513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:22 smithi116 ceph-mon[37097]: pgmap v6514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:22 smithi116 ceph-mon[37097]: pgmap v6515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:51:22.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:22 smithi171 ceph-mon[37789]: pgmap v6513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:22 smithi171 ceph-mon[37789]: pgmap v6514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:22 smithi171 ceph-mon[37789]: pgmap v6515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:51:22.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:22 smithi044 ceph-mon[32404]: pgmap v6513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:22 smithi044 ceph-mon[32404]: pgmap v6514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:22 smithi044 ceph-mon[32404]: pgmap v6515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:51:22.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:51:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:23 smithi044 ceph-mon[32404]: pgmap v6516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:23 smithi116 ceph-mon[37097]: pgmap v6516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:23 smithi171 ceph-mon[37789]: pgmap v6516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:25 smithi044 ceph-mon[32404]: pgmap v6517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:25 smithi116 ceph-mon[37097]: pgmap v6517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:25 smithi171 ceph-mon[37789]: pgmap v6517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11430.889840 secs 2024-07-26T11:51:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11430.889840 secs 2024-07-26T11:51:26.962 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11430.889840 secs 2024-07-26T11:51:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:27 smithi171 ceph-mon[37789]: pgmap v6518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:51:27.876 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:27 smithi116 ceph-mon[37097]: pgmap v6518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:51:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:27 smithi044 ceph-mon[32404]: pgmap v6518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T11:51:29.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:29 smithi044 ceph-mon[32404]: pgmap v6519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:29.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:29.438+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:51:29.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:29.438+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:29.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:29.438+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:51:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:29 smithi116 ceph-mon[37097]: pgmap v6519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:29 smithi171 ceph-mon[37789]: pgmap v6519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:51:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11435.889997 secs 2024-07-26T11:51:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11435.889997 secs 2024-07-26T11:51:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11435.889997 secs 2024-07-26T11:51:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:31 smithi044 ceph-mon[32404]: pgmap v6520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:31 smithi116 ceph-mon[37097]: pgmap v6520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:31 smithi171 ceph-mon[37789]: pgmap v6520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:33 smithi044 ceph-mon[32404]: pgmap v6521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:33 smithi116 ceph-mon[37097]: pgmap v6521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:33 smithi171 ceph-mon[37789]: pgmap v6521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:35 smithi044 ceph-mon[32404]: pgmap v6522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:35 smithi116 ceph-mon[37097]: pgmap v6522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:35 smithi171 ceph-mon[37789]: pgmap v6522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11440.890107 secs 2024-07-26T11:51:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11440.890107 secs 2024-07-26T11:51:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11440.890107 secs 2024-07-26T11:51:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:37 smithi044 ceph-mon[32404]: pgmap v6523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:37 smithi116 ceph-mon[37097]: pgmap v6523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:37 smithi171 ceph-mon[37789]: pgmap v6523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:39 smithi044 ceph-mon[32404]: pgmap v6524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:39 smithi116 ceph-mon[37097]: pgmap v6524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:39 smithi171 ceph-mon[37789]: pgmap v6524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11445.890268 secs 2024-07-26T11:51:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11445.890268 secs 2024-07-26T11:51:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11445.890268 secs 2024-07-26T11:51:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:41 smithi044 ceph-mon[32404]: pgmap v6525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:41 smithi116 ceph-mon[37097]: pgmap v6525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:41 smithi171 ceph-mon[37789]: pgmap v6525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:43 smithi044 ceph-mon[32404]: pgmap v6526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:43 smithi116 ceph-mon[37097]: pgmap v6526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:43 smithi171 ceph-mon[37789]: pgmap v6526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:44.444+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:51:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:44.444+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:44.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:44.444+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:51:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:45 smithi044 ceph-mon[32404]: pgmap v6527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:45 smithi116 ceph-mon[37097]: pgmap v6527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:45 smithi171 ceph-mon[37789]: pgmap v6527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11450.890443 secs 2024-07-26T11:51:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11450.890443 secs 2024-07-26T11:51:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11450.890443 secs 2024-07-26T11:51:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:47 smithi044 ceph-mon[32404]: pgmap v6528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:47 smithi116 ceph-mon[37097]: pgmap v6528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:47 smithi171 ceph-mon[37789]: pgmap v6528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:49 smithi044 ceph-mon[32404]: pgmap v6529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:49 smithi116 ceph-mon[37097]: pgmap v6529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:49 smithi171 ceph-mon[37789]: pgmap v6529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11455.890577 secs 2024-07-26T11:51:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:51 smithi171 ceph-mon[37789]: pgmap v6530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11455.890577 secs 2024-07-26T11:51:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:51 smithi044 ceph-mon[32404]: pgmap v6530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11455.890577 secs 2024-07-26T11:51:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:51 smithi116 ceph-mon[37097]: pgmap v6530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:53 smithi044 ceph-mon[32404]: pgmap v6531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:53 smithi116 ceph-mon[37097]: pgmap v6531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:53 smithi171 ceph-mon[37789]: pgmap v6531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:55.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:55 smithi044 ceph-mon[32404]: pgmap v6532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:55 smithi116 ceph-mon[37097]: pgmap v6532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:55 smithi171 ceph-mon[37789]: pgmap v6532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11460.890726 secs 2024-07-26T11:51:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11460.890726 secs 2024-07-26T11:51:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11460.890726 secs 2024-07-26T11:51:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:57 smithi044 ceph-mon[32404]: pgmap v6533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:57 smithi116 ceph-mon[37097]: pgmap v6533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:57 smithi171 ceph-mon[37789]: pgmap v6533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:59.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:59.455+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:51:59.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:59.455+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:51:59.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:51:59.455+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:59.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:51:59 smithi044 ceph-mon[32404]: pgmap v6534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:51:59 smithi116 ceph-mon[37097]: pgmap v6534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:51:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:51:59 smithi171 ceph-mon[37789]: pgmap v6534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11465.890892 secs 2024-07-26T11:52:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11465.890892 secs 2024-07-26T11:52:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11465.890892 secs 2024-07-26T11:52:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:01 smithi044 ceph-mon[32404]: pgmap v6535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:01 smithi116 ceph-mon[37097]: pgmap v6535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:01 smithi171 ceph-mon[37789]: pgmap v6535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:03 smithi044 ceph-mon[32404]: pgmap v6536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:03 smithi116 ceph-mon[37097]: pgmap v6536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:03 smithi171 ceph-mon[37789]: pgmap v6536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:05 smithi044 ceph-mon[32404]: pgmap v6537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:05 smithi116 ceph-mon[37097]: pgmap v6537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:05 smithi171 ceph-mon[37789]: pgmap v6537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11470.891045 secs 2024-07-26T11:52:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11470.891045 secs 2024-07-26T11:52:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11470.891045 secs 2024-07-26T11:52:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:08 smithi044 ceph-mon[32404]: pgmap v6538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:08 smithi116 ceph-mon[37097]: pgmap v6538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:08 smithi171 ceph-mon[37789]: pgmap v6538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:10 smithi044 ceph-mon[32404]: pgmap v6539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:10 smithi116 ceph-mon[37097]: pgmap v6539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:10 smithi171 ceph-mon[37789]: pgmap v6539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11475.891209 secs 2024-07-26T11:52:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11475.891209 secs 2024-07-26T11:52:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11475.891209 secs 2024-07-26T11:52:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:12 smithi044 ceph-mon[32404]: pgmap v6540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:12 smithi116 ceph-mon[37097]: pgmap v6540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:12 smithi171 ceph-mon[37789]: pgmap v6540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:14.459+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:52:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:14.460+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:52:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:14.460+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:14 smithi044 ceph-mon[32404]: pgmap v6541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:52:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:14 smithi116 ceph-mon[37097]: pgmap v6541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:52:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:14 smithi171 ceph-mon[37789]: pgmap v6541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:16 smithi044 ceph-mon[32404]: pgmap v6542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11480.891344 secs 2024-07-26T11:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:16 smithi116 ceph-mon[37097]: pgmap v6542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11480.891344 secs 2024-07-26T11:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:16 smithi171 ceph-mon[37789]: pgmap v6542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11480.891344 secs 2024-07-26T11:52:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:18 smithi044 ceph-mon[32404]: pgmap v6543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:18 smithi116 ceph-mon[37097]: pgmap v6543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:18 smithi171 ceph-mon[37789]: pgmap v6543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:52:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:20 smithi044 ceph-mon[32404]: pgmap v6544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:20 smithi116 ceph-mon[37097]: pgmap v6544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:20 smithi171 ceph-mon[37789]: pgmap v6544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11485.891463 secs 2024-07-26T11:52:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11485.891463 secs 2024-07-26T11:52:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11485.891463 secs 2024-07-26T11:52:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:22 smithi044 ceph-mon[32404]: pgmap v6545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:22 smithi116 ceph-mon[37097]: pgmap v6545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:22 smithi171 ceph-mon[37789]: pgmap v6545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:23 smithi044 ceph-mon[32404]: pgmap v6546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:23 smithi044 ceph-mon[32404]: pgmap v6547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:23.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:52:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:23 smithi116 ceph-mon[37097]: pgmap v6546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:23 smithi116 ceph-mon[37097]: pgmap v6547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:23.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:52:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:23 smithi171 ceph-mon[37789]: pgmap v6546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:23 smithi171 ceph-mon[37789]: pgmap v6547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:52:23.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:52:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:25 smithi044 ceph-mon[32404]: pgmap v6548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:25 smithi116 ceph-mon[37097]: pgmap v6548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:25 smithi171 ceph-mon[37789]: pgmap v6548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11490.891591 secs 2024-07-26T11:52:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11490.891591 secs 2024-07-26T11:52:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11490.891591 secs 2024-07-26T11:52:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:27 smithi044 ceph-mon[32404]: pgmap v6549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:52:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:27 smithi116 ceph-mon[37097]: pgmap v6549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:52:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:27 smithi171 ceph-mon[37789]: pgmap v6549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:52:29.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:29.465+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:52:29.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:29.465+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:52:29.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:29.465+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:29 smithi044 ceph-mon[32404]: pgmap v6550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:29 smithi116 ceph-mon[37097]: pgmap v6550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:29 smithi171 ceph-mon[37789]: pgmap v6550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:31 smithi116 ceph-mon[37097]: pgmap v6551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11495.891759 secs 2024-07-26T11:52:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:31 smithi171 ceph-mon[37789]: pgmap v6551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11495.891759 secs 2024-07-26T11:52:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:31 smithi044 ceph-mon[32404]: pgmap v6551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:52:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11495.891759 secs 2024-07-26T11:52:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:33 smithi171 ceph-mon[37789]: pgmap v6552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:33 smithi044 ceph-mon[32404]: pgmap v6552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:33 smithi116 ceph-mon[37097]: pgmap v6552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:35 smithi171 ceph-mon[37789]: pgmap v6553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:35 smithi044 ceph-mon[32404]: pgmap v6553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:35 smithi116 ceph-mon[37097]: pgmap v6553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11500.891857 secs 2024-07-26T11:52:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11500.891857 secs 2024-07-26T11:52:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11500.891857 secs 2024-07-26T11:52:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:37 smithi171 ceph-mon[37789]: pgmap v6554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:37 smithi044 ceph-mon[32404]: pgmap v6554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:38.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:37 smithi116 ceph-mon[37097]: pgmap v6554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:39 smithi171 ceph-mon[37789]: pgmap v6555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:39 smithi044 ceph-mon[32404]: pgmap v6555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:39 smithi116 ceph-mon[37097]: pgmap v6555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:41 smithi171 ceph-mon[37789]: pgmap v6556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11505.892017 secs 2024-07-26T11:52:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:41 smithi044 ceph-mon[32404]: pgmap v6556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11505.892017 secs 2024-07-26T11:52:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:41 smithi116 ceph-mon[37097]: pgmap v6556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:42.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11505.892017 secs 2024-07-26T11:52:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:43 smithi171 ceph-mon[37789]: pgmap v6557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:43 smithi044 ceph-mon[32404]: pgmap v6557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:43 smithi116 ceph-mon[37097]: pgmap v6557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:44.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:44.471+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:52:44.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:44.471+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:44.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:44.471+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:52:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:45 smithi171 ceph-mon[37789]: pgmap v6558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11510.892117 secs 2024-07-26T11:52:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:45 smithi044 ceph-mon[32404]: pgmap v6558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11510.892117 secs 2024-07-26T11:52:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:45 smithi116 ceph-mon[37097]: pgmap v6558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11510.892117 secs 2024-07-26T11:52:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:47 smithi171 ceph-mon[37789]: pgmap v6559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:47 smithi044 ceph-mon[32404]: pgmap v6559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:47 smithi116 ceph-mon[37097]: pgmap v6559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:49 smithi171 ceph-mon[37789]: pgmap v6560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:49 smithi044 ceph-mon[32404]: pgmap v6560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:49 smithi116 ceph-mon[37097]: pgmap v6560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11515.892262 secs 2024-07-26T11:52:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11515.892262 secs 2024-07-26T11:52:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11515.892262 secs 2024-07-26T11:52:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:51 smithi044 ceph-mon[32404]: pgmap v6561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:51 smithi116 ceph-mon[37097]: pgmap v6561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:51 smithi171 ceph-mon[37789]: pgmap v6561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:53 smithi044 ceph-mon[32404]: pgmap v6562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:53 smithi116 ceph-mon[37097]: pgmap v6562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:53 smithi171 ceph-mon[37789]: pgmap v6562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:52:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:55 smithi044 ceph-mon[32404]: pgmap v6563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:55 smithi044 ceph-mon[32404]: 1 slow requests, 1 included below; oldest blocked for > 11520.892394 secs 2024-07-26T11:52:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:55 smithi044 ceph-mon[32404]: slow request 11520.892393 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:52:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:55 smithi116 ceph-mon[37097]: pgmap v6563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:55 smithi116 ceph-mon[37097]: 1 slow requests, 1 included below; oldest blocked for > 11520.892394 secs 2024-07-26T11:52:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:55 smithi116 ceph-mon[37097]: slow request 11520.892393 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:52:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:55 smithi171 ceph-mon[37789]: pgmap v6563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:52:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:55 smithi171 ceph-mon[37789]: 1 slow requests, 1 included below; oldest blocked for > 11520.892394 secs 2024-07-26T11:52:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:55 smithi171 ceph-mon[37789]: slow request 11520.892393 seconds old, received at 2024-07-26T08:40:54.652186+0000: client_request(client.25387:7567 rename #0x10000000299/builddefs #0x10000001668/out 2024-07-26T08:40:54.650445+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T11:52:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:57 smithi044 ceph-mon[32404]: pgmap v6564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:57 smithi116 ceph-mon[37097]: pgmap v6564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:57 smithi171 ceph-mon[37789]: pgmap v6564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:59.473+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:52:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:59.473+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:52:59.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:52:59.473+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:52:59 smithi044 ceph-mon[32404]: pgmap v6565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:52:59 smithi116 ceph-mon[37097]: pgmap v6565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:52:59 smithi171 ceph-mon[37789]: pgmap v6565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11525.892538 secs 2024-07-26T11:53:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11525.892538 secs 2024-07-26T11:53:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11525.892538 secs 2024-07-26T11:53:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:01 smithi044 ceph-mon[32404]: pgmap v6566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:01 smithi116 ceph-mon[37097]: pgmap v6566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:01 smithi171 ceph-mon[37789]: pgmap v6566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:03.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:03 smithi171 ceph-mon[37789]: pgmap v6567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:03 smithi044 ceph-mon[32404]: pgmap v6567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:03 smithi116 ceph-mon[37097]: pgmap v6567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:05 smithi044 ceph-mon[32404]: pgmap v6568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11530.892676 secs 2024-07-26T11:53:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:05 smithi116 ceph-mon[37097]: pgmap v6568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11530.892676 secs 2024-07-26T11:53:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:05 smithi171 ceph-mon[37789]: pgmap v6568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11530.892676 secs 2024-07-26T11:53:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:07 smithi044 ceph-mon[32404]: pgmap v6569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:07 smithi116 ceph-mon[37097]: pgmap v6569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:07 smithi171 ceph-mon[37789]: pgmap v6569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:09 smithi044 ceph-mon[32404]: pgmap v6570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:09 smithi116 ceph-mon[37097]: pgmap v6570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:09 smithi171 ceph-mon[37789]: pgmap v6570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11535.892778 secs 2024-07-26T11:53:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11535.892778 secs 2024-07-26T11:53:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11535.892778 secs 2024-07-26T11:53:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:11 smithi044 ceph-mon[32404]: pgmap v6571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:11 smithi116 ceph-mon[37097]: pgmap v6571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:11 smithi171 ceph-mon[37789]: pgmap v6571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:13 smithi044 ceph-mon[32404]: pgmap v6572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:13 smithi116 ceph-mon[37097]: pgmap v6572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:13 smithi171 ceph-mon[37789]: pgmap v6572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:14.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:14.476+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:53:14.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:14.476+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:53:14.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:14.476+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:15 smithi044 ceph-mon[32404]: pgmap v6573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11540.892925 secs 2024-07-26T11:53:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:15 smithi116 ceph-mon[37097]: pgmap v6573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11540.892925 secs 2024-07-26T11:53:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:15 smithi171 ceph-mon[37789]: pgmap v6573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11540.892925 secs 2024-07-26T11:53:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:16.747+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:53:16.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:16.747+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:53:16.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:16.747+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:53:16.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:16.747+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:53:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:17 smithi044 ceph-mon[32404]: pgmap v6574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:17 smithi116 ceph-mon[37097]: pgmap v6574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:17 smithi171 ceph-mon[37789]: pgmap v6574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:19 smithi044 ceph-mon[32404]: pgmap v6575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:19 smithi116 ceph-mon[37097]: pgmap v6575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:19 smithi171 ceph-mon[37789]: pgmap v6575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11545.893085 secs 2024-07-26T11:53:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11545.893085 secs 2024-07-26T11:53:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11545.893085 secs 2024-07-26T11:53:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:21 smithi044 ceph-mon[32404]: pgmap v6576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:21 smithi116 ceph-mon[37097]: pgmap v6576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:21 smithi171 ceph-mon[37789]: pgmap v6576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:23 smithi044 ceph-mon[32404]: pgmap v6577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:24.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:53:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:23 smithi116 ceph-mon[37097]: pgmap v6577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:24.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:24.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:24.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:53:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:23 smithi171 ceph-mon[37789]: pgmap v6577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:24.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:53:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:24 smithi044 ceph-mon[32404]: pgmap v6578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:53:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:24 smithi044 ceph-mon[32404]: pgmap v6579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:24 smithi116 ceph-mon[37097]: pgmap v6578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:53:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:24 smithi116 ceph-mon[37097]: pgmap v6579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:24 smithi171 ceph-mon[37789]: pgmap v6578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:53:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:24 smithi171 ceph-mon[37789]: pgmap v6579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11550.893246 secs 2024-07-26T11:53:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11550.893246 secs 2024-07-26T11:53:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11550.893246 secs 2024-07-26T11:53:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:26 smithi044 ceph-mon[32404]: pgmap v6580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:53:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:26 smithi116 ceph-mon[37097]: pgmap v6580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:53:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:26 smithi171 ceph-mon[37789]: pgmap v6580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:53:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:28 smithi044 ceph-mon[32404]: pgmap v6581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:28 smithi116 ceph-mon[37097]: pgmap v6581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:28 smithi171 ceph-mon[37789]: pgmap v6581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:29.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:29.480+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:53:29.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:29.480+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:53:29.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:29.480+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:30 smithi044 ceph-mon[32404]: pgmap v6582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11555.893415 secs 2024-07-26T11:53:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:30 smithi116 ceph-mon[37097]: pgmap v6582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11555.893415 secs 2024-07-26T11:53:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:30 smithi171 ceph-mon[37789]: pgmap v6582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11555.893415 secs 2024-07-26T11:53:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:32 smithi044 ceph-mon[32404]: pgmap v6583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:32 smithi116 ceph-mon[37097]: pgmap v6583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:32 smithi171 ceph-mon[37789]: pgmap v6583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:53:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:34 smithi044 ceph-mon[32404]: pgmap v6584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:34 smithi116 ceph-mon[37097]: pgmap v6584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:34 smithi171 ceph-mon[37789]: pgmap v6584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11560.893550 secs 2024-07-26T11:53:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11560.893550 secs 2024-07-26T11:53:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11560.893550 secs 2024-07-26T11:53:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:36 smithi171 ceph-mon[37789]: pgmap v6585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:36 smithi044 ceph-mon[32404]: pgmap v6585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:36 smithi116 ceph-mon[37097]: pgmap v6585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:38 smithi171 ceph-mon[37789]: pgmap v6586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:38 smithi044 ceph-mon[32404]: pgmap v6586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:38 smithi116 ceph-mon[37097]: pgmap v6586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:40 smithi171 ceph-mon[37789]: pgmap v6587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11565.893696 secs 2024-07-26T11:53:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:40 smithi044 ceph-mon[32404]: pgmap v6587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11565.893696 secs 2024-07-26T11:53:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:40 smithi116 ceph-mon[37097]: pgmap v6587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11565.893696 secs 2024-07-26T11:53:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:42 smithi171 ceph-mon[37789]: pgmap v6588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:42 smithi044 ceph-mon[32404]: pgmap v6588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:42 smithi116 ceph-mon[37097]: pgmap v6588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:44.485+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:53:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:44.485+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:53:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:44.485+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:44 smithi171 ceph-mon[37789]: pgmap v6589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:44 smithi044 ceph-mon[32404]: pgmap v6589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:44 smithi116 ceph-mon[37097]: pgmap v6589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11570.893862 secs 2024-07-26T11:53:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11570.893862 secs 2024-07-26T11:53:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11570.893862 secs 2024-07-26T11:53:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:46 smithi171 ceph-mon[37789]: pgmap v6590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:46 smithi044 ceph-mon[32404]: pgmap v6590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:46 smithi116 ceph-mon[37097]: pgmap v6590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:49.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:48 smithi116 ceph-mon[37097]: pgmap v6591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:48 smithi171 ceph-mon[37789]: pgmap v6591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:48 smithi044 ceph-mon[32404]: pgmap v6591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:50 smithi171 ceph-mon[37789]: pgmap v6592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11575.894030 secs 2024-07-26T11:53:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:50 smithi044 ceph-mon[32404]: pgmap v6592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11575.894030 secs 2024-07-26T11:53:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:50 smithi116 ceph-mon[37097]: pgmap v6592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11575.894030 secs 2024-07-26T11:53:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:52 smithi171 ceph-mon[37789]: pgmap v6593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:52 smithi044 ceph-mon[32404]: pgmap v6593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:52 smithi116 ceph-mon[37097]: pgmap v6593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:54 smithi044 ceph-mon[32404]: pgmap v6594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:54 smithi116 ceph-mon[37097]: pgmap v6594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:54 smithi171 ceph-mon[37789]: pgmap v6594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:53:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11580.894194 secs 2024-07-26T11:53:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11580.894194 secs 2024-07-26T11:53:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11580.894194 secs 2024-07-26T11:53:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:56 smithi044 ceph-mon[32404]: pgmap v6595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:56 smithi116 ceph-mon[37097]: pgmap v6595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:56 smithi171 ceph-mon[37789]: pgmap v6595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:53:58 smithi044 ceph-mon[32404]: pgmap v6596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:53:58 smithi116 ceph-mon[37097]: pgmap v6596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:53:58 smithi171 ceph-mon[37789]: pgmap v6596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:53:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:59.489+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:53:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:59.489+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:59.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:53:59.489+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:54:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:00 smithi044 ceph-mon[32404]: pgmap v6597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11585.894364 secs 2024-07-26T11:54:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:00 smithi116 ceph-mon[37097]: pgmap v6597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11585.894364 secs 2024-07-26T11:54:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:00 smithi171 ceph-mon[37789]: pgmap v6597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11585.894364 secs 2024-07-26T11:54:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:02 smithi044 ceph-mon[32404]: pgmap v6598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:02 smithi116 ceph-mon[37097]: pgmap v6598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:02 smithi171 ceph-mon[37789]: pgmap v6598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:04.678+0000 7f76cb400640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f76ac006140 con 0x7f76d410f440 2024-07-26T11:54:04.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:04.678+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f76ac006140 con 0x7f76d410f440 2024-07-26T11:54:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:04 smithi044 ceph-mon[32404]: pgmap v6599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:04 smithi116 ceph-mon[37097]: pgmap v6599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:04 smithi171 ceph-mon[37789]: pgmap v6599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11590.894522 secs 2024-07-26T11:54:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11590.894522 secs 2024-07-26T11:54:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11590.894522 secs 2024-07-26T11:54:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:06 smithi044 ceph-mon[32404]: pgmap v6600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:06 smithi116 ceph-mon[37097]: pgmap v6600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:06 smithi171 ceph-mon[37789]: pgmap v6600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:08 smithi044 ceph-mon[32404]: pgmap v6601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:08 smithi116 ceph-mon[37097]: pgmap v6601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:08 smithi171 ceph-mon[37789]: pgmap v6601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:10 smithi044 ceph-mon[32404]: pgmap v6602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11595.894682 secs 2024-07-26T11:54:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:10 smithi116 ceph-mon[37097]: pgmap v6602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11595.894682 secs 2024-07-26T11:54:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:10 smithi171 ceph-mon[37789]: pgmap v6602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11595.894682 secs 2024-07-26T11:54:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:12 smithi044 ceph-mon[32404]: pgmap v6603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:12 smithi116 ceph-mon[37097]: pgmap v6603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:12 smithi171 ceph-mon[37789]: pgmap v6603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:14.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:14.494+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:54:14.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:14.494+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:14.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:14.494+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:54:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:14 smithi044 ceph-mon[32404]: pgmap v6604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:14 smithi116 ceph-mon[37097]: pgmap v6604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:14 smithi171 ceph-mon[37789]: pgmap v6604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11600.894843 secs 2024-07-26T11:54:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11600.894843 secs 2024-07-26T11:54:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11600.894843 secs 2024-07-26T11:54:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:16 smithi044 ceph-mon[32404]: pgmap v6605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:16 smithi116 ceph-mon[37097]: pgmap v6605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:16 smithi171 ceph-mon[37789]: pgmap v6605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:18 smithi044 ceph-mon[32404]: pgmap v6606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:18 smithi116 ceph-mon[37097]: pgmap v6606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:18 smithi171 ceph-mon[37789]: pgmap v6606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:20 smithi044 ceph-mon[32404]: pgmap v6607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11605.895006 secs 2024-07-26T11:54:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:20 smithi116 ceph-mon[37097]: pgmap v6607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11605.895006 secs 2024-07-26T11:54:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:20 smithi171 ceph-mon[37789]: pgmap v6607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11605.895006 secs 2024-07-26T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:22 smithi044 ceph-mon[32404]: pgmap v6608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:22 smithi116 ceph-mon[37097]: pgmap v6608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:22 smithi171 ceph-mon[37789]: pgmap v6608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:24 smithi044 ceph-mon[32404]: pgmap v6609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:24 smithi116 ceph-mon[37097]: pgmap v6609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:24 smithi171 ceph-mon[37789]: pgmap v6609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:25 smithi044 ceph-mon[32404]: pgmap v6610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:54:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:25 smithi044 ceph-mon[32404]: pgmap v6611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:54:26.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11610.895169 secs 2024-07-26T11:54:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:25 smithi116 ceph-mon[37097]: pgmap v6610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:54:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:25 smithi116 ceph-mon[37097]: pgmap v6611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:54:26.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11610.895169 secs 2024-07-26T11:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:25 smithi171 ceph-mon[37789]: pgmap v6610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:25 smithi171 ceph-mon[37789]: pgmap v6611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:54:26.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11610.895169 secs 2024-07-26T11:54:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:28 smithi171 ceph-mon[37789]: pgmap v6612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:28 smithi044 ceph-mon[32404]: pgmap v6612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:28 smithi116 ceph-mon[37097]: pgmap v6612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:29.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:29.501+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:54:29.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:29.501+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:54:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:29.501+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:30 smithi171 ceph-mon[37789]: pgmap v6613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:30 smithi044 ceph-mon[32404]: pgmap v6613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:30 smithi116 ceph-mon[37097]: pgmap v6613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:54:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11615.895303 secs 2024-07-26T11:54:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11615.895303 secs 2024-07-26T11:54:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11615.895303 secs 2024-07-26T11:54:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:32 smithi171 ceph-mon[37789]: pgmap v6614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:32 smithi044 ceph-mon[32404]: pgmap v6614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:32 smithi116 ceph-mon[37097]: pgmap v6614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:34 smithi171 ceph-mon[37789]: pgmap v6615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:34 smithi044 ceph-mon[32404]: pgmap v6615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:34 smithi116 ceph-mon[37097]: pgmap v6615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:54:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:36 smithi171 ceph-mon[37789]: pgmap v6616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11620.895410 secs 2024-07-26T11:54:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:36 smithi044 ceph-mon[32404]: pgmap v6616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11620.895410 secs 2024-07-26T11:54:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:36 smithi116 ceph-mon[37097]: pgmap v6616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11620.895410 secs 2024-07-26T11:54:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:38 smithi171 ceph-mon[37789]: pgmap v6617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:38 smithi044 ceph-mon[32404]: pgmap v6617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:38 smithi116 ceph-mon[37097]: pgmap v6617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:40 smithi044 ceph-mon[32404]: pgmap v6618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:40 smithi116 ceph-mon[37097]: pgmap v6618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:40 smithi171 ceph-mon[37789]: pgmap v6618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11625.895544 secs 2024-07-26T11:54:41.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11625.895544 secs 2024-07-26T11:54:41.593 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11625.895544 secs 2024-07-26T11:54:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:42 smithi044 ceph-mon[32404]: pgmap v6619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:42 smithi116 ceph-mon[37097]: pgmap v6619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:42 smithi171 ceph-mon[37789]: pgmap v6619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:44.507+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:54:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:44.507+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:54:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:44.507+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:44.510 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:44 smithi044 ceph-mon[32404]: pgmap v6620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:44 smithi116 ceph-mon[37097]: pgmap v6620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:44 smithi171 ceph-mon[37789]: pgmap v6620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:46 smithi044 ceph-mon[32404]: pgmap v6621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11630.895708 secs 2024-07-26T11:54:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:46 smithi116 ceph-mon[37097]: pgmap v6621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11630.895708 secs 2024-07-26T11:54:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:46 smithi171 ceph-mon[37789]: pgmap v6621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11630.895708 secs 2024-07-26T11:54:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:48 smithi044 ceph-mon[32404]: pgmap v6622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:48 smithi116 ceph-mon[37097]: pgmap v6622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:48 smithi171 ceph-mon[37789]: pgmap v6622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:50 smithi044 ceph-mon[32404]: pgmap v6623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:50 smithi116 ceph-mon[37097]: pgmap v6623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:50 smithi171 ceph-mon[37789]: pgmap v6623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11635.895871 secs 2024-07-26T11:54:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11635.895871 secs 2024-07-26T11:54:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11635.895871 secs 2024-07-26T11:54:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:52 smithi044 ceph-mon[32404]: pgmap v6624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:52 smithi116 ceph-mon[37097]: pgmap v6624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:52 smithi171 ceph-mon[37789]: pgmap v6624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:54 smithi044 ceph-mon[32404]: pgmap v6625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:54 smithi116 ceph-mon[37097]: pgmap v6625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:54 smithi171 ceph-mon[37789]: pgmap v6625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:56 smithi044 ceph-mon[32404]: pgmap v6626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11640.895966 secs 2024-07-26T11:54:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:56 smithi116 ceph-mon[37097]: pgmap v6626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11640.895966 secs 2024-07-26T11:54:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:56 smithi171 ceph-mon[37789]: pgmap v6626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:54:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11640.895966 secs 2024-07-26T11:54:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:54:58 smithi044 ceph-mon[32404]: pgmap v6627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:54:58 smithi116 ceph-mon[37097]: pgmap v6627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:54:58 smithi171 ceph-mon[37789]: pgmap v6627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:54:59.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:59.517+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:54:59.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:59.518+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:54:59.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:54:59.518+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:00 smithi044 ceph-mon[32404]: pgmap v6628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:00 smithi116 ceph-mon[37097]: pgmap v6628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:00 smithi171 ceph-mon[37789]: pgmap v6628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11645.896126 secs 2024-07-26T11:55:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11645.896126 secs 2024-07-26T11:55:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11645.896126 secs 2024-07-26T11:55:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:02 smithi044 ceph-mon[32404]: pgmap v6629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:02 smithi116 ceph-mon[37097]: pgmap v6629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:02 smithi171 ceph-mon[37789]: pgmap v6629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:03 smithi044 ceph-mon[32404]: pgmap v6630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:03 smithi116 ceph-mon[37097]: pgmap v6630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:03 smithi171 ceph-mon[37789]: pgmap v6630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:06 smithi171 ceph-mon[37789]: pgmap v6631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11650.896266 secs 2024-07-26T11:55:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:06 smithi044 ceph-mon[32404]: pgmap v6631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11650.896266 secs 2024-07-26T11:55:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:06 smithi116 ceph-mon[37097]: pgmap v6631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11650.896266 secs 2024-07-26T11:55:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:08 smithi171 ceph-mon[37789]: pgmap v6632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:55:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:08 smithi044 ceph-mon[32404]: pgmap v6632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:55:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:08 smithi116 ceph-mon[37097]: pgmap v6632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:55:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:10 smithi171 ceph-mon[37789]: pgmap v6633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:55:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:10 smithi044 ceph-mon[32404]: pgmap v6633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:55:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:10 smithi116 ceph-mon[37097]: pgmap v6633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T11:55:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11655.896427 secs 2024-07-26T11:55:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11655.896427 secs 2024-07-26T11:55:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11655.896427 secs 2024-07-26T11:55:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:12 smithi044 ceph-mon[32404]: pgmap v6634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:12 smithi116 ceph-mon[37097]: pgmap v6634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:12 smithi171 ceph-mon[37789]: pgmap v6634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:14 smithi044 ceph-mon[32404]: pgmap v6635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:55:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:14 smithi116 ceph-mon[37097]: pgmap v6635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:55:14.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:14.523+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:55:14.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:14.523+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:55:14.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:14.523+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:14 smithi171 ceph-mon[37789]: pgmap v6635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:55:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:16 smithi044 ceph-mon[32404]: pgmap v6636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11660.896569 secs 2024-07-26T11:55:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:16 smithi116 ceph-mon[37097]: pgmap v6636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11660.896569 secs 2024-07-26T11:55:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:16 smithi171 ceph-mon[37789]: pgmap v6636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11660.896569 secs 2024-07-26T11:55:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:18 smithi044 ceph-mon[32404]: pgmap v6637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:18 smithi116 ceph-mon[37097]: pgmap v6637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:18 smithi171 ceph-mon[37789]: pgmap v6637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:20 smithi044 ceph-mon[32404]: pgmap v6638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:20 smithi116 ceph-mon[37097]: pgmap v6638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:20 smithi171 ceph-mon[37789]: pgmap v6638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11665.896734 secs 2024-07-26T11:55:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11665.896734 secs 2024-07-26T11:55:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11665.896734 secs 2024-07-26T11:55:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:22 smithi044 ceph-mon[32404]: pgmap v6639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:22 smithi116 ceph-mon[37097]: pgmap v6639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:22 smithi171 ceph-mon[37789]: pgmap v6639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:24 smithi044 ceph-mon[32404]: pgmap v6640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:24 smithi116 ceph-mon[37097]: pgmap v6640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:24 smithi171 ceph-mon[37789]: pgmap v6640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:26 smithi044 ceph-mon[32404]: pgmap v6641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11670.896872 secs 2024-07-26T11:55:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:26 smithi116 ceph-mon[37097]: pgmap v6641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11670.896872 secs 2024-07-26T11:55:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:26 smithi171 ceph-mon[37789]: pgmap v6641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11670.896872 secs 2024-07-26T11:55:27.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:27.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:27.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:55:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:55:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:55:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:28 smithi044 ceph-mon[32404]: pgmap v6642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:28 smithi044 ceph-mon[32404]: pgmap v6643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:28 smithi116 ceph-mon[37097]: pgmap v6642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:28 smithi116 ceph-mon[37097]: pgmap v6643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:28 smithi171 ceph-mon[37789]: pgmap v6642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:28 smithi171 ceph-mon[37789]: pgmap v6643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:29.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:29.533+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:55:29.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:29.533+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:29.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:29.533+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:55:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:30 smithi044 ceph-mon[32404]: pgmap v6644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:30 smithi116 ceph-mon[37097]: pgmap v6644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:30 smithi171 ceph-mon[37789]: pgmap v6644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11675.897036 secs 2024-07-26T11:55:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11675.897036 secs 2024-07-26T11:55:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11675.897036 secs 2024-07-26T11:55:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:32 smithi044 ceph-mon[32404]: pgmap v6645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:32 smithi116 ceph-mon[37097]: pgmap v6645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:32 smithi171 ceph-mon[37789]: pgmap v6645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:33 smithi044 ceph-mon[32404]: pgmap v6646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:33 smithi116 ceph-mon[37097]: pgmap v6646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:33 smithi171 ceph-mon[37789]: pgmap v6646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:35 smithi171 ceph-mon[37789]: pgmap v6647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:35 smithi044 ceph-mon[32404]: pgmap v6647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:35 smithi116 ceph-mon[37097]: pgmap v6647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:55:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11680.897203 secs 2024-07-26T11:55:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11680.897203 secs 2024-07-26T11:55:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11680.897203 secs 2024-07-26T11:55:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:37 smithi171 ceph-mon[37789]: pgmap v6648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:37 smithi044 ceph-mon[32404]: pgmap v6648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:37 smithi116 ceph-mon[37097]: pgmap v6648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:55:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:39 smithi044 ceph-mon[32404]: pgmap v6649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:39 smithi116 ceph-mon[37097]: pgmap v6649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:39 smithi171 ceph-mon[37789]: pgmap v6649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:41 smithi044 ceph-mon[32404]: pgmap v6650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11685.897365 secs 2024-07-26T11:55:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:41 smithi116 ceph-mon[37097]: pgmap v6650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11685.897365 secs 2024-07-26T11:55:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:41 smithi171 ceph-mon[37789]: pgmap v6650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11685.897365 secs 2024-07-26T11:55:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:43 smithi044 ceph-mon[32404]: pgmap v6651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:43 smithi116 ceph-mon[37097]: pgmap v6651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:43 smithi171 ceph-mon[37789]: pgmap v6651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:44.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:44.538+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:55:44.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:44.539+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:55:44.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:44.539+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:45 smithi044 ceph-mon[32404]: pgmap v6652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:45 smithi116 ceph-mon[37097]: pgmap v6652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:45 smithi171 ceph-mon[37789]: pgmap v6652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11690.897528 secs 2024-07-26T11:55:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11690.897528 secs 2024-07-26T11:55:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11690.897528 secs 2024-07-26T11:55:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:47 smithi044 ceph-mon[32404]: pgmap v6653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:47 smithi116 ceph-mon[37097]: pgmap v6653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:47 smithi171 ceph-mon[37789]: pgmap v6653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:49 smithi044 ceph-mon[32404]: pgmap v6654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:49 smithi116 ceph-mon[37097]: pgmap v6654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:49 smithi171 ceph-mon[37789]: pgmap v6654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:51 smithi044 ceph-mon[32404]: pgmap v6655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11695.897688 secs 2024-07-26T11:55:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:51 smithi116 ceph-mon[37097]: pgmap v6655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11695.897688 secs 2024-07-26T11:55:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:51 smithi171 ceph-mon[37789]: pgmap v6655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11695.897688 secs 2024-07-26T11:55:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:53 smithi044 ceph-mon[32404]: pgmap v6656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:53 smithi116 ceph-mon[37097]: pgmap v6656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:53.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:53 smithi171 ceph-mon[37789]: pgmap v6656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:55 smithi044 ceph-mon[32404]: pgmap v6657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:55 smithi116 ceph-mon[37097]: pgmap v6657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:55 smithi171 ceph-mon[37789]: pgmap v6657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:55:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11700.897849 secs 2024-07-26T11:55:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11700.897849 secs 2024-07-26T11:55:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11700.897849 secs 2024-07-26T11:55:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:57 smithi044 ceph-mon[32404]: pgmap v6658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:57 smithi116 ceph-mon[37097]: pgmap v6658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:57 smithi171 ceph-mon[37789]: pgmap v6658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:59.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:59.549+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:55:59.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:59.549+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:59.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:55:59.549+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:55:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:55:59 smithi044 ceph-mon[32404]: pgmap v6659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:55:59 smithi116 ceph-mon[37097]: pgmap v6659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:55:59 smithi171 ceph-mon[37789]: pgmap v6659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:01 smithi044 ceph-mon[32404]: pgmap v6660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11705.897992 secs 2024-07-26T11:56:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:01 smithi116 ceph-mon[37097]: pgmap v6660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11705.897992 secs 2024-07-26T11:56:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:01 smithi171 ceph-mon[37789]: pgmap v6660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11705.897992 secs 2024-07-26T11:56:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:03 smithi044 ceph-mon[32404]: pgmap v6661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:03 smithi116 ceph-mon[37097]: pgmap v6661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:03 smithi171 ceph-mon[37789]: pgmap v6661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:05 smithi044 ceph-mon[32404]: pgmap v6662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:05 smithi116 ceph-mon[37097]: pgmap v6662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:05 smithi171 ceph-mon[37789]: pgmap v6662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:06.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11710.898132 secs 2024-07-26T11:56:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11710.898132 secs 2024-07-26T11:56:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11710.898132 secs 2024-07-26T11:56:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:07 smithi044 ceph-mon[32404]: pgmap v6663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:07 smithi116 ceph-mon[37097]: pgmap v6663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:07 smithi171 ceph-mon[37789]: pgmap v6663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:09 smithi044 ceph-mon[32404]: pgmap v6664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:09 smithi116 ceph-mon[37097]: pgmap v6664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:09 smithi171 ceph-mon[37789]: pgmap v6664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:11 smithi044 ceph-mon[32404]: pgmap v6665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11715.898272 secs 2024-07-26T11:56:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:11 smithi116 ceph-mon[37097]: pgmap v6665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11715.898272 secs 2024-07-26T11:56:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:11 smithi171 ceph-mon[37789]: pgmap v6665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11715.898272 secs 2024-07-26T11:56:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:13 smithi044 ceph-mon[32404]: pgmap v6666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:13 smithi116 ceph-mon[37097]: pgmap v6666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:13 smithi171 ceph-mon[37789]: pgmap v6666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:14.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.555+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:56:14.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.555+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:14.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.555+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:56:14.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.723+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T11:56:14.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.723+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:56:14.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf5807e070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:56:14.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf5807e070 comp rx=0 tx=0).stop 2024-07-26T11:56:14.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T11:56:14.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:56:14.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:56:14.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:56:14.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.724+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:56:14.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:14.727+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58074040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:56:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:15 smithi044 ceph-mon[32404]: pgmap v6667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:15 smithi116 ceph-mon[37097]: pgmap v6667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:15.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:15 smithi171 ceph-mon[37789]: pgmap v6667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11720.898438 secs 2024-07-26T11:56:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11720.898438 secs 2024-07-26T11:56:16.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:16.780+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:56:16.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:16.780+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:56:16.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:16.780+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:56:16.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:16.780+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:56:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11720.898438 secs 2024-07-26T11:56:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:17 smithi044 ceph-mon[32404]: pgmap v6668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:17 smithi116 ceph-mon[37097]: pgmap v6668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:17 smithi171 ceph-mon[37789]: pgmap v6668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:19 smithi044 ceph-mon[32404]: pgmap v6669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:19 smithi116 ceph-mon[37097]: pgmap v6669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:19 smithi171 ceph-mon[37789]: pgmap v6669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:21 smithi044 ceph-mon[32404]: pgmap v6670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11725.898590 secs 2024-07-26T11:56:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:21 smithi116 ceph-mon[37097]: pgmap v6670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11725.898590 secs 2024-07-26T11:56:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:21 smithi171 ceph-mon[37789]: pgmap v6670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11725.898590 secs 2024-07-26T11:56:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:23 smithi044 ceph-mon[32404]: pgmap v6671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:23 smithi116 ceph-mon[37097]: pgmap v6671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:23 smithi171 ceph-mon[37789]: pgmap v6671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:25 smithi044 ceph-mon[32404]: pgmap v6672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:25 smithi116 ceph-mon[37097]: pgmap v6672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:25 smithi171 ceph-mon[37789]: pgmap v6672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11730.898744 secs 2024-07-26T11:56:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11730.898744 secs 2024-07-26T11:56:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11730.898744 secs 2024-07-26T11:56:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:27 smithi044 ceph-mon[32404]: pgmap v6673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:28.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:56:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:27 smithi116 ceph-mon[37097]: pgmap v6673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:56:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:27 smithi171 ceph-mon[37789]: pgmap v6673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:56:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:28 smithi044 ceph-mon[32404]: pgmap v6674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:28 smithi044 ceph-mon[32404]: pgmap v6675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:28 smithi116 ceph-mon[37097]: pgmap v6674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:28 smithi116 ceph-mon[37097]: pgmap v6675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:28 smithi171 ceph-mon[37789]: pgmap v6674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:56:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:28 smithi171 ceph-mon[37789]: pgmap v6675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:29.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:29.564+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:56:29.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:29.564+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:29.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:29.564+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:56:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:30 smithi044 ceph-mon[32404]: pgmap v6676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11735.898876 secs 2024-07-26T11:56:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:30 smithi116 ceph-mon[37097]: pgmap v6676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11735.898876 secs 2024-07-26T11:56:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:30 smithi171 ceph-mon[37789]: pgmap v6676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11735.898876 secs 2024-07-26T11:56:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:32 smithi044 ceph-mon[32404]: pgmap v6677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:56:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:32 smithi116 ceph-mon[37097]: pgmap v6677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:56:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:32 smithi171 ceph-mon[37789]: pgmap v6677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:56:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:34 smithi044 ceph-mon[32404]: pgmap v6678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:34 smithi116 ceph-mon[37097]: pgmap v6678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:34 smithi171 ceph-mon[37789]: pgmap v6678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11740.899037 secs 2024-07-26T11:56:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11740.899037 secs 2024-07-26T11:56:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11740.899037 secs 2024-07-26T11:56:36.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:36 smithi171 ceph-mon[37789]: pgmap v6679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:36 smithi044 ceph-mon[32404]: pgmap v6679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:36 smithi116 ceph-mon[37097]: pgmap v6679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:38 smithi044 ceph-mon[32404]: pgmap v6680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:38 smithi116 ceph-mon[37097]: pgmap v6680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:38 smithi171 ceph-mon[37789]: pgmap v6680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:56:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:40 smithi044 ceph-mon[32404]: pgmap v6681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11745.899182 secs 2024-07-26T11:56:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:40 smithi116 ceph-mon[37097]: pgmap v6681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11745.899182 secs 2024-07-26T11:56:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:40 smithi171 ceph-mon[37789]: pgmap v6681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11745.899182 secs 2024-07-26T11:56:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:42 smithi044 ceph-mon[32404]: pgmap v6682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:42 smithi116 ceph-mon[37097]: pgmap v6682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:42 smithi171 ceph-mon[37789]: pgmap v6682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:44.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:44.569+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:56:44.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:44.569+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:44.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:44.569+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:56:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:44 smithi044 ceph-mon[32404]: pgmap v6683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:44 smithi116 ceph-mon[37097]: pgmap v6683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:44 smithi171 ceph-mon[37789]: pgmap v6683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11750.899301 secs 2024-07-26T11:56:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11750.899301 secs 2024-07-26T11:56:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11750.899301 secs 2024-07-26T11:56:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:46 smithi044 ceph-mon[32404]: pgmap v6684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:46 smithi116 ceph-mon[37097]: pgmap v6684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:46 smithi171 ceph-mon[37789]: pgmap v6684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:48 smithi044 ceph-mon[32404]: pgmap v6685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:48 smithi116 ceph-mon[37097]: pgmap v6685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:48 smithi171 ceph-mon[37789]: pgmap v6685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:50 smithi044 ceph-mon[32404]: pgmap v6686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11755.899448 secs 2024-07-26T11:56:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:50 smithi116 ceph-mon[37097]: pgmap v6686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11755.899448 secs 2024-07-26T11:56:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:50 smithi171 ceph-mon[37789]: pgmap v6686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11755.899448 secs 2024-07-26T11:56:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:52 smithi044 ceph-mon[32404]: pgmap v6687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:52 smithi116 ceph-mon[37097]: pgmap v6687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:52 smithi171 ceph-mon[37789]: pgmap v6687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:54 smithi044 ceph-mon[32404]: pgmap v6688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:54 smithi116 ceph-mon[37097]: pgmap v6688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:54 smithi171 ceph-mon[37789]: pgmap v6688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11760.899609 secs 2024-07-26T11:56:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11760.899609 secs 2024-07-26T11:56:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11760.899609 secs 2024-07-26T11:56:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:56 smithi171 ceph-mon[37789]: pgmap v6689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:56 smithi044 ceph-mon[32404]: pgmap v6689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:56 smithi116 ceph-mon[37097]: pgmap v6689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:56:58 smithi171 ceph-mon[37789]: pgmap v6690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:56:58 smithi044 ceph-mon[32404]: pgmap v6690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:56:58 smithi116 ceph-mon[37097]: pgmap v6690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:56:59.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:59.575+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:56:59.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:59.576+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:56:59.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:56:59.576+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:00 smithi171 ceph-mon[37789]: pgmap v6691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11765.899740 secs 2024-07-26T11:57:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:00 smithi044 ceph-mon[32404]: pgmap v6691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11765.899740 secs 2024-07-26T11:57:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:00 smithi116 ceph-mon[37097]: pgmap v6691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11765.899740 secs 2024-07-26T11:57:03.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:02 smithi044 ceph-mon[32404]: pgmap v6692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:02 smithi171 ceph-mon[37789]: pgmap v6692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:02 smithi116 ceph-mon[37097]: pgmap v6692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:04.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.559+0000 7f76d3400640 1 -- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 msgr2=0x7f76b80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900005110) for more than 900000000 us, fault. 2024-07-26T11:57:04.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.559+0000 7f76d3400640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 0x7f76b80b62d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f76c40096c0 tx=0x7f76c4005a20 comp rx=0 tx=0).stop 2024-07-26T11:57:04.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.559+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 msgr2=0x7f76b80b62d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:57:04.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.559+0000 7f76d0c00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80b3dc0 0x7f76b80b62d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:57:04.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.559+0000 7f76d0c00640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80c06d0 0x7f76b80c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:57:04.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.559+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f76bc012e50 con 0x7f76b80c06d0 2024-07-26T11:57:04.562 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.560+0000 7f76d3400640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80c06d0 0x7f76b80c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:04.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.562+0000 7f76d3400640 1 --2- 172.21.15.44:0/1321661336 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7f76b80c06d0 0x7f76b80c2af0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005ec0 tx=0x7f76c403f000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:57:04.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.711+0000 7f76cb400640 1 -- 172.21.15.44:0/1321661336 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f76ac006400 con 0x7f76d410f440 2024-07-26T11:57:04.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:04.711+0000 7f76d0c00640 1 -- 172.21.15.44:0/1321661336 <== mon.1 v2:172.21.15.171:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f76ac006400 con 0x7f76d410f440 2024-07-26T11:57:04.817 INFO:teuthology.orchestra.run.smithi044.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-07-26T11:57:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-07-26T11:57:04.867 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/'" 2024-07-26T11:57:04.874 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-07-26T11:57:04.884 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-07-26T11:57:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:04 smithi171 ceph-mon[37789]: pgmap v6693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:04 smithi044 ceph-mon[32404]: pgmap v6693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:04 smithi116 ceph-mon[37097]: pgmap v6693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:05 smithi171 ceph-mon[37789]: from='client.25912 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:57:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11770.899855 secs 2024-07-26T11:57:06.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:05 smithi044 ceph-mon[32404]: from='client.25912 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:57:06.169 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11770.899855 secs 2024-07-26T11:57:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:05 smithi116 ceph-mon[37097]: from='client.25912 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:57:06.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11770.899855 secs 2024-07-26T11:57:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:06 smithi171 ceph-mon[37789]: pgmap v6694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:06 smithi044 ceph-mon[32404]: pgmap v6694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:06 smithi116 ceph-mon[37097]: pgmap v6694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:08 smithi171 ceph-mon[37789]: pgmap v6695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:08 smithi044 ceph-mon[32404]: pgmap v6695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:08 smithi116 ceph-mon[37097]: pgmap v6695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:10 smithi171 ceph-mon[37789]: pgmap v6696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11775.900022 secs 2024-07-26T11:57:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:10 smithi044 ceph-mon[32404]: pgmap v6696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11775.900022 secs 2024-07-26T11:57:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:10 smithi116 ceph-mon[37097]: pgmap v6696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11775.900022 secs 2024-07-26T11:57:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:12 smithi171 ceph-mon[37789]: pgmap v6697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:12 smithi044 ceph-mon[32404]: pgmap v6697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:12 smithi116 ceph-mon[37097]: pgmap v6697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:14.581+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:57:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:14.581+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:14.581+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:57:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:14 smithi044 ceph-mon[32404]: pgmap v6698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:14 smithi116 ceph-mon[37097]: pgmap v6698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:14 smithi171 ceph-mon[37789]: pgmap v6698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11780.900178 secs 2024-07-26T11:57:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11780.900178 secs 2024-07-26T11:57:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11780.900178 secs 2024-07-26T11:57:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:16 smithi044 ceph-mon[32404]: pgmap v6699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:16 smithi116 ceph-mon[37097]: pgmap v6699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:16 smithi171 ceph-mon[37789]: pgmap v6699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:18 smithi044 ceph-mon[32404]: pgmap v6700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:18 smithi116 ceph-mon[37097]: pgmap v6700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:18 smithi171 ceph-mon[37789]: pgmap v6700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:20 smithi044 ceph-mon[32404]: pgmap v6701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11785.900325 secs 2024-07-26T11:57:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:20 smithi116 ceph-mon[37097]: pgmap v6701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:21.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11785.900325 secs 2024-07-26T11:57:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:20 smithi171 ceph-mon[37789]: pgmap v6701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11785.900325 secs 2024-07-26T11:57:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:22 smithi044 ceph-mon[32404]: pgmap v6702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:22 smithi116 ceph-mon[37097]: pgmap v6702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:22 smithi171 ceph-mon[37789]: pgmap v6702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:57:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:24 smithi044 ceph-mon[32404]: pgmap v6703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:24 smithi116 ceph-mon[37097]: pgmap v6703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:24 smithi171 ceph-mon[37789]: pgmap v6703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:57:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11790.900805 secs 2024-07-26T11:57:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11790.900805 secs 2024-07-26T11:57:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11790.900805 secs 2024-07-26T11:57:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:26 smithi044 ceph-mon[32404]: pgmap v6704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:26 smithi116 ceph-mon[37097]: pgmap v6704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:26 smithi171 ceph-mon[37789]: pgmap v6704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: pgmap v6705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:29.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:57:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: pgmap v6705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:29.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:57:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: pgmap v6705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:57:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:29.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:29.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:57:29.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:29.591+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:57:29.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:29.591+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:29.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:29.591+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:57:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:29 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:57:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:29 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:57:30.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:29 smithi044 ceph-mon[32404]: pgmap v6706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:29 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:57:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:29 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:57:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:29 smithi116 ceph-mon[37097]: pgmap v6706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:29 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T11:57:30.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:29 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T11:57:30.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:29 smithi171 ceph-mon[37789]: pgmap v6706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11795.900941 secs 2024-07-26T11:57:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11795.900941 secs 2024-07-26T11:57:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11795.900941 secs 2024-07-26T11:57:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:31 smithi044 ceph-mon[32404]: pgmap v6707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:31 smithi116 ceph-mon[37097]: pgmap v6707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:31 smithi171 ceph-mon[37789]: pgmap v6707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:33 smithi044 ceph-mon[32404]: pgmap v6708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:33 smithi116 ceph-mon[37097]: pgmap v6708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:33 smithi171 ceph-mon[37789]: pgmap v6708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:35 smithi044 ceph-mon[32404]: pgmap v6709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11800.901077 secs 2024-07-26T11:57:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:35 smithi116 ceph-mon[37097]: pgmap v6709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:36.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11800.901077 secs 2024-07-26T11:57:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:35 smithi171 ceph-mon[37789]: pgmap v6709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11800.901077 secs 2024-07-26T11:57:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:37 smithi044 ceph-mon[32404]: pgmap v6710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:37 smithi116 ceph-mon[37097]: pgmap v6710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:37 smithi171 ceph-mon[37789]: pgmap v6710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:39 smithi044 ceph-mon[32404]: pgmap v6711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:39 smithi116 ceph-mon[37097]: pgmap v6711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:39 smithi171 ceph-mon[37789]: pgmap v6711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11805.901215 secs 2024-07-26T11:57:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11805.901215 secs 2024-07-26T11:57:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11805.901215 secs 2024-07-26T11:57:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:41 smithi044 ceph-mon[32404]: pgmap v6712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:41 smithi116 ceph-mon[37097]: pgmap v6712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:41 smithi171 ceph-mon[37789]: pgmap v6712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:43 smithi044 ceph-mon[32404]: pgmap v6713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:43 smithi116 ceph-mon[37097]: pgmap v6713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:43 smithi171 ceph-mon[37789]: pgmap v6713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:44.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:44.595+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:57:44.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:44.595+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:57:44.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:44.595+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:45 smithi044 ceph-mon[32404]: pgmap v6714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:57:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11810.901349 secs 2024-07-26T11:57:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:45 smithi116 ceph-mon[37097]: pgmap v6714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:57:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11810.901349 secs 2024-07-26T11:57:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:45 smithi171 ceph-mon[37789]: pgmap v6714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:57:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11810.901349 secs 2024-07-26T11:57:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:47 smithi044 ceph-mon[32404]: pgmap v6715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:47 smithi116 ceph-mon[37097]: pgmap v6715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:48.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:47 smithi171 ceph-mon[37789]: pgmap v6715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:49 smithi044 ceph-mon[32404]: pgmap v6716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:49 smithi116 ceph-mon[37097]: pgmap v6716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:49 smithi171 ceph-mon[37789]: pgmap v6716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11815.901488 secs 2024-07-26T11:57:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11815.901488 secs 2024-07-26T11:57:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11815.901488 secs 2024-07-26T11:57:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:52 smithi044 ceph-mon[32404]: pgmap v6717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:57:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:52 smithi116 ceph-mon[37097]: pgmap v6717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:57:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:52 smithi171 ceph-mon[37789]: pgmap v6717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:57:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:54 smithi044 ceph-mon[32404]: pgmap v6718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:54 smithi116 ceph-mon[37097]: pgmap v6718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:54 smithi171 ceph-mon[37789]: pgmap v6718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:56 smithi044 ceph-mon[32404]: pgmap v6719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11820.901665 secs 2024-07-26T11:57:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:56 smithi116 ceph-mon[37097]: pgmap v6719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11820.901665 secs 2024-07-26T11:57:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:56 smithi171 ceph-mon[37789]: pgmap v6719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:57:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11820.901665 secs 2024-07-26T11:57:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:57:58 smithi116 ceph-mon[37097]: pgmap v6720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:57:58 smithi171 ceph-mon[37789]: pgmap v6720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:57:58 smithi044 ceph-mon[32404]: pgmap v6720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:57:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:59.606+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:57:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:59.606+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:57:59.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:57:59.606+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:00 smithi171 ceph-mon[37789]: pgmap v6721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:00 smithi044 ceph-mon[32404]: pgmap v6721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:00 smithi116 ceph-mon[37097]: pgmap v6721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11825.901802 secs 2024-07-26T11:58:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11825.901802 secs 2024-07-26T11:58:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11825.901802 secs 2024-07-26T11:58:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:02 smithi171 ceph-mon[37789]: pgmap v6722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:58:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:02 smithi044 ceph-mon[32404]: pgmap v6722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:58:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:02 smithi116 ceph-mon[37097]: pgmap v6722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:58:04.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:04 smithi044 ceph-mon[32404]: pgmap v6723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:04 smithi171 ceph-mon[37789]: pgmap v6723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:04 smithi116 ceph-mon[37097]: pgmap v6723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:06 smithi171 ceph-mon[37789]: pgmap v6724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:58:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11830.901935 secs 2024-07-26T11:58:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:06 smithi044 ceph-mon[32404]: pgmap v6724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:58:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11830.901935 secs 2024-07-26T11:58:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:06 smithi116 ceph-mon[37097]: pgmap v6724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T11:58:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11830.901935 secs 2024-07-26T11:58:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:08 smithi171 ceph-mon[37789]: pgmap v6725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:08 smithi044 ceph-mon[32404]: pgmap v6725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:08 smithi116 ceph-mon[37097]: pgmap v6725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:58:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:10 smithi171 ceph-mon[37789]: pgmap v6726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:10 smithi044 ceph-mon[32404]: pgmap v6726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:10 smithi116 ceph-mon[37097]: pgmap v6726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11835.902087 secs 2024-07-26T11:58:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11835.902087 secs 2024-07-26T11:58:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11835.902087 secs 2024-07-26T11:58:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:12 smithi171 ceph-mon[37789]: pgmap v6727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:12 smithi044 ceph-mon[32404]: pgmap v6727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:12 smithi116 ceph-mon[37097]: pgmap v6727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:14 smithi171 ceph-mon[37789]: pgmap v6728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:14 smithi044 ceph-mon[32404]: pgmap v6728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:14 smithi116 ceph-mon[37097]: pgmap v6728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:14.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:14.611+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:58:14.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:14.612+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:14.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:14.612+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:58:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:16 smithi044 ceph-mon[32404]: pgmap v6729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11840.902238 secs 2024-07-26T11:58:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:16 smithi116 ceph-mon[37097]: pgmap v6729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11840.902238 secs 2024-07-26T11:58:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:16 smithi171 ceph-mon[37789]: pgmap v6729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11840.902238 secs 2024-07-26T11:58:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:18 smithi044 ceph-mon[32404]: pgmap v6730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:18 smithi116 ceph-mon[37097]: pgmap v6730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:18 smithi171 ceph-mon[37789]: pgmap v6730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:20 smithi044 ceph-mon[32404]: pgmap v6731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:20 smithi116 ceph-mon[37097]: pgmap v6731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:20 smithi171 ceph-mon[37789]: pgmap v6731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11845.902387 secs 2024-07-26T11:58:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11845.902387 secs 2024-07-26T11:58:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11845.902387 secs 2024-07-26T11:58:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:22 smithi044 ceph-mon[32404]: pgmap v6732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:22 smithi116 ceph-mon[37097]: pgmap v6732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:22 smithi171 ceph-mon[37789]: pgmap v6732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:24 smithi044 ceph-mon[32404]: pgmap v6733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:24 smithi116 ceph-mon[37097]: pgmap v6733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:24 smithi171 ceph-mon[37789]: pgmap v6733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:26 smithi044 ceph-mon[32404]: pgmap v6734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11850.902548 secs 2024-07-26T11:58:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:26 smithi116 ceph-mon[37097]: pgmap v6734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11850.902548 secs 2024-07-26T11:58:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:26 smithi171 ceph-mon[37789]: pgmap v6734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:58:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11850.902548 secs 2024-07-26T11:58:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:28 smithi044 ceph-mon[32404]: pgmap v6735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:28 smithi116 ceph-mon[37097]: pgmap v6735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:28 smithi171 ceph-mon[37789]: pgmap v6735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:29.622+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:58:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:29.622+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:29.622+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:58:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: pgmap v6736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:30.520 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:58:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: pgmap v6736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:30.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:30.525 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:58:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: pgmap v6736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:58:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:30.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:30.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:58:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:58:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:58:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:58:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:58:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: pgmap v6737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:58:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: pgmap v6738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:31.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11855.902670 secs 2024-07-26T11:58:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:58:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:58:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:58:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:58:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: pgmap v6737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:58:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: pgmap v6738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:31.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11855.902670 secs 2024-07-26T11:58:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T11:58:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T11:58:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:58:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T11:58:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: pgmap v6737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:58:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: pgmap v6738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:31.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11855.902670 secs 2024-07-26T11:58:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:33 smithi044 ceph-mon[32404]: pgmap v6739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:33 smithi116 ceph-mon[37097]: pgmap v6739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:33 smithi171 ceph-mon[37789]: pgmap v6739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:35 smithi044 ceph-mon[32404]: pgmap v6740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:35 smithi116 ceph-mon[37097]: pgmap v6740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:35 smithi171 ceph-mon[37789]: pgmap v6740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:36.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11860.902807 secs 2024-07-26T11:58:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11860.902807 secs 2024-07-26T11:58:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11860.902807 secs 2024-07-26T11:58:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:37 smithi044 ceph-mon[32404]: pgmap v6741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:58:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:37 smithi116 ceph-mon[37097]: pgmap v6741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:58:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:37 smithi171 ceph-mon[37789]: pgmap v6741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:58:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:39 smithi044 ceph-mon[32404]: pgmap v6742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:39 smithi116 ceph-mon[37097]: pgmap v6742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:39 smithi171 ceph-mon[37789]: pgmap v6742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:58:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:41 smithi044 ceph-mon[32404]: pgmap v6743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11865.902907 secs 2024-07-26T11:58:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:41 smithi116 ceph-mon[37097]: pgmap v6743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11865.902907 secs 2024-07-26T11:58:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:41 smithi171 ceph-mon[37789]: pgmap v6743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11865.902907 secs 2024-07-26T11:58:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:43 smithi044 ceph-mon[32404]: pgmap v6744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:43 smithi116 ceph-mon[37097]: pgmap v6744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:43 smithi171 ceph-mon[37789]: pgmap v6744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:44.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:44.627+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:58:44.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:44.627+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:44.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:44.627+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:58:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:45 smithi044 ceph-mon[32404]: pgmap v6745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:45 smithi116 ceph-mon[37097]: pgmap v6745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:45 smithi171 ceph-mon[37789]: pgmap v6745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11870.903057 secs 2024-07-26T11:58:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11870.903057 secs 2024-07-26T11:58:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11870.903057 secs 2024-07-26T11:58:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:47 smithi044 ceph-mon[32404]: pgmap v6746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:47 smithi116 ceph-mon[37097]: pgmap v6746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:47 smithi171 ceph-mon[37789]: pgmap v6746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:48 smithi044 ceph-mon[32404]: pgmap v6747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:48 smithi116 ceph-mon[37097]: pgmap v6747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:48 smithi171 ceph-mon[37789]: pgmap v6747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:51 smithi044 ceph-mon[32404]: pgmap v6748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11875.903206 secs 2024-07-26T11:58:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:51 smithi116 ceph-mon[37097]: pgmap v6748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11875.903206 secs 2024-07-26T11:58:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:51 smithi171 ceph-mon[37789]: pgmap v6748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11875.903206 secs 2024-07-26T11:58:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:53 smithi044 ceph-mon[32404]: pgmap v6749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:53 smithi116 ceph-mon[37097]: pgmap v6749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:53 smithi171 ceph-mon[37789]: pgmap v6749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:55 smithi044 ceph-mon[32404]: pgmap v6750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:55 smithi116 ceph-mon[37097]: pgmap v6750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:55 smithi171 ceph-mon[37789]: pgmap v6750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:58:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11880.903371 secs 2024-07-26T11:58:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11880.903371 secs 2024-07-26T11:58:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11880.903371 secs 2024-07-26T11:58:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:57 smithi044 ceph-mon[32404]: pgmap v6751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:57 smithi116 ceph-mon[37097]: pgmap v6751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:57 smithi171 ceph-mon[37789]: pgmap v6751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:58:59 smithi044 ceph-mon[32404]: pgmap v6752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:58:59 smithi116 ceph-mon[37097]: pgmap v6752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:58:59 smithi171 ceph-mon[37789]: pgmap v6752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:59.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:59.635+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:58:59.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:59.635+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:58:59.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:58:59.635+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:01 smithi044 ceph-mon[32404]: pgmap v6753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11885.903525 secs 2024-07-26T11:59:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:01 smithi116 ceph-mon[37097]: pgmap v6753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11885.903525 secs 2024-07-26T11:59:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:01 smithi171 ceph-mon[37789]: pgmap v6753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11885.903525 secs 2024-07-26T11:59:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:03 smithi044 ceph-mon[32404]: pgmap v6754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:03 smithi116 ceph-mon[37097]: pgmap v6754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:03 smithi171 ceph-mon[37789]: pgmap v6754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:05 smithi044 ceph-mon[32404]: pgmap v6755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:05 smithi116 ceph-mon[37097]: pgmap v6755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:05 smithi171 ceph-mon[37789]: pgmap v6755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:06.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11890.903668 secs 2024-07-26T11:59:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11890.903668 secs 2024-07-26T11:59:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11890.903668 secs 2024-07-26T11:59:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:07 smithi044 ceph-mon[32404]: pgmap v6756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:07 smithi116 ceph-mon[37097]: pgmap v6756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:07 smithi171 ceph-mon[37789]: pgmap v6756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:09 smithi044 ceph-mon[32404]: pgmap v6757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:09 smithi116 ceph-mon[37097]: pgmap v6757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:09 smithi171 ceph-mon[37789]: pgmap v6757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:11 smithi044 ceph-mon[32404]: pgmap v6758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11895.903814 secs 2024-07-26T11:59:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:11 smithi116 ceph-mon[37097]: pgmap v6758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11895.903814 secs 2024-07-26T11:59:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:11 smithi171 ceph-mon[37789]: pgmap v6758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11895.903814 secs 2024-07-26T11:59:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:12 smithi044 ceph-mon[32404]: pgmap v6759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:12 smithi116 ceph-mon[37097]: pgmap v6759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:12 smithi171 ceph-mon[37789]: pgmap v6759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:14.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:14.641+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:59:14.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:14.641+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:59:14.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:14.641+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:15 smithi044 ceph-mon[32404]: pgmap v6760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:15 smithi116 ceph-mon[37097]: pgmap v6760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:15 smithi171 ceph-mon[37789]: pgmap v6760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11900.903966 secs 2024-07-26T11:59:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11900.903966 secs 2024-07-26T11:59:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11900.903966 secs 2024-07-26T11:59:16.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.814+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:59:16.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.814+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:59:16.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.815+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T11:59:16.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.815+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T11:59:16.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.815+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:59:16.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.815+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:59:16.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.816+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T11:59:16.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:16.816+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T11:59:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:17 smithi044 ceph-mon[32404]: pgmap v6761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:17 smithi116 ceph-mon[37097]: pgmap v6761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:17 smithi171 ceph-mon[37789]: pgmap v6761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:19 smithi044 ceph-mon[32404]: pgmap v6762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:19 smithi116 ceph-mon[37097]: pgmap v6762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:19 smithi171 ceph-mon[37789]: pgmap v6762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:21 smithi044 ceph-mon[32404]: pgmap v6763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11905.904138 secs 2024-07-26T11:59:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:21 smithi116 ceph-mon[37097]: pgmap v6763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11905.904138 secs 2024-07-26T11:59:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:21 smithi171 ceph-mon[37789]: pgmap v6763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11905.904138 secs 2024-07-26T11:59:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:23 smithi044 ceph-mon[32404]: pgmap v6764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:23 smithi116 ceph-mon[37097]: pgmap v6764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:23.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:23 smithi171 ceph-mon[37789]: pgmap v6764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:25 smithi044 ceph-mon[32404]: pgmap v6765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:25 smithi116 ceph-mon[37097]: pgmap v6765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:25 smithi171 ceph-mon[37789]: pgmap v6765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11910.904304 secs 2024-07-26T11:59:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11910.904304 secs 2024-07-26T11:59:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11910.904304 secs 2024-07-26T11:59:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:27 smithi044 ceph-mon[32404]: pgmap v6766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:27 smithi116 ceph-mon[37097]: pgmap v6766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:27 smithi171 ceph-mon[37789]: pgmap v6766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:29 smithi044 ceph-mon[32404]: pgmap v6767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:29 smithi116 ceph-mon[37097]: pgmap v6767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:29 smithi171 ceph-mon[37789]: pgmap v6767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:29.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:29.643+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:59:29.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:29.643+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:29.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:29.643+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:59:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:31 smithi044 ceph-mon[32404]: pgmap v6768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11915.904428 secs 2024-07-26T11:59:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:31 smithi116 ceph-mon[37097]: pgmap v6768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11915.904428 secs 2024-07-26T11:59:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:31 smithi171 ceph-mon[37789]: pgmap v6768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11915.904428 secs 2024-07-26T11:59:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:32 smithi044 ceph-mon[32404]: pgmap v6769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:59:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:32 smithi116 ceph-mon[37097]: pgmap v6769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:59:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:32 smithi171 ceph-mon[37789]: pgmap v6769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T11:59:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:34 smithi044 ceph-mon[32404]: pgmap v6770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:59:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:34 smithi116 ceph-mon[37097]: pgmap v6770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:59:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:34 smithi171 ceph-mon[37789]: pgmap v6770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:59:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:36 smithi044 ceph-mon[32404]: pgmap v6771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11920.904601 secs 2024-07-26T11:59:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:36 smithi116 ceph-mon[37097]: pgmap v6771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11920.904601 secs 2024-07-26T11:59:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:36 smithi171 ceph-mon[37789]: pgmap v6771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11920.904601 secs 2024-07-26T11:59:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:38 smithi044 ceph-mon[32404]: pgmap v6772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:59:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:38 smithi116 ceph-mon[37097]: pgmap v6772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:59:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:38 smithi171 ceph-mon[37789]: pgmap v6772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:59:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:40 smithi044 ceph-mon[32404]: pgmap v6773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:40 smithi116 ceph-mon[37097]: pgmap v6773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:40 smithi171 ceph-mon[37789]: pgmap v6773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11925.904758 secs 2024-07-26T11:59:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11925.904758 secs 2024-07-26T11:59:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11925.904758 secs 2024-07-26T11:59:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:42 smithi044 ceph-mon[32404]: pgmap v6774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:42 smithi116 ceph-mon[37097]: pgmap v6774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:42 smithi171 ceph-mon[37789]: pgmap v6774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:44.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:44.643+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:59:44.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:44.643+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:59:44.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:44.643+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:44 smithi044 ceph-mon[32404]: pgmap v6775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:44 smithi116 ceph-mon[37097]: pgmap v6775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:44 smithi171 ceph-mon[37789]: pgmap v6775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11930.904895 secs 2024-07-26T11:59:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11930.904895 secs 2024-07-26T11:59:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11930.904895 secs 2024-07-26T11:59:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:46 smithi171 ceph-mon[37789]: pgmap v6776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:46 smithi044 ceph-mon[32404]: pgmap v6776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:46 smithi116 ceph-mon[37097]: pgmap v6776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:48 smithi171 ceph-mon[37789]: pgmap v6777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:48 smithi044 ceph-mon[32404]: pgmap v6777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:48 smithi116 ceph-mon[37097]: pgmap v6777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:50 smithi171 ceph-mon[37789]: pgmap v6778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11935.905032 secs 2024-07-26T11:59:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:50 smithi044 ceph-mon[32404]: pgmap v6778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11935.905032 secs 2024-07-26T11:59:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:50 smithi116 ceph-mon[37097]: pgmap v6778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:59:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11935.905032 secs 2024-07-26T11:59:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:52 smithi171 ceph-mon[37789]: pgmap v6779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:52 smithi044 ceph-mon[32404]: pgmap v6779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:52 smithi116 ceph-mon[37097]: pgmap v6779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:55.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:54 smithi116 ceph-mon[37097]: pgmap v6780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:54 smithi171 ceph-mon[37789]: pgmap v6780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:54 smithi044 ceph-mon[32404]: pgmap v6780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11940.905182 secs 2024-07-26T11:59:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11940.905182 secs 2024-07-26T11:59:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11940.905182 secs 2024-07-26T11:59:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:56 smithi171 ceph-mon[37789]: pgmap v6781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:56 smithi044 ceph-mon[32404]: pgmap v6781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:56 smithi116 ceph-mon[37097]: pgmap v6781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 11:59:58 smithi171 ceph-mon[37789]: pgmap v6782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 11:59:58 smithi044 ceph-mon[32404]: pgmap v6782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 11:59:58 smithi116 ceph-mon[37097]: pgmap v6782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:59:59.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:59.654+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T11:59:59.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:59.654+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T11:59:59.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T11:59:59.654+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:00 smithi171 ceph-mon[37789]: pgmap v6783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:01.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11945.905326 secs 2024-07-26T12:00:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:00 smithi044 ceph-mon[32404]: pgmap v6783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11945.905326 secs 2024-07-26T12:00:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:00 smithi116 ceph-mon[37097]: pgmap v6783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:00:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11945.905326 secs 2024-07-26T12:00:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:02 smithi171 ceph-mon[37789]: pgmap v6784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:00:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:02 smithi044 ceph-mon[32404]: pgmap v6784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:00:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:02 smithi116 ceph-mon[37097]: pgmap v6784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:00:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:04 smithi044 ceph-mon[32404]: pgmap v6785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:04 smithi116 ceph-mon[37097]: pgmap v6785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:04 smithi171 ceph-mon[37789]: pgmap v6785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:06.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11950.905484 secs 2024-07-26T12:00:06.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11950.905484 secs 2024-07-26T12:00:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11950.905484 secs 2024-07-26T12:00:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:06 smithi044 ceph-mon[32404]: pgmap v6786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:06 smithi116 ceph-mon[37097]: pgmap v6786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:06 smithi171 ceph-mon[37789]: pgmap v6786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:08 smithi044 ceph-mon[32404]: pgmap v6787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:08 smithi116 ceph-mon[37097]: pgmap v6787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:08 smithi171 ceph-mon[37789]: pgmap v6787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:10 smithi044 ceph-mon[32404]: pgmap v6788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11955.905658 secs 2024-07-26T12:00:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:10 smithi116 ceph-mon[37097]: pgmap v6788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11955.905658 secs 2024-07-26T12:00:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:10 smithi171 ceph-mon[37789]: pgmap v6788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11955.905658 secs 2024-07-26T12:00:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:12 smithi044 ceph-mon[32404]: pgmap v6789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:12 smithi116 ceph-mon[37097]: pgmap v6789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:12 smithi171 ceph-mon[37789]: pgmap v6789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:14.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:14.658+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:00:14.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:14.658+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:00:14.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:14.658+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:14 smithi044 ceph-mon[32404]: pgmap v6790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:14 smithi116 ceph-mon[37097]: pgmap v6790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:14 smithi171 ceph-mon[37789]: pgmap v6790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11960.905823 secs 2024-07-26T12:00:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11960.905823 secs 2024-07-26T12:00:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11960.905823 secs 2024-07-26T12:00:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:16 smithi044 ceph-mon[32404]: pgmap v6791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:16 smithi116 ceph-mon[37097]: pgmap v6791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:16 smithi171 ceph-mon[37789]: pgmap v6791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:18 smithi044 ceph-mon[32404]: pgmap v6792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:18 smithi116 ceph-mon[37097]: pgmap v6792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:18 smithi171 ceph-mon[37789]: pgmap v6792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:20 smithi044 ceph-mon[32404]: pgmap v6793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11965.905961 secs 2024-07-26T12:00:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:20 smithi116 ceph-mon[37097]: pgmap v6793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11965.905961 secs 2024-07-26T12:00:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:20 smithi171 ceph-mon[37789]: pgmap v6793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11965.905961 secs 2024-07-26T12:00:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:22 smithi044 ceph-mon[32404]: pgmap v6794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:22 smithi116 ceph-mon[37097]: pgmap v6794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:22 smithi171 ceph-mon[37789]: pgmap v6794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:24 smithi044 ceph-mon[32404]: pgmap v6795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:24 smithi116 ceph-mon[37097]: pgmap v6795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:24 smithi171 ceph-mon[37789]: pgmap v6795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11970.906112 secs 2024-07-26T12:00:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11970.906112 secs 2024-07-26T12:00:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11970.906112 secs 2024-07-26T12:00:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:26 smithi044 ceph-mon[32404]: pgmap v6796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:26 smithi116 ceph-mon[37097]: pgmap v6796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:26 smithi171 ceph-mon[37789]: pgmap v6796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:28 smithi044 ceph-mon[32404]: pgmap v6797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:28 smithi116 ceph-mon[37097]: pgmap v6797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:28 smithi171 ceph-mon[37789]: pgmap v6797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:29.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:29.669+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:00:29.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:29.669+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:29.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:29.669+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:00:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:30 smithi044 ceph-mon[32404]: pgmap v6798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11975.906283 secs 2024-07-26T12:00:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:30 smithi116 ceph-mon[37097]: pgmap v6798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11975.906283 secs 2024-07-26T12:00:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:30 smithi171 ceph-mon[37789]: pgmap v6798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11975.906283 secs 2024-07-26T12:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:32 smithi171 ceph-mon[37789]: pgmap v6799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:32 smithi044 ceph-mon[32404]: pgmap v6799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:32 smithi116 ceph-mon[37097]: pgmap v6799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:00:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:34 smithi044 ceph-mon[32404]: pgmap v6800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:34 smithi044 ceph-mon[32404]: pgmap v6801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:35.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:34 smithi044 ceph-mon[32404]: pgmap v6802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:00:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:34 smithi116 ceph-mon[37097]: pgmap v6800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:34 smithi116 ceph-mon[37097]: pgmap v6801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:34 smithi116 ceph-mon[37097]: pgmap v6802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:00:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:34 smithi171 ceph-mon[37789]: pgmap v6800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:34 smithi171 ceph-mon[37789]: pgmap v6801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:34 smithi171 ceph-mon[37789]: pgmap v6802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:00:36.237 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11980.906454 secs 2024-07-26T12:00:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11980.906454 secs 2024-07-26T12:00:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11980.906454 secs 2024-07-26T12:00:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:36 smithi044 ceph-mon[32404]: pgmap v6803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:00:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:36 smithi116 ceph-mon[37097]: pgmap v6803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:00:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:36 smithi171 ceph-mon[37789]: pgmap v6803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:00:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:38 smithi044 ceph-mon[32404]: pgmap v6804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:38 smithi116 ceph-mon[37097]: pgmap v6804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:38 smithi171 ceph-mon[37789]: pgmap v6804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:40 smithi044 ceph-mon[32404]: pgmap v6805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11985.906649 secs 2024-07-26T12:00:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:40 smithi116 ceph-mon[37097]: pgmap v6805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11985.906649 secs 2024-07-26T12:00:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:40 smithi171 ceph-mon[37789]: pgmap v6805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11985.906649 secs 2024-07-26T12:00:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:43 smithi044 ceph-mon[32404]: pgmap v6806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:00:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:43 smithi116 ceph-mon[37097]: pgmap v6806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:00:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:43 smithi171 ceph-mon[37789]: pgmap v6806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:00:44.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:44.671+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:00:44.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:44.671+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:00:44.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:44.671+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:45 smithi044 ceph-mon[32404]: pgmap v6807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:45 smithi116 ceph-mon[37097]: pgmap v6807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:45 smithi171 ceph-mon[37789]: pgmap v6807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11990.906748 secs 2024-07-26T12:00:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11990.906748 secs 2024-07-26T12:00:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11990.906748 secs 2024-07-26T12:00:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:47 smithi171 ceph-mon[37789]: pgmap v6808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:47 smithi044 ceph-mon[32404]: pgmap v6808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:47 smithi116 ceph-mon[37097]: pgmap v6808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:49 smithi171 ceph-mon[37789]: pgmap v6809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:49 smithi044 ceph-mon[32404]: pgmap v6809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:49 smithi116 ceph-mon[37097]: pgmap v6809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:51 smithi171 ceph-mon[37789]: pgmap v6810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 11995.906883 secs 2024-07-26T12:00:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:51 smithi044 ceph-mon[32404]: pgmap v6810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 11995.906883 secs 2024-07-26T12:00:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:51 smithi116 ceph-mon[37097]: pgmap v6810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:00:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 11995.906883 secs 2024-07-26T12:00:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:53 smithi171 ceph-mon[37789]: pgmap v6811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:53 smithi044 ceph-mon[32404]: pgmap v6811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:53 smithi116 ceph-mon[37097]: pgmap v6811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:55.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:55 smithi116 ceph-mon[37097]: pgmap v6812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:55 smithi171 ceph-mon[37789]: pgmap v6812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:55 smithi044 ceph-mon[32404]: pgmap v6812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12000.907034 secs 2024-07-26T12:00:56.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12000.907034 secs 2024-07-26T12:00:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12000.907034 secs 2024-07-26T12:00:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:57 smithi171 ceph-mon[37789]: pgmap v6813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:57 smithi044 ceph-mon[32404]: pgmap v6813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:57 smithi116 ceph-mon[37097]: pgmap v6813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:00:59 smithi171 ceph-mon[37789]: pgmap v6814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:00:59 smithi044 ceph-mon[32404]: pgmap v6814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:00:59 smithi116 ceph-mon[37097]: pgmap v6814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:00:59.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:59.681+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:00:59.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:59.681+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:59.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:00:59.681+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:01:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:01 smithi171 ceph-mon[37789]: pgmap v6815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12005.907161 secs 2024-07-26T12:01:01.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:01 smithi116 ceph-mon[37097]: pgmap v6815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:01.452 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12005.907161 secs 2024-07-26T12:01:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:01 smithi044 ceph-mon[32404]: pgmap v6815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12005.907161 secs 2024-07-26T12:01:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:03 smithi171 ceph-mon[37789]: pgmap v6816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:03 smithi044 ceph-mon[32404]: pgmap v6816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:03 smithi116 ceph-mon[37097]: pgmap v6816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:05 smithi171 ceph-mon[37789]: pgmap v6817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:05 smithi044 ceph-mon[32404]: pgmap v6817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:05 smithi116 ceph-mon[37097]: pgmap v6817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:06.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12010.907315 secs 2024-07-26T12:01:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12010.907315 secs 2024-07-26T12:01:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12010.907315 secs 2024-07-26T12:01:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:07 smithi044 ceph-mon[32404]: pgmap v6818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:07 smithi116 ceph-mon[37097]: pgmap v6818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:07 smithi171 ceph-mon[37789]: pgmap v6818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:09 smithi044 ceph-mon[32404]: pgmap v6819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:09 smithi116 ceph-mon[37097]: pgmap v6819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:09 smithi171 ceph-mon[37789]: pgmap v6819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:11 smithi044 ceph-mon[32404]: pgmap v6820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12015.907468 secs 2024-07-26T12:01:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:11 smithi116 ceph-mon[37097]: pgmap v6820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12015.907468 secs 2024-07-26T12:01:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:11 smithi171 ceph-mon[37789]: pgmap v6820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12015.907468 secs 2024-07-26T12:01:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:13 smithi044 ceph-mon[32404]: pgmap v6821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:13 smithi116 ceph-mon[37097]: pgmap v6821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:13 smithi171 ceph-mon[37789]: pgmap v6821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:14.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:14.686+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:01:14.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:14.686+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:01:14.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:14.686+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:15 smithi044 ceph-mon[32404]: pgmap v6822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:15 smithi116 ceph-mon[37097]: pgmap v6822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:15 smithi171 ceph-mon[37789]: pgmap v6822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12020.907645 secs 2024-07-26T12:01:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12020.907645 secs 2024-07-26T12:01:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12020.907645 secs 2024-07-26T12:01:17.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:17 smithi044 ceph-mon[32404]: pgmap v6823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:17 smithi116 ceph-mon[37097]: pgmap v6823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:17 smithi171 ceph-mon[37789]: pgmap v6823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:19 smithi044 ceph-mon[32404]: pgmap v6824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:19 smithi116 ceph-mon[37097]: pgmap v6824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:19 smithi171 ceph-mon[37789]: pgmap v6824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:21 smithi044 ceph-mon[32404]: pgmap v6825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12025.907778 secs 2024-07-26T12:01:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:21 smithi116 ceph-mon[37097]: pgmap v6825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12025.907778 secs 2024-07-26T12:01:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:21 smithi171 ceph-mon[37789]: pgmap v6825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12025.907778 secs 2024-07-26T12:01:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:23 smithi044 ceph-mon[32404]: pgmap v6826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:23 smithi116 ceph-mon[37097]: pgmap v6826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:23 smithi171 ceph-mon[37789]: pgmap v6826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:25 smithi044 ceph-mon[32404]: pgmap v6827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:25 smithi116 ceph-mon[37097]: pgmap v6827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:25 smithi171 ceph-mon[37789]: pgmap v6827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12030.907920 secs 2024-07-26T12:01:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12030.907920 secs 2024-07-26T12:01:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12030.907920 secs 2024-07-26T12:01:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:27 smithi044 ceph-mon[32404]: pgmap v6828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:27 smithi116 ceph-mon[37097]: pgmap v6828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:27 smithi171 ceph-mon[37789]: pgmap v6828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:29 smithi044 ceph-mon[32404]: pgmap v6829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:29 smithi116 ceph-mon[37097]: pgmap v6829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:29 smithi171 ceph-mon[37789]: pgmap v6829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:29.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:29.693+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:01:29.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:29.693+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:01:29.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:29.693+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:31 smithi044 ceph-mon[32404]: pgmap v6830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12035.908111 secs 2024-07-26T12:01:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:31 smithi116 ceph-mon[37097]: pgmap v6830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12035.908111 secs 2024-07-26T12:01:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:31 smithi171 ceph-mon[37789]: pgmap v6830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12035.908111 secs 2024-07-26T12:01:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:33 smithi044 ceph-mon[32404]: pgmap v6831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:33 smithi116 ceph-mon[37097]: pgmap v6831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:33 smithi171 ceph-mon[37789]: pgmap v6831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:35 smithi044 ceph-mon[32404]: pgmap v6832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:35 smithi116 ceph-mon[37097]: pgmap v6832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:35 smithi171 ceph-mon[37789]: pgmap v6832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: pgmap v6833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12040.908231 secs 2024-07-26T12:01:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: pgmap v6833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12040.908231 secs 2024-07-26T12:01:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: pgmap v6833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12040.908231 secs 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:37.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:38.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:39 smithi044 ceph-mon[32404]: pgmap v6834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:39.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:39.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:39 smithi116 ceph-mon[37097]: pgmap v6834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:39.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:39 smithi171 ceph-mon[37789]: pgmap v6834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:39.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:01:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:40 smithi044 ceph-mon[32404]: pgmap v6835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:40 smithi044 ceph-mon[32404]: pgmap v6836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:40 smithi116 ceph-mon[37097]: pgmap v6835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:40 smithi116 ceph-mon[37097]: pgmap v6836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:40 smithi171 ceph-mon[37789]: pgmap v6835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:40 smithi171 ceph-mon[37789]: pgmap v6836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12045.908373 secs 2024-07-26T12:01:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12045.908373 secs 2024-07-26T12:01:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12045.908373 secs 2024-07-26T12:01:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:42 smithi044 ceph-mon[32404]: pgmap v6837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:42 smithi116 ceph-mon[37097]: pgmap v6837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:42 smithi171 ceph-mon[37789]: pgmap v6837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:44 smithi044 ceph-mon[32404]: pgmap v6838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:44 smithi116 ceph-mon[37097]: pgmap v6838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:44 smithi171 ceph-mon[37789]: pgmap v6838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:44.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:44.698+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:01:44.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:44.698+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:44.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:44.698+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:01:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:45 smithi044 ceph-mon[32404]: pgmap v6839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:45 smithi116 ceph-mon[37097]: pgmap v6839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:45 smithi171 ceph-mon[37789]: pgmap v6839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:01:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12050.908495 secs 2024-07-26T12:01:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12050.908495 secs 2024-07-26T12:01:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12050.908495 secs 2024-07-26T12:01:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:47 smithi171 ceph-mon[37789]: pgmap v6840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:47 smithi044 ceph-mon[32404]: pgmap v6840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:47 smithi116 ceph-mon[37097]: pgmap v6840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:49 smithi044 ceph-mon[32404]: pgmap v6841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:49 smithi116 ceph-mon[37097]: pgmap v6841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:49 smithi171 ceph-mon[37789]: pgmap v6841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12055.908662 secs 2024-07-26T12:01:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12055.908662 secs 2024-07-26T12:01:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12055.908662 secs 2024-07-26T12:01:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:51 smithi044 ceph-mon[32404]: pgmap v6842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:51 smithi116 ceph-mon[37097]: pgmap v6842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:51 smithi171 ceph-mon[37789]: pgmap v6842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:53 smithi044 ceph-mon[32404]: pgmap v6843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:53 smithi116 ceph-mon[37097]: pgmap v6843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:53 smithi171 ceph-mon[37789]: pgmap v6843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:55 smithi044 ceph-mon[32404]: pgmap v6844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12060.908805 secs 2024-07-26T12:01:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:55 smithi116 ceph-mon[37097]: pgmap v6844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12060.908805 secs 2024-07-26T12:01:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:55 smithi171 ceph-mon[37789]: pgmap v6844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12060.908805 secs 2024-07-26T12:01:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:57 smithi044 ceph-mon[32404]: pgmap v6845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:57 smithi116 ceph-mon[37097]: pgmap v6845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:57 smithi171 ceph-mon[37789]: pgmap v6845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:01:59.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:59.707+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:01:59.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:59.707+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:01:59.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:01:59.707+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:01:59 smithi044 ceph-mon[32404]: pgmap v6846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:01:59 smithi116 ceph-mon[37097]: pgmap v6846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:01:59 smithi171 ceph-mon[37789]: pgmap v6846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12065.908950 secs 2024-07-26T12:02:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12065.908950 secs 2024-07-26T12:02:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12065.908950 secs 2024-07-26T12:02:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:01 smithi044 ceph-mon[32404]: pgmap v6847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:01 smithi116 ceph-mon[37097]: pgmap v6847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:01 smithi171 ceph-mon[37789]: pgmap v6847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:03 smithi044 ceph-mon[32404]: pgmap v6848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:03 smithi116 ceph-mon[37097]: pgmap v6848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:03 smithi171 ceph-mon[37789]: pgmap v6848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:05 smithi044 ceph-mon[32404]: pgmap v6849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12070.909092 secs 2024-07-26T12:02:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:05 smithi116 ceph-mon[37097]: pgmap v6849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12070.909092 secs 2024-07-26T12:02:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:05 smithi171 ceph-mon[37789]: pgmap v6849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12070.909092 secs 2024-07-26T12:02:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:07 smithi044 ceph-mon[32404]: pgmap v6850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:07 smithi116 ceph-mon[37097]: pgmap v6850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:07 smithi171 ceph-mon[37789]: pgmap v6850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:09 smithi044 ceph-mon[32404]: pgmap v6851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:09 smithi116 ceph-mon[37097]: pgmap v6851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:09 smithi171 ceph-mon[37789]: pgmap v6851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12075.909229 secs 2024-07-26T12:02:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12075.909229 secs 2024-07-26T12:02:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12075.909229 secs 2024-07-26T12:02:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:12 smithi044 ceph-mon[32404]: pgmap v6852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:12 smithi116 ceph-mon[37097]: pgmap v6852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:12 smithi171 ceph-mon[37789]: pgmap v6852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:14 smithi044 ceph-mon[32404]: pgmap v6853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:14 smithi116 ceph-mon[37097]: pgmap v6853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:14 smithi171 ceph-mon[37789]: pgmap v6853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:14.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:14.713+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:02:14.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:14.713+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:14.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:14.713+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:02:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:16 smithi044 ceph-mon[32404]: pgmap v6854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12080.909372 secs 2024-07-26T12:02:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:16 smithi116 ceph-mon[37097]: pgmap v6854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12080.909372 secs 2024-07-26T12:02:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:16 smithi171 ceph-mon[37789]: pgmap v6854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12080.909372 secs 2024-07-26T12:02:16.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:16.847+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:02:16.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:16.847+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T12:02:16.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:16.848+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003710 con 0x7fcf6c1057b0 2024-07-26T12:02:16.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:16.848+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:02:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:18 smithi116 ceph-mon[37097]: pgmap v6855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:18 smithi171 ceph-mon[37789]: pgmap v6855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:18 smithi044 ceph-mon[32404]: pgmap v6855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:20 smithi171 ceph-mon[37789]: pgmap v6856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:20 smithi044 ceph-mon[32404]: pgmap v6856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:20 smithi116 ceph-mon[37097]: pgmap v6856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12085.909524 secs 2024-07-26T12:02:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12085.909524 secs 2024-07-26T12:02:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12085.909524 secs 2024-07-26T12:02:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:22 smithi171 ceph-mon[37789]: pgmap v6857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:22 smithi044 ceph-mon[32404]: pgmap v6857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:22 smithi116 ceph-mon[37097]: pgmap v6857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:24 smithi171 ceph-mon[37789]: pgmap v6858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:24 smithi044 ceph-mon[32404]: pgmap v6858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:24 smithi116 ceph-mon[37097]: pgmap v6858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:26.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:26 smithi116 ceph-mon[37097]: pgmap v6859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:26.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12090.909683 secs 2024-07-26T12:02:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:26 smithi171 ceph-mon[37789]: pgmap v6859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12090.909683 secs 2024-07-26T12:02:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:26 smithi044 ceph-mon[32404]: pgmap v6859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12090.909683 secs 2024-07-26T12:02:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:28 smithi171 ceph-mon[37789]: pgmap v6860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:28 smithi044 ceph-mon[32404]: pgmap v6860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:28 smithi116 ceph-mon[37097]: pgmap v6860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:29.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:29.724+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:02:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:29.724+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:29.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:29.724+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:02:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:30 smithi171 ceph-mon[37789]: pgmap v6861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:30 smithi044 ceph-mon[32404]: pgmap v6861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:30 smithi116 ceph-mon[37097]: pgmap v6861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12095.909819 secs 2024-07-26T12:02:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12095.909819 secs 2024-07-26T12:02:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12095.909819 secs 2024-07-26T12:02:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:32 smithi044 ceph-mon[32404]: pgmap v6862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:32 smithi171 ceph-mon[37789]: pgmap v6862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:32 smithi116 ceph-mon[37097]: pgmap v6862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:34 smithi044 ceph-mon[32404]: pgmap v6863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:34 smithi116 ceph-mon[37097]: pgmap v6863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:34 smithi171 ceph-mon[37789]: pgmap v6863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:36 smithi044 ceph-mon[32404]: pgmap v6864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12100.909985 secs 2024-07-26T12:02:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:36 smithi116 ceph-mon[37097]: pgmap v6864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12100.909985 secs 2024-07-26T12:02:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:36 smithi171 ceph-mon[37789]: pgmap v6864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12100.909985 secs 2024-07-26T12:02:38.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:38 smithi171 ceph-mon[37789]: pgmap v6865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:02:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:38 smithi044 ceph-mon[32404]: pgmap v6865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:02:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:38 smithi116 ceph-mon[37097]: pgmap v6865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:02:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:40 smithi044 ceph-mon[32404]: pgmap v6866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:02:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:40 smithi116 ceph-mon[37097]: pgmap v6866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:02:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:40 smithi171 ceph-mon[37789]: pgmap v6866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:02:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:41 smithi044 ceph-mon[32404]: pgmap v6867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:02:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:41 smithi044 ceph-mon[32404]: pgmap v6868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:02:41.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12105.910148 secs 2024-07-26T12:02:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:41 smithi116 ceph-mon[37097]: pgmap v6867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:02:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:41 smithi116 ceph-mon[37097]: pgmap v6868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:02:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12105.910148 secs 2024-07-26T12:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:41 smithi171 ceph-mon[37789]: pgmap v6867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:41 smithi171 ceph-mon[37789]: pgmap v6868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:02:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12105.910148 secs 2024-07-26T12:02:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:43 smithi044 ceph-mon[32404]: pgmap v6869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T12:02:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:43 smithi116 ceph-mon[37097]: pgmap v6869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T12:02:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:43 smithi171 ceph-mon[37789]: pgmap v6869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T12:02:44.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:44.729+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:02:44.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:44.729+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:44.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:44.729+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:02:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:45 smithi044 ceph-mon[32404]: pgmap v6870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:45 smithi116 ceph-mon[37097]: pgmap v6870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:45 smithi171 ceph-mon[37789]: pgmap v6870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12110.910286 secs 2024-07-26T12:02:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12110.910286 secs 2024-07-26T12:02:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12110.910286 secs 2024-07-26T12:02:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:47 smithi116 ceph-mon[37097]: pgmap v6871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:02:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:47 smithi171 ceph-mon[37789]: pgmap v6871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:02:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:47 smithi044 ceph-mon[32404]: pgmap v6871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:02:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:48 smithi171 ceph-mon[37789]: pgmap v6872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:48 smithi044 ceph-mon[32404]: pgmap v6872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:48 smithi116 ceph-mon[37097]: pgmap v6872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:51 smithi044 ceph-mon[32404]: pgmap v6873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12115.910457 secs 2024-07-26T12:02:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:51 smithi116 ceph-mon[37097]: pgmap v6873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12115.910457 secs 2024-07-26T12:02:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:51 smithi171 ceph-mon[37789]: pgmap v6873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12115.910457 secs 2024-07-26T12:02:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:53 smithi044 ceph-mon[32404]: pgmap v6874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:53 smithi116 ceph-mon[37097]: pgmap v6874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:53 smithi171 ceph-mon[37789]: pgmap v6874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:02:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:55 smithi044 ceph-mon[32404]: pgmap v6875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:55 smithi116 ceph-mon[37097]: pgmap v6875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:55 smithi171 ceph-mon[37789]: pgmap v6875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:02:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12120.910668 secs 2024-07-26T12:02:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12120.910668 secs 2024-07-26T12:02:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12120.910668 secs 2024-07-26T12:02:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:57 smithi171 ceph-mon[37789]: pgmap v6876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:57 smithi044 ceph-mon[32404]: pgmap v6876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:57 smithi116 ceph-mon[37097]: pgmap v6876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:02:58 smithi171 ceph-mon[37789]: pgmap v6877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:02:58 smithi044 ceph-mon[32404]: pgmap v6877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:02:58 smithi116 ceph-mon[37097]: pgmap v6877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:59.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:59.739+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:02:59.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:59.739+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:59.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:02:59.739+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:03:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:01 smithi044 ceph-mon[32404]: pgmap v6878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12125.910771 secs 2024-07-26T12:03:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:01 smithi116 ceph-mon[37097]: pgmap v6878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12125.910771 secs 2024-07-26T12:03:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:01 smithi171 ceph-mon[37789]: pgmap v6878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12125.910771 secs 2024-07-26T12:03:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:03 smithi044 ceph-mon[32404]: pgmap v6879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:03 smithi116 ceph-mon[37097]: pgmap v6879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:03.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:03 smithi171 ceph-mon[37789]: pgmap v6879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:05 smithi116 ceph-mon[37097]: pgmap v6880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:05 smithi171 ceph-mon[37789]: pgmap v6880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:05 smithi044 ceph-mon[32404]: pgmap v6880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12130.910922 secs 2024-07-26T12:03:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:06 smithi171 ceph-mon[37789]: pgmap v6881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12130.910922 secs 2024-07-26T12:03:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:06 smithi044 ceph-mon[32404]: pgmap v6881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12130.910922 secs 2024-07-26T12:03:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:06 smithi116 ceph-mon[37097]: pgmap v6881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:09 smithi044 ceph-mon[32404]: pgmap v6882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:09 smithi116 ceph-mon[37097]: pgmap v6882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:09 smithi171 ceph-mon[37789]: pgmap v6882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:11 smithi116 ceph-mon[37097]: pgmap v6883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12135.911068 secs 2024-07-26T12:03:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:11 smithi171 ceph-mon[37789]: pgmap v6883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12135.911068 secs 2024-07-26T12:03:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:11 smithi044 ceph-mon[32404]: pgmap v6883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12135.911068 secs 2024-07-26T12:03:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:12 smithi171 ceph-mon[37789]: pgmap v6884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:12 smithi044 ceph-mon[32404]: pgmap v6884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:12 smithi116 ceph-mon[37097]: pgmap v6884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:14.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:14.745+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:03:14.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:14.745+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:14.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:14.745+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:03:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:15 smithi116 ceph-mon[37097]: pgmap v6885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:15 smithi171 ceph-mon[37789]: pgmap v6885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:15 smithi044 ceph-mon[32404]: pgmap v6885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12140.911237 secs 2024-07-26T12:03:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12140.911237 secs 2024-07-26T12:03:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12140.911237 secs 2024-07-26T12:03:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:17 smithi171 ceph-mon[37789]: pgmap v6886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:17 smithi044 ceph-mon[32404]: pgmap v6886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:17 smithi116 ceph-mon[37097]: pgmap v6886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:18 smithi171 ceph-mon[37789]: pgmap v6887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:18 smithi044 ceph-mon[32404]: pgmap v6887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:18 smithi116 ceph-mon[37097]: pgmap v6887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:21 smithi116 ceph-mon[37097]: pgmap v6888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12145.911374 secs 2024-07-26T12:03:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:21 smithi171 ceph-mon[37789]: pgmap v6888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12145.911374 secs 2024-07-26T12:03:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:21 smithi044 ceph-mon[32404]: pgmap v6888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12145.911374 secs 2024-07-26T12:03:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:22 smithi171 ceph-mon[37789]: pgmap v6889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:22 smithi044 ceph-mon[32404]: pgmap v6889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:22 smithi116 ceph-mon[37097]: pgmap v6889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:24 smithi171 ceph-mon[37789]: pgmap v6890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:24 smithi044 ceph-mon[32404]: pgmap v6890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:24 smithi116 ceph-mon[37097]: pgmap v6890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:03:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12150.911517 secs 2024-07-26T12:03:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12150.911517 secs 2024-07-26T12:03:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12150.911517 secs 2024-07-26T12:03:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:26 smithi044 ceph-mon[32404]: pgmap v6891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:26 smithi116 ceph-mon[37097]: pgmap v6891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:26 smithi171 ceph-mon[37789]: pgmap v6891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:28 smithi171 ceph-mon[37789]: pgmap v6892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:28 smithi044 ceph-mon[32404]: pgmap v6892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:28 smithi116 ceph-mon[37097]: pgmap v6892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:29.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:29.755+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:03:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:29.755+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:29.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:29.755+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:03:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:30 smithi171 ceph-mon[37789]: pgmap v6893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:30 smithi044 ceph-mon[32404]: pgmap v6893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:30 smithi116 ceph-mon[37097]: pgmap v6893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12155.911681 secs 2024-07-26T12:03:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12155.911681 secs 2024-07-26T12:03:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12155.911681 secs 2024-07-26T12:03:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:32 smithi171 ceph-mon[37789]: pgmap v6894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:32 smithi044 ceph-mon[32404]: pgmap v6894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:32 smithi116 ceph-mon[37097]: pgmap v6894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:03:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:34 smithi171 ceph-mon[37789]: pgmap v6895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:34 smithi044 ceph-mon[32404]: pgmap v6895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:34 smithi116 ceph-mon[37097]: pgmap v6895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12160.911842 secs 2024-07-26T12:03:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12160.911842 secs 2024-07-26T12:03:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12160.911842 secs 2024-07-26T12:03:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:36 smithi044 ceph-mon[32404]: pgmap v6896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:03:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:36 smithi116 ceph-mon[37097]: pgmap v6896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:03:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:36 smithi171 ceph-mon[37789]: pgmap v6896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:03:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:38 smithi171 ceph-mon[37789]: pgmap v6897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:03:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:38 smithi044 ceph-mon[32404]: pgmap v6897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:03:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:38 smithi116 ceph-mon[37097]: pgmap v6897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:03:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:40 smithi171 ceph-mon[37789]: pgmap v6898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:03:40.636 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:40 smithi044 ceph-mon[32404]: pgmap v6898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:03:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:40 smithi116 ceph-mon[37097]: pgmap v6898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:03:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12165.911942 secs 2024-07-26T12:03:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12165.911942 secs 2024-07-26T12:03:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12165.911942 secs 2024-07-26T12:03:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:42 smithi044 ceph-mon[32404]: pgmap v6899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:03:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:42 smithi044 ceph-mon[32404]: pgmap v6900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:03:42.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:03:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:42 smithi116 ceph-mon[37097]: pgmap v6899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:03:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:42 smithi116 ceph-mon[37097]: pgmap v6900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:03:42.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:03:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:42 smithi171 ceph-mon[37789]: pgmap v6899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:03:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:42 smithi171 ceph-mon[37789]: pgmap v6900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:03:42.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:03:44.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:44 smithi044 ceph-mon[32404]: pgmap v6901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:44.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:44.761+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:03:44.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:44.761+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:44.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:44.761+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:03:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:44 smithi116 ceph-mon[37097]: pgmap v6901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:44 smithi171 ceph-mon[37789]: pgmap v6901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:46 smithi044 ceph-mon[32404]: pgmap v6902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12170.912101 secs 2024-07-26T12:03:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:46 smithi116 ceph-mon[37097]: pgmap v6902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12170.912101 secs 2024-07-26T12:03:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:46 smithi171 ceph-mon[37789]: pgmap v6902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12170.912101 secs 2024-07-26T12:03:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:48 smithi044 ceph-mon[32404]: pgmap v6903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:48 smithi116 ceph-mon[37097]: pgmap v6903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:48 smithi171 ceph-mon[37789]: pgmap v6903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:50 smithi044 ceph-mon[32404]: pgmap v6904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:50 smithi116 ceph-mon[37097]: pgmap v6904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:50 smithi171 ceph-mon[37789]: pgmap v6904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:03:51.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12175.912240 secs 2024-07-26T12:03:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12175.912240 secs 2024-07-26T12:03:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12175.912240 secs 2024-07-26T12:03:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:52 smithi044 ceph-mon[32404]: pgmap v6905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:03:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:52 smithi116 ceph-mon[37097]: pgmap v6905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:03:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:52 smithi171 ceph-mon[37789]: pgmap v6905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:03:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:54 smithi044 ceph-mon[32404]: pgmap v6906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:54 smithi116 ceph-mon[37097]: pgmap v6906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:54 smithi171 ceph-mon[37789]: pgmap v6906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:03:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:56 smithi044 ceph-mon[32404]: pgmap v6907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12180.912409 secs 2024-07-26T12:03:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:56 smithi116 ceph-mon[37097]: pgmap v6907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12180.912409 secs 2024-07-26T12:03:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:56 smithi171 ceph-mon[37789]: pgmap v6907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12180.912409 secs 2024-07-26T12:03:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:03:58 smithi116 ceph-mon[37097]: pgmap v6908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:03:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:03:58 smithi171 ceph-mon[37789]: pgmap v6908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:03:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:03:58 smithi044 ceph-mon[32404]: pgmap v6908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:03:59.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:59.772+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:03:59.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:59.772+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:03:59.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:03:59.772+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:00 smithi116 ceph-mon[37097]: pgmap v6909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:04:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:00 smithi171 ceph-mon[37789]: pgmap v6909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:04:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:00 smithi044 ceph-mon[32404]: pgmap v6909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:04:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12185.912577 secs 2024-07-26T12:04:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12185.912577 secs 2024-07-26T12:04:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12185.912577 secs 2024-07-26T12:04:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:02 smithi171 ceph-mon[37789]: pgmap v6910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:02 smithi044 ceph-mon[32404]: pgmap v6910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:02 smithi116 ceph-mon[37097]: pgmap v6910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:04 smithi171 ceph-mon[37789]: pgmap v6911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:04:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:04 smithi044 ceph-mon[32404]: pgmap v6911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:04:05.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:04 smithi116 ceph-mon[37097]: pgmap v6911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:04:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:06 smithi171 ceph-mon[37789]: pgmap v6912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12190.912749 secs 2024-07-26T12:04:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:06 smithi044 ceph-mon[32404]: pgmap v6912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12190.912749 secs 2024-07-26T12:04:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:06 smithi116 ceph-mon[37097]: pgmap v6912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12190.912749 secs 2024-07-26T12:04:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:08 smithi171 ceph-mon[37789]: pgmap v6913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:04:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:08 smithi044 ceph-mon[32404]: pgmap v6913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:04:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:08 smithi116 ceph-mon[37097]: pgmap v6913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:04:10.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:10 smithi044 ceph-mon[32404]: pgmap v6914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:10 smithi171 ceph-mon[37789]: pgmap v6914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:10 smithi116 ceph-mon[37097]: pgmap v6914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12195.912851 secs 2024-07-26T12:04:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12195.912851 secs 2024-07-26T12:04:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12195.912851 secs 2024-07-26T12:04:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:12 smithi171 ceph-mon[37789]: pgmap v6915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:12 smithi044 ceph-mon[32404]: pgmap v6915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:12 smithi116 ceph-mon[37097]: pgmap v6915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:14.777+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:04:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:14.777+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:04:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:14.777+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:14 smithi171 ceph-mon[37789]: pgmap v6916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:14 smithi044 ceph-mon[32404]: pgmap v6916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:14 smithi116 ceph-mon[37097]: pgmap v6916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:16 smithi171 ceph-mon[37789]: pgmap v6917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12200.912996 secs 2024-07-26T12:04:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:16 smithi044 ceph-mon[32404]: pgmap v6917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12200.912996 secs 2024-07-26T12:04:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:16 smithi116 ceph-mon[37097]: pgmap v6917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12200.912996 secs 2024-07-26T12:04:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:18 smithi044 ceph-mon[32404]: pgmap v6918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:18 smithi116 ceph-mon[37097]: pgmap v6918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:18 smithi171 ceph-mon[37789]: pgmap v6918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:20 smithi044 ceph-mon[32404]: pgmap v6919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:20 smithi116 ceph-mon[37097]: pgmap v6919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:20 smithi171 ceph-mon[37789]: pgmap v6919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12205.913137 secs 2024-07-26T12:04:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12205.913137 secs 2024-07-26T12:04:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12205.913137 secs 2024-07-26T12:04:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:22 smithi044 ceph-mon[32404]: pgmap v6920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:22 smithi116 ceph-mon[37097]: pgmap v6920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:22 smithi171 ceph-mon[37789]: pgmap v6920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:24 smithi044 ceph-mon[32404]: pgmap v6921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:24 smithi116 ceph-mon[37097]: pgmap v6921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:24 smithi171 ceph-mon[37789]: pgmap v6921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:26 smithi044 ceph-mon[32404]: pgmap v6922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12210.913307 secs 2024-07-26T12:04:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:26 smithi116 ceph-mon[37097]: pgmap v6922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12210.913307 secs 2024-07-26T12:04:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:26 smithi171 ceph-mon[37789]: pgmap v6922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12210.913307 secs 2024-07-26T12:04:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:28 smithi044 ceph-mon[32404]: pgmap v6923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:28 smithi116 ceph-mon[37097]: pgmap v6923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:28 smithi171 ceph-mon[37789]: pgmap v6923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:29.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:29.788+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:04:29.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:29.788+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:04:29.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:29.788+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:30 smithi044 ceph-mon[32404]: pgmap v6924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:30 smithi116 ceph-mon[37097]: pgmap v6924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:30 smithi171 ceph-mon[37789]: pgmap v6924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12215.913468 secs 2024-07-26T12:04:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12215.913468 secs 2024-07-26T12:04:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12215.913468 secs 2024-07-26T12:04:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:32 smithi044 ceph-mon[32404]: pgmap v6925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:32 smithi116 ceph-mon[37097]: pgmap v6925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:32 smithi171 ceph-mon[37789]: pgmap v6925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:34 smithi044 ceph-mon[32404]: pgmap v6926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:34 smithi116 ceph-mon[37097]: pgmap v6926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:34 smithi171 ceph-mon[37789]: pgmap v6926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:36 smithi044 ceph-mon[32404]: pgmap v6927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12220.913663 secs 2024-07-26T12:04:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:36 smithi116 ceph-mon[37097]: pgmap v6927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12220.913663 secs 2024-07-26T12:04:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:36 smithi171 ceph-mon[37789]: pgmap v6927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12220.913663 secs 2024-07-26T12:04:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:38 smithi044 ceph-mon[32404]: pgmap v6928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:38 smithi116 ceph-mon[37097]: pgmap v6928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:38 smithi171 ceph-mon[37789]: pgmap v6928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:40 smithi044 ceph-mon[32404]: pgmap v6929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:40 smithi116 ceph-mon[37097]: pgmap v6929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:40 smithi171 ceph-mon[37789]: pgmap v6929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12225.913824 secs 2024-07-26T12:04:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12225.913824 secs 2024-07-26T12:04:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12225.913824 secs 2024-07-26T12:04:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:42 smithi044 ceph-mon[32404]: pgmap v6930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:04:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:42 smithi116 ceph-mon[37097]: pgmap v6930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:04:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:42 smithi171 ceph-mon[37789]: pgmap v6930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:04:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:43 smithi044 ceph-mon[32404]: pgmap v6931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:04:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:43 smithi044 ceph-mon[32404]: pgmap v6932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:43 smithi116 ceph-mon[37097]: pgmap v6931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:04:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:43 smithi116 ceph-mon[37097]: pgmap v6932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:43 smithi171 ceph-mon[37789]: pgmap v6931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:04:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:43 smithi171 ceph-mon[37789]: pgmap v6932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:44.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:44.794+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:04:44.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:44.794+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:04:44.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:44.794+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:45 smithi044 ceph-mon[32404]: pgmap v6933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:45 smithi116 ceph-mon[37097]: pgmap v6933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:45 smithi171 ceph-mon[37789]: pgmap v6933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12230.913960 secs 2024-07-26T12:04:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12230.913960 secs 2024-07-26T12:04:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12230.913960 secs 2024-07-26T12:04:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:47 smithi044 ceph-mon[32404]: pgmap v6934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:47 smithi116 ceph-mon[37097]: pgmap v6934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:47 smithi171 ceph-mon[37789]: pgmap v6934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:49 smithi044 ceph-mon[32404]: pgmap v6935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:49 smithi116 ceph-mon[37097]: pgmap v6935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:49 smithi171 ceph-mon[37789]: pgmap v6935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:04:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12235.914098 secs 2024-07-26T12:04:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12235.914098 secs 2024-07-26T12:04:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12235.914098 secs 2024-07-26T12:04:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:51 smithi044 ceph-mon[32404]: pgmap v6936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:04:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:51 smithi116 ceph-mon[37097]: pgmap v6936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:04:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:51 smithi171 ceph-mon[37789]: pgmap v6936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:04:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:53 smithi044 ceph-mon[32404]: pgmap v6937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:53 smithi116 ceph-mon[37097]: pgmap v6937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:53 smithi171 ceph-mon[37789]: pgmap v6937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:55 smithi044 ceph-mon[32404]: pgmap v6938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12240.914232 secs 2024-07-26T12:04:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:55 smithi116 ceph-mon[37097]: pgmap v6938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12240.914232 secs 2024-07-26T12:04:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:55 smithi171 ceph-mon[37789]: pgmap v6938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:04:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12240.914232 secs 2024-07-26T12:04:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:57 smithi044 ceph-mon[32404]: pgmap v6939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:57 smithi116 ceph-mon[37097]: pgmap v6939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:57 smithi171 ceph-mon[37789]: pgmap v6939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:59.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:59.804+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:04:59.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:59.804+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:59.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:04:59.804+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:05:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:04:59 smithi116 ceph-mon[37097]: pgmap v6940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:04:59 smithi171 ceph-mon[37789]: pgmap v6940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:04:59 smithi044 ceph-mon[32404]: pgmap v6940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12245.914400 secs 2024-07-26T12:05:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12245.914400 secs 2024-07-26T12:05:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12245.914400 secs 2024-07-26T12:05:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:01 smithi171 ceph-mon[37789]: pgmap v6941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:01 smithi044 ceph-mon[32404]: pgmap v6941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:01 smithi116 ceph-mon[37097]: pgmap v6941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:03 smithi171 ceph-mon[37789]: pgmap v6942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:03 smithi044 ceph-mon[32404]: pgmap v6942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:03 smithi116 ceph-mon[37097]: pgmap v6942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:05 smithi171 ceph-mon[37789]: pgmap v6943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12250.914540 secs 2024-07-26T12:05:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:05 smithi044 ceph-mon[32404]: pgmap v6943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12250.914540 secs 2024-07-26T12:05:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:05 smithi116 ceph-mon[37097]: pgmap v6943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12250.914540 secs 2024-07-26T12:05:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:07 smithi044 ceph-mon[32404]: pgmap v6944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:07 smithi171 ceph-mon[37789]: pgmap v6944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:07 smithi116 ceph-mon[37097]: pgmap v6944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:09 smithi171 ceph-mon[37789]: pgmap v6945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:09 smithi044 ceph-mon[32404]: pgmap v6945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:09 smithi116 ceph-mon[37097]: pgmap v6945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12255.914686 secs 2024-07-26T12:05:11.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12255.914686 secs 2024-07-26T12:05:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12255.914686 secs 2024-07-26T12:05:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:11 smithi171 ceph-mon[37789]: pgmap v6946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:11 smithi044 ceph-mon[32404]: pgmap v6946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:11 smithi116 ceph-mon[37097]: pgmap v6946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:13 smithi171 ceph-mon[37789]: pgmap v6947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:13 smithi044 ceph-mon[32404]: pgmap v6947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:13 smithi116 ceph-mon[37097]: pgmap v6947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:14.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:14.809+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:05:14.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:14.809+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:14.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:14.809+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:05:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:15 smithi171 ceph-mon[37789]: pgmap v6948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12260.914823 secs 2024-07-26T12:05:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:15 smithi044 ceph-mon[32404]: pgmap v6948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12260.914823 secs 2024-07-26T12:05:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:15 smithi116 ceph-mon[37097]: pgmap v6948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12260.914823 secs 2024-07-26T12:05:16.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:16.880+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400036b0 con 0x7fcf6c1057b0 2024-07-26T12:05:16.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:16.880+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:05:16.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:16.881+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400036b0 con 0x7fcf6c1057b0 2024-07-26T12:05:16.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:16.881+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:05:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:17 smithi171 ceph-mon[37789]: pgmap v6949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:17 smithi044 ceph-mon[32404]: pgmap v6949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:17 smithi116 ceph-mon[37097]: pgmap v6949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:19 smithi044 ceph-mon[32404]: pgmap v6950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:19 smithi116 ceph-mon[37097]: pgmap v6950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:20.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:19 smithi171 ceph-mon[37789]: pgmap v6950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12265.914984 secs 2024-07-26T12:05:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12265.914984 secs 2024-07-26T12:05:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12265.914984 secs 2024-07-26T12:05:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:21 smithi044 ceph-mon[32404]: pgmap v6951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:05:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:21 smithi116 ceph-mon[37097]: pgmap v6951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:05:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:21 smithi171 ceph-mon[37789]: pgmap v6951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:05:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:23 smithi044 ceph-mon[32404]: pgmap v6952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:23 smithi116 ceph-mon[37097]: pgmap v6952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:23 smithi171 ceph-mon[37789]: pgmap v6952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:25 smithi044 ceph-mon[32404]: pgmap v6953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12270.915144 secs 2024-07-26T12:05:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:25 smithi116 ceph-mon[37097]: pgmap v6953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12270.915144 secs 2024-07-26T12:05:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:25 smithi171 ceph-mon[37789]: pgmap v6953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12270.915144 secs 2024-07-26T12:05:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:27 smithi044 ceph-mon[32404]: pgmap v6954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:27 smithi116 ceph-mon[37097]: pgmap v6954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:27 smithi171 ceph-mon[37789]: pgmap v6954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:29.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:29.819+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:05:29.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:29.819+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:05:29.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:29.819+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:29 smithi044 ceph-mon[32404]: pgmap v6955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:29 smithi116 ceph-mon[37097]: pgmap v6955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:29 smithi171 ceph-mon[37789]: pgmap v6955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12275.915309 secs 2024-07-26T12:05:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12275.915309 secs 2024-07-26T12:05:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12275.915309 secs 2024-07-26T12:05:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:31 smithi044 ceph-mon[32404]: pgmap v6956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:31 smithi116 ceph-mon[37097]: pgmap v6956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:31 smithi171 ceph-mon[37789]: pgmap v6956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:33 smithi044 ceph-mon[32404]: pgmap v6957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:33 smithi116 ceph-mon[37097]: pgmap v6957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:33 smithi171 ceph-mon[37789]: pgmap v6957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:35 smithi044 ceph-mon[32404]: pgmap v6958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12280.915475 secs 2024-07-26T12:05:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:35 smithi116 ceph-mon[37097]: pgmap v6958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12280.915475 secs 2024-07-26T12:05:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:35 smithi171 ceph-mon[37789]: pgmap v6958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:05:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12280.915475 secs 2024-07-26T12:05:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:37 smithi044 ceph-mon[32404]: pgmap v6959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:37 smithi116 ceph-mon[37097]: pgmap v6959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:37 smithi171 ceph-mon[37789]: pgmap v6959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:39 smithi044 ceph-mon[32404]: pgmap v6960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:39 smithi116 ceph-mon[37097]: pgmap v6960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:39 smithi171 ceph-mon[37789]: pgmap v6960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12285.915667 secs 2024-07-26T12:05:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12285.915667 secs 2024-07-26T12:05:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12285.915667 secs 2024-07-26T12:05:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:41 smithi044 ceph-mon[32404]: pgmap v6961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:42.283 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:41 smithi116 ceph-mon[37097]: pgmap v6961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:41 smithi171 ceph-mon[37789]: pgmap v6961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:43 smithi044 ceph-mon[32404]: pgmap v6962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:05:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:43 smithi116 ceph-mon[37097]: pgmap v6962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:05:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:43 smithi171 ceph-mon[37789]: pgmap v6962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:05:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:05:44.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:44.824+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:05:44.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:44.824+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:05:44.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:44.824+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:44 smithi044 ceph-mon[32404]: pgmap v6963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:05:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:44 smithi044 ceph-mon[32404]: pgmap v6964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:44 smithi116 ceph-mon[37097]: pgmap v6963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:05:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:44 smithi116 ceph-mon[37097]: pgmap v6964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:44 smithi171 ceph-mon[37789]: pgmap v6963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:05:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:44 smithi171 ceph-mon[37789]: pgmap v6964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12290.915803 secs 2024-07-26T12:05:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12290.915803 secs 2024-07-26T12:05:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12290.915803 secs 2024-07-26T12:05:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:46 smithi044 ceph-mon[32404]: pgmap v6965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:05:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:46 smithi116 ceph-mon[37097]: pgmap v6965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:05:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:46 smithi171 ceph-mon[37789]: pgmap v6965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:05:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:48 smithi044 ceph-mon[32404]: pgmap v6966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:48 smithi116 ceph-mon[37097]: pgmap v6966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:48 smithi171 ceph-mon[37789]: pgmap v6966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:50 smithi044 ceph-mon[32404]: pgmap v6967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12295.915946 secs 2024-07-26T12:05:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:50 smithi116 ceph-mon[37097]: pgmap v6967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12295.915946 secs 2024-07-26T12:05:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:50 smithi171 ceph-mon[37789]: pgmap v6967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12295.915946 secs 2024-07-26T12:05:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:52 smithi044 ceph-mon[32404]: pgmap v6968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:52 smithi116 ceph-mon[37097]: pgmap v6968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:52 smithi171 ceph-mon[37789]: pgmap v6968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:54 smithi044 ceph-mon[32404]: pgmap v6969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:54 smithi116 ceph-mon[37097]: pgmap v6969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:54 smithi171 ceph-mon[37789]: pgmap v6969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12300.916112 secs 2024-07-26T12:05:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12300.916112 secs 2024-07-26T12:05:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12300.916112 secs 2024-07-26T12:05:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:57 smithi044 ceph-mon[32404]: pgmap v6970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:57 smithi116 ceph-mon[37097]: pgmap v6970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:57 smithi171 ceph-mon[37789]: pgmap v6970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:05:59 smithi044 ceph-mon[32404]: pgmap v6971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:05:59 smithi116 ceph-mon[37097]: pgmap v6971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:05:59 smithi171 ceph-mon[37789]: pgmap v6971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:59.835+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:05:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:59.835+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:05:59.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:05:59.835+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:01 smithi044 ceph-mon[32404]: pgmap v6972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12305.916261 secs 2024-07-26T12:06:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:01 smithi116 ceph-mon[37097]: pgmap v6972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12305.916261 secs 2024-07-26T12:06:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:01 smithi171 ceph-mon[37789]: pgmap v6972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12305.916261 secs 2024-07-26T12:06:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:03 smithi116 ceph-mon[37097]: pgmap v6973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:03 smithi171 ceph-mon[37789]: pgmap v6973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:03 smithi044 ceph-mon[32404]: pgmap v6973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:05 smithi171 ceph-mon[37789]: pgmap v6974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:05 smithi044 ceph-mon[32404]: pgmap v6974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:05 smithi116 ceph-mon[37097]: pgmap v6974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12310.916397 secs 2024-07-26T12:06:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12310.916397 secs 2024-07-26T12:06:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12310.916397 secs 2024-07-26T12:06:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:07 smithi171 ceph-mon[37789]: pgmap v6975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:07 smithi044 ceph-mon[32404]: pgmap v6975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:07 smithi116 ceph-mon[37097]: pgmap v6975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:09 smithi171 ceph-mon[37789]: pgmap v6976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:09 smithi044 ceph-mon[32404]: pgmap v6976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:09 smithi116 ceph-mon[37097]: pgmap v6976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:11 smithi171 ceph-mon[37789]: pgmap v6977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12315.916552 secs 2024-07-26T12:06:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:11 smithi044 ceph-mon[32404]: pgmap v6977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12315.916552 secs 2024-07-26T12:06:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:11 smithi116 ceph-mon[37097]: pgmap v6977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12315.916552 secs 2024-07-26T12:06:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:13 smithi171 ceph-mon[37789]: pgmap v6978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:13 smithi044 ceph-mon[32404]: pgmap v6978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:13 smithi116 ceph-mon[37097]: pgmap v6978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:14.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:14.839+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:06:14.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:14.839+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:06:14.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:14.839+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:15 smithi044 ceph-mon[32404]: pgmap v6979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:15 smithi171 ceph-mon[37789]: pgmap v6979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:15 smithi116 ceph-mon[37097]: pgmap v6979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12320.916717 secs 2024-07-26T12:06:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12320.916717 secs 2024-07-26T12:06:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12320.916717 secs 2024-07-26T12:06:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:17 smithi171 ceph-mon[37789]: pgmap v6980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:17 smithi044 ceph-mon[32404]: pgmap v6980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:17 smithi116 ceph-mon[37097]: pgmap v6980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:19 smithi171 ceph-mon[37789]: pgmap v6981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:19 smithi044 ceph-mon[32404]: pgmap v6981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:19 smithi116 ceph-mon[37097]: pgmap v6981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:21 smithi044 ceph-mon[32404]: pgmap v6982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12325.916881 secs 2024-07-26T12:06:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:21 smithi116 ceph-mon[37097]: pgmap v6982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12325.916881 secs 2024-07-26T12:06:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:21 smithi171 ceph-mon[37789]: pgmap v6982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12325.916881 secs 2024-07-26T12:06:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:23 smithi044 ceph-mon[32404]: pgmap v6983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:23 smithi116 ceph-mon[37097]: pgmap v6983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:23 smithi171 ceph-mon[37789]: pgmap v6983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:25 smithi044 ceph-mon[32404]: pgmap v6984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:25 smithi116 ceph-mon[37097]: pgmap v6984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:25 smithi171 ceph-mon[37789]: pgmap v6984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12330.917042 secs 2024-07-26T12:06:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12330.917042 secs 2024-07-26T12:06:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12330.917042 secs 2024-07-26T12:06:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:27 smithi044 ceph-mon[32404]: pgmap v6985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:27 smithi116 ceph-mon[37097]: pgmap v6985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:27 smithi171 ceph-mon[37789]: pgmap v6985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:29 smithi044 ceph-mon[32404]: pgmap v6986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:29 smithi116 ceph-mon[37097]: pgmap v6986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:29 smithi171 ceph-mon[37789]: pgmap v6986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:29.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:29.841+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:06:29.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:29.842+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:06:29.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:29.842+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:31 smithi044 ceph-mon[32404]: pgmap v6987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12335.917214 secs 2024-07-26T12:06:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:31 smithi116 ceph-mon[37097]: pgmap v6987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12335.917214 secs 2024-07-26T12:06:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:31 smithi171 ceph-mon[37789]: pgmap v6987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12335.917214 secs 2024-07-26T12:06:33.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:33 smithi044 ceph-mon[32404]: pgmap v6988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:33 smithi116 ceph-mon[37097]: pgmap v6988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:33 smithi171 ceph-mon[37789]: pgmap v6988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:35 smithi044 ceph-mon[32404]: pgmap v6989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:35 smithi116 ceph-mon[37097]: pgmap v6989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:35 smithi171 ceph-mon[37789]: pgmap v6989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12340.917377 secs 2024-07-26T12:06:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12340.917377 secs 2024-07-26T12:06:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12340.917377 secs 2024-07-26T12:06:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:37 smithi044 ceph-mon[32404]: pgmap v6990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:37 smithi116 ceph-mon[37097]: pgmap v6990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:37 smithi171 ceph-mon[37789]: pgmap v6990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:39 smithi044 ceph-mon[32404]: pgmap v6991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:39 smithi116 ceph-mon[37097]: pgmap v6991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:39 smithi171 ceph-mon[37789]: pgmap v6991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:41 smithi044 ceph-mon[32404]: pgmap v6992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12345.917523 secs 2024-07-26T12:06:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:41 smithi116 ceph-mon[37097]: pgmap v6992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12345.917523 secs 2024-07-26T12:06:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:41 smithi171 ceph-mon[37789]: pgmap v6992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12345.917523 secs 2024-07-26T12:06:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:43 smithi044 ceph-mon[32404]: pgmap v6993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:43 smithi116 ceph-mon[37097]: pgmap v6993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:43 smithi171 ceph-mon[37789]: pgmap v6993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:44.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:44.847+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:06:44.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:44.847+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:06:44.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:44.847+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:45 smithi044 ceph-mon[32404]: pgmap v6994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:06:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:45 smithi116 ceph-mon[37097]: pgmap v6994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:45.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:06:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:45 smithi171 ceph-mon[37789]: pgmap v6994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:06:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:46 smithi044 ceph-mon[32404]: pgmap v6995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:06:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:46 smithi044 ceph-mon[32404]: pgmap v6996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12350.917692 secs 2024-07-26T12:06:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:46 smithi116 ceph-mon[37097]: pgmap v6995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:06:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:46 smithi116 ceph-mon[37097]: pgmap v6996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12350.917692 secs 2024-07-26T12:06:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:46 smithi171 ceph-mon[37789]: pgmap v6995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:06:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:46 smithi171 ceph-mon[37789]: pgmap v6996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12350.917692 secs 2024-07-26T12:06:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:48 smithi044 ceph-mon[32404]: pgmap v6997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:48 smithi116 ceph-mon[37097]: pgmap v6997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:48 smithi171 ceph-mon[37789]: pgmap v6997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:50 smithi044 ceph-mon[32404]: pgmap v6998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:50 smithi116 ceph-mon[37097]: pgmap v6998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:50 smithi171 ceph-mon[37789]: pgmap v6998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12355.917823 secs 2024-07-26T12:06:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12355.917823 secs 2024-07-26T12:06:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12355.917823 secs 2024-07-26T12:06:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:52 smithi044 ceph-mon[32404]: pgmap v6999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:52 smithi116 ceph-mon[37097]: pgmap v6999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:52 smithi171 ceph-mon[37789]: pgmap v6999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:54 smithi044 ceph-mon[32404]: pgmap v7000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:54 smithi116 ceph-mon[37097]: pgmap v7000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:54 smithi171 ceph-mon[37789]: pgmap v7000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:56 smithi044 ceph-mon[32404]: pgmap v7001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12360.917991 secs 2024-07-26T12:06:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:56 smithi116 ceph-mon[37097]: pgmap v7001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12360.917991 secs 2024-07-26T12:06:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:56 smithi171 ceph-mon[37789]: pgmap v7001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12360.917991 secs 2024-07-26T12:06:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:06:58 smithi044 ceph-mon[32404]: pgmap v7002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:06:58 smithi116 ceph-mon[37097]: pgmap v7002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:06:58 smithi171 ceph-mon[37789]: pgmap v7002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:06:59.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:59.857+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:06:59.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:59.857+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:06:59.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:06:59.857+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:00 smithi044 ceph-mon[32404]: pgmap v7003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:00 smithi116 ceph-mon[37097]: pgmap v7003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:00 smithi171 ceph-mon[37789]: pgmap v7003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12365.918092 secs 2024-07-26T12:07:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12365.918092 secs 2024-07-26T12:07:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12365.918092 secs 2024-07-26T12:07:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:02 smithi044 ceph-mon[32404]: pgmap v7004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:02 smithi116 ceph-mon[37097]: pgmap v7004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:02 smithi171 ceph-mon[37789]: pgmap v7004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:04 smithi044 ceph-mon[32404]: pgmap v7005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:04 smithi116 ceph-mon[37097]: pgmap v7005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:04.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:04 smithi171 ceph-mon[37789]: pgmap v7005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:06 smithi044 ceph-mon[32404]: pgmap v7006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12370.918243 secs 2024-07-26T12:07:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:06 smithi116 ceph-mon[37097]: pgmap v7006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12370.918243 secs 2024-07-26T12:07:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:06 smithi171 ceph-mon[37789]: pgmap v7006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12370.918243 secs 2024-07-26T12:07:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:08 smithi116 ceph-mon[37097]: pgmap v7007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:08 smithi171 ceph-mon[37789]: pgmap v7007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:08 smithi044 ceph-mon[32404]: pgmap v7007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:10 smithi171 ceph-mon[37789]: pgmap v7008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:10 smithi044 ceph-mon[32404]: pgmap v7008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:10 smithi116 ceph-mon[37097]: pgmap v7008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12375.918385 secs 2024-07-26T12:07:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12375.918385 secs 2024-07-26T12:07:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12375.918385 secs 2024-07-26T12:07:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:12 smithi171 ceph-mon[37789]: pgmap v7009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:12 smithi044 ceph-mon[32404]: pgmap v7009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:12 smithi116 ceph-mon[37097]: pgmap v7009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:14 smithi171 ceph-mon[37789]: pgmap v7010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:14 smithi044 ceph-mon[32404]: pgmap v7010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:14 smithi116 ceph-mon[37097]: pgmap v7010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:14.863+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:07:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:14.863+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:07:14.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:14.863+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:16 smithi171 ceph-mon[37789]: pgmap v7011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12380.918551 secs 2024-07-26T12:07:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:16 smithi044 ceph-mon[32404]: pgmap v7011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12380.918551 secs 2024-07-26T12:07:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:16 smithi116 ceph-mon[37097]: pgmap v7011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12380.918551 secs 2024-07-26T12:07:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:18 smithi171 ceph-mon[37789]: pgmap v7012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:18 smithi044 ceph-mon[32404]: pgmap v7012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:18 smithi116 ceph-mon[37097]: pgmap v7012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:20 smithi171 ceph-mon[37789]: pgmap v7013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:20 smithi044 ceph-mon[32404]: pgmap v7013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:20 smithi116 ceph-mon[37097]: pgmap v7013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12385.918716 secs 2024-07-26T12:07:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12385.918716 secs 2024-07-26T12:07:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12385.918716 secs 2024-07-26T12:07:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:22 smithi171 ceph-mon[37789]: pgmap v7014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:22 smithi044 ceph-mon[32404]: pgmap v7014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:22 smithi116 ceph-mon[37097]: pgmap v7014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:23 smithi044 ceph-mon[32404]: pgmap v7015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:23 smithi116 ceph-mon[37097]: pgmap v7015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:23 smithi171 ceph-mon[37789]: pgmap v7015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:26 smithi044 ceph-mon[32404]: pgmap v7016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12390.918880 secs 2024-07-26T12:07:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:26 smithi116 ceph-mon[37097]: pgmap v7016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12390.918880 secs 2024-07-26T12:07:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:26 smithi171 ceph-mon[37789]: pgmap v7016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12390.918880 secs 2024-07-26T12:07:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:28 smithi044 ceph-mon[32404]: pgmap v7017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:28 smithi116 ceph-mon[37097]: pgmap v7017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:28 smithi171 ceph-mon[37789]: pgmap v7017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:29.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:29.874+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:07:29.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:29.874+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:07:29.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:29.874+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:30 smithi044 ceph-mon[32404]: pgmap v7018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:30 smithi116 ceph-mon[37097]: pgmap v7018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:30 smithi171 ceph-mon[37789]: pgmap v7018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12395.918980 secs 2024-07-26T12:07:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12395.918980 secs 2024-07-26T12:07:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12395.918980 secs 2024-07-26T12:07:31.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:31.907+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:07:31.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:31.907+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:07:31.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:31.908+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:07:31.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:31.908+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:07:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:32 smithi116 ceph-mon[37097]: pgmap v7019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:32 smithi171 ceph-mon[37789]: pgmap v7019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:32 smithi044 ceph-mon[32404]: pgmap v7019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:34 smithi116 ceph-mon[37097]: pgmap v7020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:34 smithi171 ceph-mon[37789]: pgmap v7020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:34 smithi044 ceph-mon[32404]: pgmap v7020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:36 smithi171 ceph-mon[37789]: pgmap v7021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12400.919154 secs 2024-07-26T12:07:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:36 smithi044 ceph-mon[32404]: pgmap v7021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12400.919154 secs 2024-07-26T12:07:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:36 smithi116 ceph-mon[37097]: pgmap v7021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12400.919154 secs 2024-07-26T12:07:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:38 smithi171 ceph-mon[37789]: pgmap v7022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:38 smithi044 ceph-mon[32404]: pgmap v7022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:38 smithi116 ceph-mon[37097]: pgmap v7022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:40 smithi171 ceph-mon[37789]: pgmap v7023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:40 smithi044 ceph-mon[32404]: pgmap v7023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:40 smithi116 ceph-mon[37097]: pgmap v7023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12405.919314 secs 2024-07-26T12:07:41.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12405.919314 secs 2024-07-26T12:07:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12405.919314 secs 2024-07-26T12:07:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:42 smithi171 ceph-mon[37789]: pgmap v7024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:42.754 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:42 smithi044 ceph-mon[32404]: pgmap v7024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:42 smithi116 ceph-mon[37097]: pgmap v7024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:44 smithi171 ceph-mon[37789]: pgmap v7025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:44 smithi044 ceph-mon[32404]: pgmap v7025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:44 smithi116 ceph-mon[37097]: pgmap v7025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:44.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:44.880+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:07:44.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:44.880+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:07:44.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:44.880+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:46 smithi171 ceph-mon[37789]: pgmap v7026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12410.919434 secs 2024-07-26T12:07:46.598 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:46 smithi116 ceph-mon[37097]: pgmap v7026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:46.599 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:46.599 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12410.919434 secs 2024-07-26T12:07:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:46 smithi044 ceph-mon[32404]: pgmap v7026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12410.919434 secs 2024-07-26T12:07:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:48 smithi044 ceph-mon[32404]: pgmap v7027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:48 smithi171 ceph-mon[37789]: pgmap v7027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:48.668 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:48 smithi116 ceph-mon[37097]: pgmap v7027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:50 smithi171 ceph-mon[37789]: pgmap v7028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:50.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:50.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:50 smithi044 ceph-mon[32404]: pgmap v7028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:50 smithi116 ceph-mon[37097]: pgmap v7028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:07:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12415.919527 secs 2024-07-26T12:07:51.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12415.919527 secs 2024-07-26T12:07:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12415.919527 secs 2024-07-26T12:07:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:52 smithi044 ceph-mon[32404]: pgmap v7029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:52 smithi116 ceph-mon[37097]: pgmap v7029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:52 smithi171 ceph-mon[37789]: pgmap v7029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: pgmap v7030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:07:53.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: pgmap v7031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:53.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: pgmap v7030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: pgmap v7031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:53.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:07:53.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:53.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:53.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: pgmap v7030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:07:53.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: pgmap v7031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:53.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:07:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:54 smithi044 ceph-mon[32404]: pgmap v7032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:54 smithi116 ceph-mon[37097]: pgmap v7032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:54 smithi171 ceph-mon[37789]: pgmap v7032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12420.919672 secs 2024-07-26T12:07:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12420.919672 secs 2024-07-26T12:07:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12420.919672 secs 2024-07-26T12:07:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:56 smithi044 ceph-mon[32404]: pgmap v7033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:56 smithi116 ceph-mon[37097]: pgmap v7033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:56 smithi171 ceph-mon[37789]: pgmap v7033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:07:58 smithi044 ceph-mon[32404]: pgmap v7034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:07:58 smithi116 ceph-mon[37097]: pgmap v7034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:07:58 smithi171 ceph-mon[37789]: pgmap v7034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:59.888+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:07:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:59.888+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:59.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:07:59.888+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:08:00.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:00 smithi044 ceph-mon[32404]: pgmap v7035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:00 smithi116 ceph-mon[37097]: pgmap v7035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:00 smithi171 ceph-mon[37789]: pgmap v7035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12425.919800 secs 2024-07-26T12:08:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12425.919800 secs 2024-07-26T12:08:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12425.919800 secs 2024-07-26T12:08:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:02 smithi044 ceph-mon[32404]: pgmap v7036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:02 smithi116 ceph-mon[37097]: pgmap v7036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:02 smithi171 ceph-mon[37789]: pgmap v7036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:04 smithi044 ceph-mon[32404]: pgmap v7037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:04 smithi116 ceph-mon[37097]: pgmap v7037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:04 smithi171 ceph-mon[37789]: pgmap v7037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12430.919935 secs 2024-07-26T12:08:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12430.919935 secs 2024-07-26T12:08:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12430.919935 secs 2024-07-26T12:08:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:06 smithi171 ceph-mon[37789]: pgmap v7038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:06 smithi044 ceph-mon[32404]: pgmap v7038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:06 smithi116 ceph-mon[37097]: pgmap v7038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:08 smithi044 ceph-mon[32404]: pgmap v7039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:08 smithi116 ceph-mon[37097]: pgmap v7039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:08 smithi171 ceph-mon[37789]: pgmap v7039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:10 smithi044 ceph-mon[32404]: pgmap v7040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:10.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:10 smithi171 ceph-mon[37789]: pgmap v7040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:10.842 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:10 smithi116 ceph-mon[37097]: pgmap v7040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:11.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12435.920072 secs 2024-07-26T12:08:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12435.920072 secs 2024-07-26T12:08:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12435.920072 secs 2024-07-26T12:08:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:12 smithi044 ceph-mon[32404]: pgmap v7041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:12 smithi116 ceph-mon[37097]: pgmap v7041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:12 smithi171 ceph-mon[37789]: pgmap v7041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:14 smithi044 ceph-mon[32404]: pgmap v7042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:14 smithi116 ceph-mon[37097]: pgmap v7042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:14 smithi171 ceph-mon[37789]: pgmap v7042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:14.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:14.889+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:08:14.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:14.889+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:14.889+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:08:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12440.920214 secs 2024-07-26T12:08:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12440.920214 secs 2024-07-26T12:08:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12440.920214 secs 2024-07-26T12:08:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:16.916+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:08:16.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:16.916+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:08:16.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:16.916+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:08:16.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:16.916+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:08:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:16 smithi171 ceph-mon[37789]: pgmap v7043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:08:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:16 smithi044 ceph-mon[32404]: pgmap v7043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:08:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:16 smithi116 ceph-mon[37097]: pgmap v7043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:08:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:18 smithi044 ceph-mon[32404]: pgmap v7044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:18 smithi116 ceph-mon[37097]: pgmap v7044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:18 smithi171 ceph-mon[37789]: pgmap v7044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:20 smithi044 ceph-mon[32404]: pgmap v7045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:20 smithi116 ceph-mon[37097]: pgmap v7045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:20 smithi171 ceph-mon[37789]: pgmap v7045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12445.920365 secs 2024-07-26T12:08:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12445.920365 secs 2024-07-26T12:08:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12445.920365 secs 2024-07-26T12:08:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:22 smithi044 ceph-mon[32404]: pgmap v7046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:08:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:22 smithi116 ceph-mon[37097]: pgmap v7046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:08:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:22 smithi171 ceph-mon[37789]: pgmap v7046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:08:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:24 smithi044 ceph-mon[32404]: pgmap v7047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:08:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:24 smithi116 ceph-mon[37097]: pgmap v7047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:08:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:24 smithi171 ceph-mon[37789]: pgmap v7047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:08:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12450.920519 secs 2024-07-26T12:08:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12450.920519 secs 2024-07-26T12:08:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12450.920519 secs 2024-07-26T12:08:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:26 smithi171 ceph-mon[37789]: pgmap v7048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:08:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:26 smithi044 ceph-mon[32404]: pgmap v7048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:08:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:26 smithi116 ceph-mon[37097]: pgmap v7048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:08:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:28 smithi044 ceph-mon[32404]: pgmap v7049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:28 smithi116 ceph-mon[37097]: pgmap v7049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:28 smithi171 ceph-mon[37789]: pgmap v7049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:29.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:29.898+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:08:29.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:29.898+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:29.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:29.898+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:08:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:30 smithi044 ceph-mon[32404]: pgmap v7050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:30 smithi116 ceph-mon[37097]: pgmap v7050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:30 smithi171 ceph-mon[37789]: pgmap v7050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12455.920689 secs 2024-07-26T12:08:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12455.920689 secs 2024-07-26T12:08:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12455.920689 secs 2024-07-26T12:08:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:32 smithi044 ceph-mon[32404]: pgmap v7051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:08:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:32 smithi116 ceph-mon[37097]: pgmap v7051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:08:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:32 smithi171 ceph-mon[37789]: pgmap v7051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:08:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:34 smithi044 ceph-mon[32404]: pgmap v7052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:34 smithi116 ceph-mon[37097]: pgmap v7052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:34 smithi171 ceph-mon[37789]: pgmap v7052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12460.920784 secs 2024-07-26T12:08:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12460.920784 secs 2024-07-26T12:08:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12460.920784 secs 2024-07-26T12:08:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:36 smithi171 ceph-mon[37789]: pgmap v7053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:36 smithi044 ceph-mon[32404]: pgmap v7053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:36 smithi116 ceph-mon[37097]: pgmap v7053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:38 smithi044 ceph-mon[32404]: pgmap v7054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:38 smithi116 ceph-mon[37097]: pgmap v7054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:38 smithi171 ceph-mon[37789]: pgmap v7054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:40 smithi044 ceph-mon[32404]: pgmap v7055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:40 smithi116 ceph-mon[37097]: pgmap v7055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:40 smithi171 ceph-mon[37789]: pgmap v7055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:41.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12465.920954 secs 2024-07-26T12:08:41.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12465.920954 secs 2024-07-26T12:08:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12465.920954 secs 2024-07-26T12:08:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:42 smithi044 ceph-mon[32404]: pgmap v7056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:42 smithi116 ceph-mon[37097]: pgmap v7056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:42 smithi171 ceph-mon[37789]: pgmap v7056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:44.706 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:44 smithi044 ceph-mon[32404]: pgmap v7057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:44 smithi116 ceph-mon[37097]: pgmap v7057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:44 smithi171 ceph-mon[37789]: pgmap v7057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:44.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:44.904+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:08:44.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:44.904+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:44.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:44.904+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:08:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12470.921113 secs 2024-07-26T12:08:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12470.921113 secs 2024-07-26T12:08:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12470.921113 secs 2024-07-26T12:08:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:46 smithi171 ceph-mon[37789]: pgmap v7058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:46 smithi044 ceph-mon[32404]: pgmap v7058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:46 smithi116 ceph-mon[37097]: pgmap v7058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:48 smithi044 ceph-mon[32404]: pgmap v7059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:48 smithi116 ceph-mon[37097]: pgmap v7059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:48 smithi171 ceph-mon[37789]: pgmap v7059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:50 smithi044 ceph-mon[32404]: pgmap v7060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:50 smithi116 ceph-mon[37097]: pgmap v7060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:50 smithi171 ceph-mon[37789]: pgmap v7060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12475.921266 secs 2024-07-26T12:08:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12475.921266 secs 2024-07-26T12:08:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12475.921266 secs 2024-07-26T12:08:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:52 smithi044 ceph-mon[32404]: pgmap v7061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:52 smithi116 ceph-mon[37097]: pgmap v7061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:52 smithi171 ceph-mon[37789]: pgmap v7061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:08:53.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:53.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:08:53.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:53 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:08:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:08:53.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:53 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:08:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:53 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:08:54.686 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:54 smithi171 ceph-mon[37789]: pgmap v7062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:54 smithi044 ceph-mon[32404]: pgmap v7062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:54 smithi116 ceph-mon[37097]: pgmap v7062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12480.921385 secs 2024-07-26T12:08:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12480.921385 secs 2024-07-26T12:08:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12480.921385 secs 2024-07-26T12:08:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:08:57.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:08:57.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:57.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:57.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: pgmap v7063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:57.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:57 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:08:57.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:08:57.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:57.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:57.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: pgmap v7063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:57.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:57 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: pgmap v7063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:57.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:57 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:08:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:08:58 smithi044 ceph-mon[32404]: pgmap v7064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:08:58 smithi116 ceph-mon[37097]: pgmap v7064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:08:58 smithi171 ceph-mon[37789]: pgmap v7064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:59.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:59.914+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:08:59.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:59.915+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:08:59.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:08:59.915+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:00.662 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:00 smithi044 ceph-mon[32404]: pgmap v7065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:00 smithi116 ceph-mon[37097]: pgmap v7065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:00 smithi171 ceph-mon[37789]: pgmap v7065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12485.921518 secs 2024-07-26T12:09:01.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12485.921518 secs 2024-07-26T12:09:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12485.921518 secs 2024-07-26T12:09:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:02 smithi044 ceph-mon[32404]: pgmap v7066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:02 smithi116 ceph-mon[37097]: pgmap v7066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:02 smithi171 ceph-mon[37789]: pgmap v7066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:04 smithi044 ceph-mon[32404]: pgmap v7067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:04 smithi116 ceph-mon[37097]: pgmap v7067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:04 smithi171 ceph-mon[37789]: pgmap v7067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12490.921661 secs 2024-07-26T12:09:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12490.921661 secs 2024-07-26T12:09:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12490.921661 secs 2024-07-26T12:09:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:06 smithi171 ceph-mon[37789]: pgmap v7068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:06 smithi044 ceph-mon[32404]: pgmap v7068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:06 smithi116 ceph-mon[37097]: pgmap v7068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:08 smithi044 ceph-mon[32404]: pgmap v7069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:08 smithi116 ceph-mon[37097]: pgmap v7069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:08 smithi171 ceph-mon[37789]: pgmap v7069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:10 smithi044 ceph-mon[32404]: pgmap v7070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:10 smithi116 ceph-mon[37097]: pgmap v7070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:10 smithi171 ceph-mon[37789]: pgmap v7070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:11.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12495.921823 secs 2024-07-26T12:09:11.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12495.921823 secs 2024-07-26T12:09:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12495.921823 secs 2024-07-26T12:09:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:12 smithi044 ceph-mon[32404]: pgmap v7071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:12 smithi116 ceph-mon[37097]: pgmap v7071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:12 smithi171 ceph-mon[37789]: pgmap v7071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:14 smithi044 ceph-mon[32404]: pgmap v7072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:14 smithi116 ceph-mon[37097]: pgmap v7072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:14 smithi171 ceph-mon[37789]: pgmap v7072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:14.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:14.920+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:09:14.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:14.920+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:14.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:14.920+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:09:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12500.921975 secs 2024-07-26T12:09:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12500.921975 secs 2024-07-26T12:09:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12500.921975 secs 2024-07-26T12:09:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:16 smithi171 ceph-mon[37789]: pgmap v7073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:16 smithi044 ceph-mon[32404]: pgmap v7073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:16 smithi116 ceph-mon[37097]: pgmap v7073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:18 smithi044 ceph-mon[32404]: pgmap v7074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:18 smithi116 ceph-mon[37097]: pgmap v7074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:18 smithi171 ceph-mon[37789]: pgmap v7074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:20 smithi044 ceph-mon[32404]: pgmap v7075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:20 smithi116 ceph-mon[37097]: pgmap v7075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:20 smithi171 ceph-mon[37789]: pgmap v7075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12505.922134 secs 2024-07-26T12:09:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12505.922134 secs 2024-07-26T12:09:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12505.922134 secs 2024-07-26T12:09:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:22 smithi044 ceph-mon[32404]: pgmap v7076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:22 smithi116 ceph-mon[37097]: pgmap v7076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:22 smithi171 ceph-mon[37789]: pgmap v7076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:24 smithi044 ceph-mon[32404]: pgmap v7077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:24 smithi116 ceph-mon[37097]: pgmap v7077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:24 smithi171 ceph-mon[37789]: pgmap v7077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12510.922292 secs 2024-07-26T12:09:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12510.922292 secs 2024-07-26T12:09:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12510.922292 secs 2024-07-26T12:09:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:26 smithi171 ceph-mon[37789]: pgmap v7078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:26 smithi044 ceph-mon[32404]: pgmap v7078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:26 smithi116 ceph-mon[37097]: pgmap v7078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:28 smithi044 ceph-mon[32404]: pgmap v7079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:28 smithi116 ceph-mon[37097]: pgmap v7079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:28 smithi171 ceph-mon[37789]: pgmap v7079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:29.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:29.928+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:09:29.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:29.928+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:29.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:29.928+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:09:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:30 smithi044 ceph-mon[32404]: pgmap v7080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:30 smithi116 ceph-mon[37097]: pgmap v7080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:30 smithi171 ceph-mon[37789]: pgmap v7080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12515.922435 secs 2024-07-26T12:09:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12515.922435 secs 2024-07-26T12:09:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12515.922435 secs 2024-07-26T12:09:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:32 smithi044 ceph-mon[32404]: pgmap v7081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:32 smithi116 ceph-mon[37097]: pgmap v7081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:32 smithi171 ceph-mon[37789]: pgmap v7081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:34 smithi044 ceph-mon[32404]: pgmap v7082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:34 smithi116 ceph-mon[37097]: pgmap v7082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:34 smithi171 ceph-mon[37789]: pgmap v7082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12520.922598 secs 2024-07-26T12:09:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12520.922598 secs 2024-07-26T12:09:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12520.922598 secs 2024-07-26T12:09:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:37 smithi171 ceph-mon[37789]: pgmap v7083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:37 smithi044 ceph-mon[32404]: pgmap v7083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:37 smithi116 ceph-mon[37097]: pgmap v7083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:38 smithi044 ceph-mon[32404]: pgmap v7084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:38 smithi116 ceph-mon[37097]: pgmap v7084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:38 smithi171 ceph-mon[37789]: pgmap v7084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:40 smithi044 ceph-mon[32404]: pgmap v7085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:40 smithi116 ceph-mon[37097]: pgmap v7085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:40 smithi171 ceph-mon[37789]: pgmap v7085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:41.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12525.922714 secs 2024-07-26T12:09:41.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12525.922714 secs 2024-07-26T12:09:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12525.922714 secs 2024-07-26T12:09:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:42 smithi044 ceph-mon[32404]: pgmap v7086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:42 smithi116 ceph-mon[37097]: pgmap v7086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:42.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:42 smithi171 ceph-mon[37789]: pgmap v7086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:44 smithi044 ceph-mon[32404]: pgmap v7087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:44 smithi116 ceph-mon[37097]: pgmap v7087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:44 smithi171 ceph-mon[37789]: pgmap v7087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:44.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:44.929+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:09:44.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:44.929+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:44.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:44.929+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:09:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12530.922880 secs 2024-07-26T12:09:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12530.922880 secs 2024-07-26T12:09:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12530.922880 secs 2024-07-26T12:09:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:46 smithi171 ceph-mon[37789]: pgmap v7088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:46 smithi044 ceph-mon[32404]: pgmap v7088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:46 smithi116 ceph-mon[37097]: pgmap v7088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:48 smithi044 ceph-mon[32404]: pgmap v7089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:48 smithi116 ceph-mon[37097]: pgmap v7089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:48 smithi171 ceph-mon[37789]: pgmap v7089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:50 smithi044 ceph-mon[32404]: pgmap v7090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:50 smithi116 ceph-mon[37097]: pgmap v7090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:50 smithi171 ceph-mon[37789]: pgmap v7090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12535.923008 secs 2024-07-26T12:09:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12535.923008 secs 2024-07-26T12:09:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12535.923008 secs 2024-07-26T12:09:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:52 smithi044 ceph-mon[32404]: pgmap v7091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:52 smithi116 ceph-mon[37097]: pgmap v7091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:52 smithi171 ceph-mon[37789]: pgmap v7091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:55 smithi044 ceph-mon[32404]: pgmap v7092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:55 smithi116 ceph-mon[37097]: pgmap v7092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:55 smithi171 ceph-mon[37789]: pgmap v7092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12540.923175 secs 2024-07-26T12:09:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12540.923175 secs 2024-07-26T12:09:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12540.923175 secs 2024-07-26T12:09:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:57 smithi044 ceph-mon[32404]: pgmap v7093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:57 smithi116 ceph-mon[37097]: pgmap v7093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:57 smithi171 ceph-mon[37789]: pgmap v7093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:57.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:58 smithi044 ceph-mon[32404]: pgmap v7094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:09:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:58 smithi044 ceph-mon[32404]: pgmap v7095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:09:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:09:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:09:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:58 smithi116 ceph-mon[37097]: pgmap v7094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:09:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:58 smithi116 ceph-mon[37097]: pgmap v7095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:09:58.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:09:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:09:58.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:58.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:58 smithi171 ceph-mon[37789]: pgmap v7094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:09:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:58 smithi171 ceph-mon[37789]: pgmap v7095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:09:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:09:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:59.934+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:59.934+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:09:59.934+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:00 smithi044 ceph-mon[32404]: pgmap v7096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:00 smithi116 ceph-mon[37097]: pgmap v7096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:00 smithi171 ceph-mon[37789]: pgmap v7096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:10:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12545.923336 secs 2024-07-26T12:10:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12545.923336 secs 2024-07-26T12:10:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12545.923336 secs 2024-07-26T12:10:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:02 smithi044 ceph-mon[32404]: pgmap v7097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:10:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:02 smithi116 ceph-mon[37097]: pgmap v7097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:10:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:02 smithi171 ceph-mon[37789]: pgmap v7097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:10:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:04 smithi044 ceph-mon[32404]: pgmap v7098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:04 smithi116 ceph-mon[37097]: pgmap v7098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:04 smithi171 ceph-mon[37789]: pgmap v7098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12550.923416 secs 2024-07-26T12:10:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:06 smithi044 ceph-mon[32404]: pgmap v7099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12550.923416 secs 2024-07-26T12:10:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:06 smithi116 ceph-mon[37097]: pgmap v7099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12550.923416 secs 2024-07-26T12:10:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:06 smithi171 ceph-mon[37789]: pgmap v7099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:10:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:08 smithi044 ceph-mon[32404]: pgmap v7100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:09.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:08 smithi171 ceph-mon[37789]: pgmap v7100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:09.258 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:08 smithi116 ceph-mon[37097]: pgmap v7100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:10 smithi044 ceph-mon[32404]: pgmap v7101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:10 smithi116 ceph-mon[37097]: pgmap v7101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:10 smithi171 ceph-mon[37789]: pgmap v7101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:11.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12555.923559 secs 2024-07-26T12:10:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12555.923559 secs 2024-07-26T12:10:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12555.923559 secs 2024-07-26T12:10:13.000 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:12 smithi171 ceph-mon[37789]: pgmap v7102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:12 smithi044 ceph-mon[32404]: pgmap v7102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:12 smithi116 ceph-mon[37097]: pgmap v7102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:14.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:14.940+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:10:14.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:14.940+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:14.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:14.940+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:10:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:14 smithi044 ceph-mon[32404]: pgmap v7103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:14 smithi116 ceph-mon[37097]: pgmap v7103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:14 smithi171 ceph-mon[37789]: pgmap v7103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12560.923726 secs 2024-07-26T12:10:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:16 smithi044 ceph-mon[32404]: pgmap v7104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12560.923726 secs 2024-07-26T12:10:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:16 smithi116 ceph-mon[37097]: pgmap v7104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12560.923726 secs 2024-07-26T12:10:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:16 smithi171 ceph-mon[37789]: pgmap v7104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:18 smithi044 ceph-mon[32404]: pgmap v7105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:18 smithi116 ceph-mon[37097]: pgmap v7105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:18 smithi171 ceph-mon[37789]: pgmap v7105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:20 smithi044 ceph-mon[32404]: pgmap v7106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:20 smithi116 ceph-mon[37097]: pgmap v7106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:20 smithi171 ceph-mon[37789]: pgmap v7106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12565.923879 secs 2024-07-26T12:10:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12565.923879 secs 2024-07-26T12:10:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12565.923879 secs 2024-07-26T12:10:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:22 smithi171 ceph-mon[37789]: pgmap v7107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:22 smithi044 ceph-mon[32404]: pgmap v7107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:22 smithi116 ceph-mon[37097]: pgmap v7107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:24 smithi171 ceph-mon[37789]: pgmap v7108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:24 smithi044 ceph-mon[32404]: pgmap v7108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:24 smithi116 ceph-mon[37097]: pgmap v7108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12570.924028 secs 2024-07-26T12:10:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12570.924028 secs 2024-07-26T12:10:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12570.924028 secs 2024-07-26T12:10:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:26 smithi171 ceph-mon[37789]: pgmap v7109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:26 smithi044 ceph-mon[32404]: pgmap v7109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:26 smithi116 ceph-mon[37097]: pgmap v7109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:28 smithi171 ceph-mon[37789]: pgmap v7110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:28 smithi044 ceph-mon[32404]: pgmap v7110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:28 smithi116 ceph-mon[37097]: pgmap v7110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:29.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:29.941+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:10:29.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:29.941+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:29.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:29.941+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:10:31.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:30 smithi116 ceph-mon[37097]: pgmap v7111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:31.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12575.924159 secs 2024-07-26T12:10:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:30 smithi171 ceph-mon[37789]: pgmap v7111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12575.924159 secs 2024-07-26T12:10:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:30 smithi044 ceph-mon[32404]: pgmap v7111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12575.924159 secs 2024-07-26T12:10:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:32 smithi171 ceph-mon[37789]: pgmap v7112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:32 smithi044 ceph-mon[32404]: pgmap v7112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:32 smithi116 ceph-mon[37097]: pgmap v7112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:35.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:34 smithi044 ceph-mon[32404]: pgmap v7113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:34 smithi171 ceph-mon[37789]: pgmap v7113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:34 smithi116 ceph-mon[37097]: pgmap v7113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12580.924320 secs 2024-07-26T12:10:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12580.924320 secs 2024-07-26T12:10:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12580.924320 secs 2024-07-26T12:10:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:36 smithi171 ceph-mon[37789]: pgmap v7114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:36 smithi044 ceph-mon[32404]: pgmap v7114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:36 smithi116 ceph-mon[37097]: pgmap v7114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:38 smithi044 ceph-mon[32404]: pgmap v7115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:38 smithi116 ceph-mon[37097]: pgmap v7115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:38 smithi171 ceph-mon[37789]: pgmap v7115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:41.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:40 smithi171 ceph-mon[37789]: pgmap v7116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:41.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12585.924458 secs 2024-07-26T12:10:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:40 smithi044 ceph-mon[32404]: pgmap v7116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12585.924458 secs 2024-07-26T12:10:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:40 smithi116 ceph-mon[37097]: pgmap v7116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12585.924458 secs 2024-07-26T12:10:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:42 smithi044 ceph-mon[32404]: pgmap v7117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:42 smithi116 ceph-mon[37097]: pgmap v7117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:42 smithi171 ceph-mon[37789]: pgmap v7117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:44.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:44.946+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:10:44.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:44.946+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:44.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:44.946+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:10:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:44 smithi044 ceph-mon[32404]: pgmap v7118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:44 smithi116 ceph-mon[37097]: pgmap v7118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:44 smithi171 ceph-mon[37789]: pgmap v7118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12590.924650 secs 2024-07-26T12:10:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12590.924650 secs 2024-07-26T12:10:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12590.924650 secs 2024-07-26T12:10:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:46 smithi044 ceph-mon[32404]: pgmap v7119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:46 smithi116 ceph-mon[37097]: pgmap v7119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:46 smithi171 ceph-mon[37789]: pgmap v7119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:48 smithi044 ceph-mon[32404]: pgmap v7120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:48 smithi116 ceph-mon[37097]: pgmap v7120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:48 smithi171 ceph-mon[37789]: pgmap v7120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:50 smithi044 ceph-mon[32404]: pgmap v7121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12595.924775 secs 2024-07-26T12:10:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:50 smithi116 ceph-mon[37097]: pgmap v7121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12595.924775 secs 2024-07-26T12:10:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:50 smithi171 ceph-mon[37789]: pgmap v7121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:10:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12595.924775 secs 2024-07-26T12:10:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:53 smithi171 ceph-mon[37789]: pgmap v7122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:53 smithi044 ceph-mon[32404]: pgmap v7122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:53 smithi116 ceph-mon[37097]: pgmap v7122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:10:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:55 smithi171 ceph-mon[37789]: pgmap v7123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:55 smithi044 ceph-mon[32404]: pgmap v7123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:55 smithi116 ceph-mon[37097]: pgmap v7123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:10:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12600.924920 secs 2024-07-26T12:10:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12600.924920 secs 2024-07-26T12:10:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12600.924920 secs 2024-07-26T12:10:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:57 smithi044 ceph-mon[32404]: pgmap v7124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:57 smithi116 ceph-mon[37097]: pgmap v7124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:57 smithi171 ceph-mon[37789]: pgmap v7124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:59 smithi044 ceph-mon[32404]: pgmap v7125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:10:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:10:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:10:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:59 smithi116 ceph-mon[37097]: pgmap v7125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:10:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:10:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:10:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:59 smithi171 ceph-mon[37789]: pgmap v7125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:10:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:10:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:10:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:10:59.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:59.947+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:10:59.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:59.947+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:59.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:10:59.947+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:11:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:11:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:11:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:00.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:11:01.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:01 smithi116 ceph-mon[37097]: pgmap v7126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:01.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12605.925080 secs 2024-07-26T12:11:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:01 smithi044 ceph-mon[32404]: pgmap v7126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12605.925080 secs 2024-07-26T12:11:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:01 smithi171 ceph-mon[37789]: pgmap v7126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12605.925080 secs 2024-07-26T12:11:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:03 smithi044 ceph-mon[32404]: pgmap v7127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:11:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:03 smithi116 ceph-mon[37097]: pgmap v7127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:11:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:03 smithi171 ceph-mon[37789]: pgmap v7127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:11:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:05 smithi044 ceph-mon[32404]: pgmap v7128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:05 smithi116 ceph-mon[37097]: pgmap v7128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:05 smithi171 ceph-mon[37789]: pgmap v7128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12610.925246 secs 2024-07-26T12:11:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12610.925246 secs 2024-07-26T12:11:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12610.925246 secs 2024-07-26T12:11:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:07 smithi044 ceph-mon[32404]: pgmap v7129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:11:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:07 smithi116 ceph-mon[37097]: pgmap v7129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:11:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:07 smithi171 ceph-mon[37789]: pgmap v7129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:11:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:09 smithi044 ceph-mon[32404]: pgmap v7130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:09 smithi116 ceph-mon[37097]: pgmap v7130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:09 smithi171 ceph-mon[37789]: pgmap v7130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:11 smithi044 ceph-mon[32404]: pgmap v7131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12615.925387 secs 2024-07-26T12:11:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:11 smithi116 ceph-mon[37097]: pgmap v7131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12615.925387 secs 2024-07-26T12:11:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:11 smithi171 ceph-mon[37789]: pgmap v7131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12615.925387 secs 2024-07-26T12:11:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:13 smithi044 ceph-mon[32404]: pgmap v7132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:13 smithi116 ceph-mon[37097]: pgmap v7132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:13 smithi171 ceph-mon[37789]: pgmap v7132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:14.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T12:11:14.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:11:14.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58074040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:11:14.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58074040 comp rx=0 tx=0).stop 2024-07-26T12:11:14.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T12:11:14.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:11:14.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:11:14.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.727+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:11:14.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.728+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:11:14.731 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.730+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58086000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:11:14.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.948+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:11:14.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.948+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:14.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:14.948+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:11:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:15 smithi044 ceph-mon[32404]: pgmap v7133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:15 smithi116 ceph-mon[37097]: pgmap v7133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:15 smithi171 ceph-mon[37789]: pgmap v7133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12620.925537 secs 2024-07-26T12:11:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12620.925537 secs 2024-07-26T12:11:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12620.925537 secs 2024-07-26T12:11:16.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:16.949+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:11:16.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:16.949+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:11:16.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:16.949+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:11:16.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:16.949+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:11:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:17 smithi044 ceph-mon[32404]: pgmap v7134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:17 smithi116 ceph-mon[37097]: pgmap v7134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:17 smithi171 ceph-mon[37789]: pgmap v7134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:19 smithi044 ceph-mon[32404]: pgmap v7135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:19 smithi116 ceph-mon[37097]: pgmap v7135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:19 smithi171 ceph-mon[37789]: pgmap v7135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:11:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:21 smithi044 ceph-mon[32404]: pgmap v7136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12625.925689 secs 2024-07-26T12:11:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:21 smithi116 ceph-mon[37097]: pgmap v7136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12625.925689 secs 2024-07-26T12:11:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:21 smithi171 ceph-mon[37789]: pgmap v7136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12625.925689 secs 2024-07-26T12:11:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:23 smithi044 ceph-mon[32404]: pgmap v7137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:23 smithi116 ceph-mon[37097]: pgmap v7137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:23 smithi171 ceph-mon[37789]: pgmap v7137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:25 smithi044 ceph-mon[32404]: pgmap v7138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:25 smithi116 ceph-mon[37097]: pgmap v7138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:25 smithi171 ceph-mon[37789]: pgmap v7138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12630.925794 secs 2024-07-26T12:11:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12630.925794 secs 2024-07-26T12:11:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12630.925794 secs 2024-07-26T12:11:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:27 smithi044 ceph-mon[32404]: pgmap v7139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:27 smithi116 ceph-mon[37097]: pgmap v7139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:27 smithi171 ceph-mon[37789]: pgmap v7139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:29 smithi044 ceph-mon[32404]: pgmap v7140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:29 smithi116 ceph-mon[37097]: pgmap v7140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:29 smithi171 ceph-mon[37789]: pgmap v7140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:29.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:29.951+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:11:29.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:29.951+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:11:29.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:29.951+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:31 smithi044 ceph-mon[32404]: pgmap v7141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12635.925888 secs 2024-07-26T12:11:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:31 smithi116 ceph-mon[37097]: pgmap v7141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12635.925888 secs 2024-07-26T12:11:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:31 smithi171 ceph-mon[37789]: pgmap v7141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:11:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12635.925888 secs 2024-07-26T12:11:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:33 smithi044 ceph-mon[32404]: pgmap v7142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:33 smithi116 ceph-mon[37097]: pgmap v7142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:33 smithi171 ceph-mon[37789]: pgmap v7142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:35 smithi044 ceph-mon[32404]: pgmap v7143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:11:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:35 smithi116 ceph-mon[37097]: pgmap v7143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:11:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:35 smithi171 ceph-mon[37789]: pgmap v7143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:11:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12640.926000 secs 2024-07-26T12:11:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12640.926000 secs 2024-07-26T12:11:36.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12640.926000 secs 2024-07-26T12:11:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:37 smithi171 ceph-mon[37789]: pgmap v7144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:11:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:37 smithi044 ceph-mon[32404]: pgmap v7144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:11:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:37 smithi116 ceph-mon[37097]: pgmap v7144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:11:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:39 smithi171 ceph-mon[37789]: pgmap v7145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:39 smithi044 ceph-mon[32404]: pgmap v7145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:39 smithi116 ceph-mon[37097]: pgmap v7145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:11:41.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:41 smithi044 ceph-mon[32404]: pgmap v7146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:41.569 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12645.926160 secs 2024-07-26T12:11:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:41 smithi171 ceph-mon[37789]: pgmap v7146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12645.926160 secs 2024-07-26T12:11:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:41 smithi116 ceph-mon[37097]: pgmap v7146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12645.926160 secs 2024-07-26T12:11:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:43 smithi171 ceph-mon[37789]: pgmap v7147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:43 smithi044 ceph-mon[32404]: pgmap v7147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:43 smithi116 ceph-mon[37097]: pgmap v7147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:44.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:44.956+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:11:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:44.956+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:11:44.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:44.956+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:45 smithi171 ceph-mon[37789]: pgmap v7148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:45 smithi044 ceph-mon[32404]: pgmap v7148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:45 smithi116 ceph-mon[37097]: pgmap v7148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12650.926309 secs 2024-07-26T12:11:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12650.926309 secs 2024-07-26T12:11:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12650.926309 secs 2024-07-26T12:11:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:47 smithi171 ceph-mon[37789]: pgmap v7149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:47 smithi044 ceph-mon[32404]: pgmap v7149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:47 smithi116 ceph-mon[37097]: pgmap v7149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:49 smithi171 ceph-mon[37789]: pgmap v7150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:49 smithi044 ceph-mon[32404]: pgmap v7150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:49 smithi116 ceph-mon[37097]: pgmap v7150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:51.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:51 smithi044 ceph-mon[32404]: pgmap v7151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:51.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12655.926459 secs 2024-07-26T12:11:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:51 smithi171 ceph-mon[37789]: pgmap v7151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12655.926459 secs 2024-07-26T12:11:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:51 smithi116 ceph-mon[37097]: pgmap v7151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12655.926459 secs 2024-07-26T12:11:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:53 smithi171 ceph-mon[37789]: pgmap v7152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:53 smithi044 ceph-mon[32404]: pgmap v7152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:53 smithi116 ceph-mon[37097]: pgmap v7152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:11:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:55 smithi171 ceph-mon[37789]: pgmap v7153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:55 smithi044 ceph-mon[32404]: pgmap v7153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:55 smithi116 ceph-mon[37097]: pgmap v7153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:11:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12660.926665 secs 2024-07-26T12:11:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12660.926665 secs 2024-07-26T12:11:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12660.926665 secs 2024-07-26T12:11:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:57 smithi044 ceph-mon[32404]: pgmap v7154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:57 smithi116 ceph-mon[37097]: pgmap v7154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:57 smithi171 ceph-mon[37789]: pgmap v7154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:11:59 smithi044 ceph-mon[32404]: pgmap v7155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:11:59 smithi116 ceph-mon[37097]: pgmap v7155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:11:59 smithi171 ceph-mon[37789]: pgmap v7155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:11:59.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:59.958+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:11:59.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:59.958+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:59.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:11:59.958+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:12:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:01.407 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:01 smithi044 ceph-mon[32404]: pgmap v7156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:01.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12665.926771 secs 2024-07-26T12:12:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:01 smithi116 ceph-mon[37097]: pgmap v7156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12665.926771 secs 2024-07-26T12:12:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:01 smithi171 ceph-mon[37789]: pgmap v7156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12665.926771 secs 2024-07-26T12:12:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:02 smithi171 ceph-mon[37789]: pgmap v7157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:02 smithi044 ceph-mon[32404]: pgmap v7157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:02 smithi116 ceph-mon[37097]: pgmap v7157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:05 smithi044 ceph-mon[32404]: pgmap v7158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:05 smithi116 ceph-mon[37097]: pgmap v7158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:05 smithi171 ceph-mon[37789]: pgmap v7158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:12:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:06 smithi044 ceph-mon[32404]: pgmap v7159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:06 smithi044 ceph-mon[32404]: pgmap v7160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12670.926911 secs 2024-07-26T12:12:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:06 smithi116 ceph-mon[37097]: pgmap v7159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:06 smithi116 ceph-mon[37097]: pgmap v7160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12670.926911 secs 2024-07-26T12:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:06 smithi171 ceph-mon[37789]: pgmap v7159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:06 smithi171 ceph-mon[37789]: pgmap v7160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12670.926911 secs 2024-07-26T12:12:08.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:08 smithi044 ceph-mon[32404]: pgmap v7161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:12:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:08 smithi116 ceph-mon[37097]: pgmap v7161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:12:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:08 smithi171 ceph-mon[37789]: pgmap v7161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:12:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:09 smithi044 ceph-mon[32404]: pgmap v7162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:09 smithi116 ceph-mon[37097]: pgmap v7162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:09 smithi171 ceph-mon[37789]: pgmap v7162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:12:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:11 smithi044 ceph-mon[32404]: pgmap v7163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12675.927077 secs 2024-07-26T12:12:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:11 smithi116 ceph-mon[37097]: pgmap v7163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12675.927077 secs 2024-07-26T12:12:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:11 smithi171 ceph-mon[37789]: pgmap v7163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12675.927077 secs 2024-07-26T12:12:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:13 smithi044 ceph-mon[32404]: pgmap v7164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:12:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:13 smithi116 ceph-mon[37097]: pgmap v7164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:12:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:13 smithi171 ceph-mon[37789]: pgmap v7164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:12:14.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:14.964+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:12:14.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:14.964+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:14.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:14.964+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:12:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:15 smithi044 ceph-mon[32404]: pgmap v7165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:15 smithi116 ceph-mon[37097]: pgmap v7165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:15 smithi171 ceph-mon[37789]: pgmap v7165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12680.927185 secs 2024-07-26T12:12:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12680.927185 secs 2024-07-26T12:12:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12680.927185 secs 2024-07-26T12:12:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:17 smithi044 ceph-mon[32404]: pgmap v7166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:17 smithi116 ceph-mon[37097]: pgmap v7166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:17.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:17 smithi171 ceph-mon[37789]: pgmap v7166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:19 smithi044 ceph-mon[32404]: pgmap v7167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:19 smithi116 ceph-mon[37097]: pgmap v7167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:19 smithi171 ceph-mon[37789]: pgmap v7167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:21 smithi044 ceph-mon[32404]: pgmap v7168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12685.927320 secs 2024-07-26T12:12:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:21 smithi116 ceph-mon[37097]: pgmap v7168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12685.927320 secs 2024-07-26T12:12:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:21 smithi171 ceph-mon[37789]: pgmap v7168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12685.927320 secs 2024-07-26T12:12:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:23 smithi044 ceph-mon[32404]: pgmap v7169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:23 smithi116 ceph-mon[37097]: pgmap v7169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:23 smithi171 ceph-mon[37789]: pgmap v7169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:25 smithi116 ceph-mon[37097]: pgmap v7170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:25 smithi171 ceph-mon[37789]: pgmap v7170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:25 smithi044 ceph-mon[32404]: pgmap v7170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12690.927480 secs 2024-07-26T12:12:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12690.927480 secs 2024-07-26T12:12:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12690.927480 secs 2024-07-26T12:12:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:27 smithi171 ceph-mon[37789]: pgmap v7171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:27 smithi044 ceph-mon[32404]: pgmap v7171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:27 smithi116 ceph-mon[37097]: pgmap v7171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:29 smithi171 ceph-mon[37789]: pgmap v7172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:29.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:29.965+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:12:29.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:29.966+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:29.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:29.966+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:12:29.968 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:29 smithi044 ceph-mon[32404]: pgmap v7172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:29 smithi116 ceph-mon[37097]: pgmap v7172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:31.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:31 smithi116 ceph-mon[37097]: pgmap v7173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:31.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12695.927762 secs 2024-07-26T12:12:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:31 smithi171 ceph-mon[37789]: pgmap v7173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12695.927762 secs 2024-07-26T12:12:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:31 smithi044 ceph-mon[32404]: pgmap v7173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12695.927762 secs 2024-07-26T12:12:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:33 smithi171 ceph-mon[37789]: pgmap v7174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:33 smithi044 ceph-mon[32404]: pgmap v7174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:33 smithi116 ceph-mon[37097]: pgmap v7174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:35 smithi171 ceph-mon[37789]: pgmap v7175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:35 smithi044 ceph-mon[32404]: pgmap v7175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:35 smithi116 ceph-mon[37097]: pgmap v7175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12700.927912 secs 2024-07-26T12:12:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12700.927912 secs 2024-07-26T12:12:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12700.927912 secs 2024-07-26T12:12:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:37 smithi171 ceph-mon[37789]: pgmap v7176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:37.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:37 smithi044 ceph-mon[32404]: pgmap v7176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:37 smithi116 ceph-mon[37097]: pgmap v7176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:39 smithi171 ceph-mon[37789]: pgmap v7177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:40.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:39 smithi044 ceph-mon[32404]: pgmap v7177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:39 smithi116 ceph-mon[37097]: pgmap v7177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:41 smithi171 ceph-mon[37789]: pgmap v7178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12705.928063 secs 2024-07-26T12:12:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:41 smithi044 ceph-mon[32404]: pgmap v7178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12705.928063 secs 2024-07-26T12:12:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:41 smithi116 ceph-mon[37097]: pgmap v7178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12705.928063 secs 2024-07-26T12:12:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:43 smithi044 ceph-mon[32404]: pgmap v7179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:43 smithi116 ceph-mon[37097]: pgmap v7179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:43 smithi171 ceph-mon[37789]: pgmap v7179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:44.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:44.967+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:12:44.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:44.967+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:12:44.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:44.967+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:45 smithi044 ceph-mon[32404]: pgmap v7180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:45 smithi116 ceph-mon[37097]: pgmap v7180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:45 smithi171 ceph-mon[37789]: pgmap v7180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12710.928244 secs 2024-07-26T12:12:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12710.928244 secs 2024-07-26T12:12:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12710.928244 secs 2024-07-26T12:12:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:47 smithi044 ceph-mon[32404]: pgmap v7181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:47 smithi116 ceph-mon[37097]: pgmap v7181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:47 smithi171 ceph-mon[37789]: pgmap v7181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:49 smithi044 ceph-mon[32404]: pgmap v7182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:49 smithi116 ceph-mon[37097]: pgmap v7182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:49 smithi171 ceph-mon[37789]: pgmap v7182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:51 smithi044 ceph-mon[32404]: pgmap v7183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12715.928407 secs 2024-07-26T12:12:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:51 smithi116 ceph-mon[37097]: pgmap v7183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12715.928407 secs 2024-07-26T12:12:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:51 smithi171 ceph-mon[37789]: pgmap v7183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12715.928407 secs 2024-07-26T12:12:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:53 smithi044 ceph-mon[32404]: pgmap v7184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:53 smithi116 ceph-mon[37097]: pgmap v7184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:53 smithi171 ceph-mon[37789]: pgmap v7184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:12:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:55 smithi044 ceph-mon[32404]: pgmap v7185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:55 smithi116 ceph-mon[37097]: pgmap v7185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:55 smithi171 ceph-mon[37789]: pgmap v7185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12720.928516 secs 2024-07-26T12:12:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12720.928516 secs 2024-07-26T12:12:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12720.928516 secs 2024-07-26T12:12:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:57 smithi044 ceph-mon[32404]: pgmap v7186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:57 smithi116 ceph-mon[37097]: pgmap v7186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:57 smithi171 ceph-mon[37789]: pgmap v7186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:59.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:59.969+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:12:59.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:59.969+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:12:59.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:12:59.969+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:59.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:12:59 smithi044 ceph-mon[32404]: pgmap v7187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:12:59 smithi116 ceph-mon[37097]: pgmap v7187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:12:59 smithi171 ceph-mon[37789]: pgmap v7187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:01 smithi044 ceph-mon[32404]: pgmap v7188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12725.928694 secs 2024-07-26T12:13:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:01 smithi116 ceph-mon[37097]: pgmap v7188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12725.928694 secs 2024-07-26T12:13:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:01 smithi171 ceph-mon[37789]: pgmap v7188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12725.928694 secs 2024-07-26T12:13:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:03 smithi044 ceph-mon[32404]: pgmap v7189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:03 smithi116 ceph-mon[37097]: pgmap v7189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:03 smithi171 ceph-mon[37789]: pgmap v7189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:04 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:04 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:04 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:05 smithi044 ceph-mon[32404]: pgmap v7190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:13:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:05 smithi116 ceph-mon[37097]: pgmap v7190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:13:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:05 smithi171 ceph-mon[37789]: pgmap v7190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:13:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12730.928771 secs 2024-07-26T12:13:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:06 smithi044 ceph-mon[32404]: pgmap v7191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:13:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:06 smithi044 ceph-mon[32404]: pgmap v7192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12730.928771 secs 2024-07-26T12:13:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:06 smithi116 ceph-mon[37097]: pgmap v7191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:13:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:06 smithi116 ceph-mon[37097]: pgmap v7192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12730.928771 secs 2024-07-26T12:13:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:06 smithi171 ceph-mon[37789]: pgmap v7191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:13:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:06 smithi171 ceph-mon[37789]: pgmap v7192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:08 smithi044 ceph-mon[32404]: pgmap v7193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:08 smithi116 ceph-mon[37097]: pgmap v7193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:08 smithi171 ceph-mon[37789]: pgmap v7193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:10 smithi044 ceph-mon[32404]: pgmap v7194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:10 smithi116 ceph-mon[37097]: pgmap v7194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:10 smithi171 ceph-mon[37789]: pgmap v7194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:11.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12735.928921 secs 2024-07-26T12:13:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12735.928921 secs 2024-07-26T12:13:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12735.928921 secs 2024-07-26T12:13:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:12 smithi044 ceph-mon[32404]: pgmap v7195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:13:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:12 smithi116 ceph-mon[37097]: pgmap v7195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:13:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:12 smithi171 ceph-mon[37789]: pgmap v7195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:13:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:14.974+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:13:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:14.974+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:13:14.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:14.974+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:14.978 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:14 smithi044 ceph-mon[32404]: pgmap v7196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:14 smithi116 ceph-mon[37097]: pgmap v7196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:14 smithi171 ceph-mon[37789]: pgmap v7196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12740.929065 secs 2024-07-26T12:13:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:16 smithi044 ceph-mon[32404]: pgmap v7197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12740.929065 secs 2024-07-26T12:13:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:16 smithi116 ceph-mon[37097]: pgmap v7197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12740.929065 secs 2024-07-26T12:13:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:16 smithi171 ceph-mon[37789]: pgmap v7197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:18 smithi044 ceph-mon[32404]: pgmap v7198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:18 smithi116 ceph-mon[37097]: pgmap v7198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:18 smithi171 ceph-mon[37789]: pgmap v7198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:20 smithi044 ceph-mon[32404]: pgmap v7199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:20 smithi116 ceph-mon[37097]: pgmap v7199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:20 smithi171 ceph-mon[37789]: pgmap v7199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12745.929217 secs 2024-07-26T12:13:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12745.929217 secs 2024-07-26T12:13:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12745.929217 secs 2024-07-26T12:13:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:22 smithi044 ceph-mon[32404]: pgmap v7200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:22 smithi116 ceph-mon[37097]: pgmap v7200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:22 smithi171 ceph-mon[37789]: pgmap v7200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:24 smithi044 ceph-mon[32404]: pgmap v7201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:24 smithi116 ceph-mon[37097]: pgmap v7201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:25.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:24 smithi171 ceph-mon[37789]: pgmap v7201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12750.929360 secs 2024-07-26T12:13:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:26 smithi044 ceph-mon[32404]: pgmap v7202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12750.929360 secs 2024-07-26T12:13:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:26 smithi116 ceph-mon[37097]: pgmap v7202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12750.929360 secs 2024-07-26T12:13:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:26 smithi171 ceph-mon[37789]: pgmap v7202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:28 smithi116 ceph-mon[37097]: pgmap v7203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:28 smithi171 ceph-mon[37789]: pgmap v7203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:28 smithi044 ceph-mon[32404]: pgmap v7203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:29.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:29.976+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:13:29.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:29.976+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:13:29.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:29.976+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:30 smithi171 ceph-mon[37789]: pgmap v7204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:30 smithi044 ceph-mon[32404]: pgmap v7204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:30 smithi116 ceph-mon[37097]: pgmap v7204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12755.929501 secs 2024-07-26T12:13:32.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12755.929501 secs 2024-07-26T12:13:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12755.929501 secs 2024-07-26T12:13:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:32 smithi171 ceph-mon[37789]: pgmap v7205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:32 smithi044 ceph-mon[32404]: pgmap v7205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:32 smithi116 ceph-mon[37097]: pgmap v7205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:34 smithi171 ceph-mon[37789]: pgmap v7206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:34 smithi044 ceph-mon[32404]: pgmap v7206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:34 smithi116 ceph-mon[37097]: pgmap v7206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12760.929657 secs 2024-07-26T12:13:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:36 smithi171 ceph-mon[37789]: pgmap v7207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12760.929657 secs 2024-07-26T12:13:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:36 smithi044 ceph-mon[32404]: pgmap v7207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12760.929657 secs 2024-07-26T12:13:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:36 smithi116 ceph-mon[37097]: pgmap v7207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:38 smithi171 ceph-mon[37789]: pgmap v7208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:38 smithi044 ceph-mon[32404]: pgmap v7208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:38 smithi116 ceph-mon[37097]: pgmap v7208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:40 smithi171 ceph-mon[37789]: pgmap v7209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:40 smithi044 ceph-mon[32404]: pgmap v7209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:40 smithi116 ceph-mon[37097]: pgmap v7209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12765.929821 secs 2024-07-26T12:13:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12765.929821 secs 2024-07-26T12:13:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12765.929821 secs 2024-07-26T12:13:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:42 smithi171 ceph-mon[37789]: pgmap v7210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:42 smithi044 ceph-mon[32404]: pgmap v7210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:42 smithi116 ceph-mon[37097]: pgmap v7210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:44.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:44.981+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:13:44.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:44.981+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:13:44.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:44.981+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:44 smithi171 ceph-mon[37789]: pgmap v7211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:44 smithi044 ceph-mon[32404]: pgmap v7211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:44 smithi116 ceph-mon[37097]: pgmap v7211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12770.929980 secs 2024-07-26T12:13:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12770.929980 secs 2024-07-26T12:13:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12770.929980 secs 2024-07-26T12:13:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:46 smithi044 ceph-mon[32404]: pgmap v7212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:46 smithi116 ceph-mon[37097]: pgmap v7212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:46 smithi171 ceph-mon[37789]: pgmap v7212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:48 smithi044 ceph-mon[32404]: pgmap v7213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:48 smithi116 ceph-mon[37097]: pgmap v7213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:48 smithi171 ceph-mon[37789]: pgmap v7213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:50 smithi044 ceph-mon[32404]: pgmap v7214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12775.930128 secs 2024-07-26T12:13:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:50 smithi116 ceph-mon[37097]: pgmap v7214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12775.930128 secs 2024-07-26T12:13:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:50 smithi171 ceph-mon[37789]: pgmap v7214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:13:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12775.930128 secs 2024-07-26T12:13:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:53 smithi044 ceph-mon[32404]: pgmap v7215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:53 smithi116 ceph-mon[37097]: pgmap v7215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:53 smithi171 ceph-mon[37789]: pgmap v7215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:55 smithi044 ceph-mon[32404]: pgmap v7216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:55 smithi116 ceph-mon[37097]: pgmap v7216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:55 smithi171 ceph-mon[37789]: pgmap v7216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12780.930257 secs 2024-07-26T12:13:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12780.930257 secs 2024-07-26T12:13:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12780.930257 secs 2024-07-26T12:13:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:57 smithi044 ceph-mon[32404]: pgmap v7217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:13:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:57 smithi116 ceph-mon[37097]: pgmap v7217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:13:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:57 smithi171 ceph-mon[37789]: pgmap v7217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:13:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:13:59 smithi044 ceph-mon[32404]: pgmap v7218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:13:59 smithi116 ceph-mon[37097]: pgmap v7218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:13:59 smithi171 ceph-mon[37789]: pgmap v7218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:59.981+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:13:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:59.981+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:13:59.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:13:59.981+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:01.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:01 smithi044 ceph-mon[32404]: pgmap v7219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:01.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12785.930397 secs 2024-07-26T12:14:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:01 smithi116 ceph-mon[37097]: pgmap v7219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12785.930397 secs 2024-07-26T12:14:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:01 smithi171 ceph-mon[37789]: pgmap v7219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12785.930397 secs 2024-07-26T12:14:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:03 smithi044 ceph-mon[32404]: pgmap v7220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:03 smithi116 ceph-mon[37097]: pgmap v7220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:03 smithi171 ceph-mon[37789]: pgmap v7220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:14:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:05 smithi044 ceph-mon[32404]: pgmap v7221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:05 smithi116 ceph-mon[37097]: pgmap v7221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:05 smithi171 ceph-mon[37789]: pgmap v7221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:14:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12790.930493 secs 2024-07-26T12:14:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12790.930493 secs 2024-07-26T12:14:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12790.930493 secs 2024-07-26T12:14:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:07 smithi044 ceph-mon[32404]: pgmap v7222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:14:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:07 smithi116 ceph-mon[37097]: pgmap v7222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:14:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:07 smithi171 ceph-mon[37789]: pgmap v7222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:14:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:08 smithi044 ceph-mon[32404]: pgmap v7223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:14:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:08 smithi044 ceph-mon[32404]: pgmap v7224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:08 smithi116 ceph-mon[37097]: pgmap v7223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:14:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:08 smithi116 ceph-mon[37097]: pgmap v7224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:08 smithi171 ceph-mon[37789]: pgmap v7223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:14:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:08 smithi171 ceph-mon[37789]: pgmap v7224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:10 smithi044 ceph-mon[32404]: pgmap v7225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:10 smithi116 ceph-mon[37097]: pgmap v7225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:10 smithi171 ceph-mon[37789]: pgmap v7225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:14:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12795.930679 secs 2024-07-26T12:14:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12795.930679 secs 2024-07-26T12:14:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12795.930679 secs 2024-07-26T12:14:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:12 smithi044 ceph-mon[32404]: pgmap v7226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:14:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:12 smithi116 ceph-mon[37097]: pgmap v7226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:14:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:12 smithi171 ceph-mon[37789]: pgmap v7226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:14:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:14 smithi044 ceph-mon[32404]: pgmap v7227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T12:14:14.617 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:14 smithi171 ceph-mon[37789]: pgmap v7227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T12:14:14.617 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:14 smithi116 ceph-mon[37097]: pgmap v7227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T12:14:14.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:14.987+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:14:14.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:14.987+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:14:14.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:14.987+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:16 smithi044 ceph-mon[32404]: pgmap v7228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12800.930827 secs 2024-07-26T12:14:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:16 smithi116 ceph-mon[37097]: pgmap v7228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12800.930827 secs 2024-07-26T12:14:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:16 smithi171 ceph-mon[37789]: pgmap v7228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12800.930827 secs 2024-07-26T12:14:16.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:16.981+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:14:16.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:16.981+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:14:16.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:16.982+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:14:16.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:16.982+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:14:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:18 smithi116 ceph-mon[37097]: pgmap v7229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:14:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:18 smithi171 ceph-mon[37789]: pgmap v7229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:14:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:18 smithi044 ceph-mon[32404]: pgmap v7229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:14:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:20 smithi171 ceph-mon[37789]: pgmap v7230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:14:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:20 smithi044 ceph-mon[32404]: pgmap v7230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:14:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:20 smithi116 ceph-mon[37097]: pgmap v7230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:14:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12805.930965 secs 2024-07-26T12:14:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12805.930965 secs 2024-07-26T12:14:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12805.930965 secs 2024-07-26T12:14:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:22 smithi171 ceph-mon[37789]: pgmap v7231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:14:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:22 smithi044 ceph-mon[32404]: pgmap v7231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:14:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:22 smithi116 ceph-mon[37097]: pgmap v7231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:14:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:24 smithi171 ceph-mon[37789]: pgmap v7232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:14:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:24 smithi044 ceph-mon[32404]: pgmap v7232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:14:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:24 smithi116 ceph-mon[37097]: pgmap v7232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:14:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:26 smithi171 ceph-mon[37789]: pgmap v7233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12810.931118 secs 2024-07-26T12:14:26.722 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:26 smithi044 ceph-mon[32404]: pgmap v7233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:26.722 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12810.931118 secs 2024-07-26T12:14:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:26 smithi116 ceph-mon[37097]: pgmap v7233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:14:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12810.931118 secs 2024-07-26T12:14:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:28 smithi171 ceph-mon[37789]: pgmap v7234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:28 smithi044 ceph-mon[32404]: pgmap v7234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:28 smithi116 ceph-mon[37097]: pgmap v7234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:29.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:29.988+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:14:29.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:29.988+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:14:29.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:29.988+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:30 smithi171 ceph-mon[37789]: pgmap v7235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:30 smithi044 ceph-mon[32404]: pgmap v7235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:30 smithi116 ceph-mon[37097]: pgmap v7235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12815.931274 secs 2024-07-26T12:14:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12815.931274 secs 2024-07-26T12:14:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12815.931274 secs 2024-07-26T12:14:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:32 smithi171 ceph-mon[37789]: pgmap v7236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:14:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:32 smithi044 ceph-mon[32404]: pgmap v7236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:14:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:32 smithi116 ceph-mon[37097]: pgmap v7236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:14:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:34 smithi171 ceph-mon[37789]: pgmap v7237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:14:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:34 smithi044 ceph-mon[32404]: pgmap v7237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:14:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:34 smithi116 ceph-mon[37097]: pgmap v7237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:14:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:36 smithi044 ceph-mon[32404]: pgmap v7238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12820.931437 secs 2024-07-26T12:14:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:36 smithi116 ceph-mon[37097]: pgmap v7238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12820.931437 secs 2024-07-26T12:14:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:36 smithi171 ceph-mon[37789]: pgmap v7238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:14:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12820.931437 secs 2024-07-26T12:14:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:38 smithi044 ceph-mon[32404]: pgmap v7239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:38 smithi116 ceph-mon[37097]: pgmap v7239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:38 smithi171 ceph-mon[37789]: pgmap v7239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:40 smithi044 ceph-mon[32404]: pgmap v7240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:40 smithi116 ceph-mon[37097]: pgmap v7240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:40 smithi171 ceph-mon[37789]: pgmap v7240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:14:41.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12825.931587 secs 2024-07-26T12:14:41.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12825.931587 secs 2024-07-26T12:14:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12825.931587 secs 2024-07-26T12:14:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:42 smithi044 ceph-mon[32404]: pgmap v7241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:42 smithi116 ceph-mon[37097]: pgmap v7241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:42 smithi171 ceph-mon[37789]: pgmap v7241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:44 smithi044 ceph-mon[32404]: pgmap v7242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:44 smithi116 ceph-mon[37097]: pgmap v7242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:44 smithi171 ceph-mon[37789]: pgmap v7242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:44.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:44.993+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:14:44.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:44.993+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:14:44.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:44.993+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:46 smithi044 ceph-mon[32404]: pgmap v7243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12830.931750 secs 2024-07-26T12:14:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:46 smithi116 ceph-mon[37097]: pgmap v7243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12830.931750 secs 2024-07-26T12:14:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:46 smithi171 ceph-mon[37789]: pgmap v7243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12830.931750 secs 2024-07-26T12:14:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:48 smithi044 ceph-mon[32404]: pgmap v7244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:48 smithi116 ceph-mon[37097]: pgmap v7244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:48 smithi171 ceph-mon[37789]: pgmap v7244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:50 smithi044 ceph-mon[32404]: pgmap v7245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:50 smithi116 ceph-mon[37097]: pgmap v7245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:50 smithi171 ceph-mon[37789]: pgmap v7245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12835.931897 secs 2024-07-26T12:14:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12835.931897 secs 2024-07-26T12:14:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12835.931897 secs 2024-07-26T12:14:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:52 smithi044 ceph-mon[32404]: pgmap v7246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:52 smithi116 ceph-mon[37097]: pgmap v7246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:52 smithi171 ceph-mon[37789]: pgmap v7246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:54 smithi044 ceph-mon[32404]: pgmap v7247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:54 smithi116 ceph-mon[37097]: pgmap v7247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:54 smithi171 ceph-mon[37789]: pgmap v7247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:55 smithi044 ceph-mon[32404]: pgmap v7248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:55 smithi116 ceph-mon[37097]: pgmap v7248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:55 smithi171 ceph-mon[37789]: pgmap v7248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12840.932067 secs 2024-07-26T12:14:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12840.932067 secs 2024-07-26T12:14:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12840.932067 secs 2024-07-26T12:14:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:14:57 smithi044 ceph-mon[32404]: pgmap v7249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:14:57 smithi116 ceph-mon[37097]: pgmap v7249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:14:57 smithi171 ceph-mon[37789]: pgmap v7249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:14:59.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:59.993+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:14:59.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:59.993+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:14:59.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:14:59.993+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:00 smithi044 ceph-mon[32404]: pgmap v7250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:00 smithi116 ceph-mon[37097]: pgmap v7250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:00 smithi171 ceph-mon[37789]: pgmap v7250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12845.932234 secs 2024-07-26T12:15:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12845.932234 secs 2024-07-26T12:15:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12845.932234 secs 2024-07-26T12:15:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:02 smithi171 ceph-mon[37789]: pgmap v7251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:02 smithi044 ceph-mon[32404]: pgmap v7251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:02 smithi116 ceph-mon[37097]: pgmap v7251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:04.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:04 smithi116 ceph-mon[37097]: pgmap v7252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:04 smithi171 ceph-mon[37789]: pgmap v7252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:04 smithi044 ceph-mon[32404]: pgmap v7252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:06 smithi171 ceph-mon[37789]: pgmap v7253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12850.932375 secs 2024-07-26T12:15:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:06 smithi044 ceph-mon[32404]: pgmap v7253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12850.932375 secs 2024-07-26T12:15:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:06 smithi116 ceph-mon[37097]: pgmap v7253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12850.932375 secs 2024-07-26T12:15:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:08 smithi171 ceph-mon[37789]: pgmap v7254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:08 smithi044 ceph-mon[32404]: pgmap v7254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:08 smithi116 ceph-mon[37097]: pgmap v7254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:09 smithi171 ceph-mon[37789]: pgmap v7255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:09 smithi171 ceph-mon[37789]: pgmap v7256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:15:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:09 smithi044 ceph-mon[32404]: pgmap v7255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:09 smithi044 ceph-mon[32404]: pgmap v7256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:15:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:09 smithi116 ceph-mon[37097]: pgmap v7255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:09 smithi116 ceph-mon[37097]: pgmap v7256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:15:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:11 smithi044 ceph-mon[32404]: pgmap v7257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12855.932529 secs 2024-07-26T12:15:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:11 smithi116 ceph-mon[37097]: pgmap v7257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12855.932529 secs 2024-07-26T12:15:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:11 smithi171 ceph-mon[37789]: pgmap v7257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12855.932529 secs 2024-07-26T12:15:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:13 smithi044 ceph-mon[32404]: pgmap v7258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:13 smithi116 ceph-mon[37097]: pgmap v7258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:13 smithi171 ceph-mon[37789]: pgmap v7258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:15.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:14.998+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:15:15.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:14.998+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:15:15.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:14.998+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:15 smithi044 ceph-mon[32404]: pgmap v7259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:15 smithi116 ceph-mon[37097]: pgmap v7259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:15 smithi171 ceph-mon[37789]: pgmap v7259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12860.932708 secs 2024-07-26T12:15:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12860.932708 secs 2024-07-26T12:15:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12860.932708 secs 2024-07-26T12:15:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:17 smithi116 ceph-mon[37097]: pgmap v7260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:17 smithi171 ceph-mon[37789]: pgmap v7260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:17 smithi044 ceph-mon[32404]: pgmap v7260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:19 smithi171 ceph-mon[37789]: pgmap v7261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:15:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:19 smithi044 ceph-mon[32404]: pgmap v7261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:15:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:19 smithi116 ceph-mon[37097]: pgmap v7261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:15:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:21 smithi171 ceph-mon[37789]: pgmap v7262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12865.932839 secs 2024-07-26T12:15:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:21 smithi044 ceph-mon[32404]: pgmap v7262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12865.932839 secs 2024-07-26T12:15:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:21 smithi116 ceph-mon[37097]: pgmap v7262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12865.932839 secs 2024-07-26T12:15:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:23 smithi171 ceph-mon[37789]: pgmap v7263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:23 smithi044 ceph-mon[32404]: pgmap v7263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:23 smithi116 ceph-mon[37097]: pgmap v7263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:25 smithi171 ceph-mon[37789]: pgmap v7264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:25 smithi044 ceph-mon[32404]: pgmap v7264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:25 smithi116 ceph-mon[37097]: pgmap v7264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12870.932940 secs 2024-07-26T12:15:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12870.932940 secs 2024-07-26T12:15:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12870.932940 secs 2024-07-26T12:15:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:27 smithi171 ceph-mon[37789]: pgmap v7265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:27 smithi044 ceph-mon[32404]: pgmap v7265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:27 smithi116 ceph-mon[37097]: pgmap v7265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:29 smithi171 ceph-mon[37789]: pgmap v7266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:29.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:29 smithi044 ceph-mon[32404]: pgmap v7266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:29 smithi116 ceph-mon[37097]: pgmap v7266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:30.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:29.999+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:15:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:29.999+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:30.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:29.999+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:15:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:31 smithi171 ceph-mon[37789]: pgmap v7267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12875.933062 secs 2024-07-26T12:15:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:31 smithi044 ceph-mon[32404]: pgmap v7267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12875.933062 secs 2024-07-26T12:15:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:31 smithi116 ceph-mon[37097]: pgmap v7267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12875.933062 secs 2024-07-26T12:15:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:33 smithi171 ceph-mon[37789]: pgmap v7268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:33 smithi044 ceph-mon[32404]: pgmap v7268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:33 smithi116 ceph-mon[37097]: pgmap v7268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:35 smithi171 ceph-mon[37789]: pgmap v7269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:35 smithi044 ceph-mon[32404]: pgmap v7269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:35 smithi116 ceph-mon[37097]: pgmap v7269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12880.933209 secs 2024-07-26T12:15:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12880.933209 secs 2024-07-26T12:15:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12880.933209 secs 2024-07-26T12:15:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:37 smithi044 ceph-mon[32404]: pgmap v7270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:37 smithi116 ceph-mon[37097]: pgmap v7270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:37 smithi171 ceph-mon[37789]: pgmap v7270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:39 smithi044 ceph-mon[32404]: pgmap v7271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:39 smithi116 ceph-mon[37097]: pgmap v7271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:39 smithi171 ceph-mon[37789]: pgmap v7271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:41.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:41 smithi171 ceph-mon[37789]: pgmap v7272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:41.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12885.933372 secs 2024-07-26T12:15:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:41 smithi044 ceph-mon[32404]: pgmap v7272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12885.933372 secs 2024-07-26T12:15:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:41 smithi116 ceph-mon[37097]: pgmap v7272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12885.933372 secs 2024-07-26T12:15:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:43 smithi044 ceph-mon[32404]: pgmap v7273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:43 smithi116 ceph-mon[37097]: pgmap v7273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:43 smithi171 ceph-mon[37789]: pgmap v7273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:45.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:45.005+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:15:45.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:45.005+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:15:45.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:45.005+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:45 smithi044 ceph-mon[32404]: pgmap v7274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:45 smithi116 ceph-mon[37097]: pgmap v7274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:45 smithi171 ceph-mon[37789]: pgmap v7274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12890.933478 secs 2024-07-26T12:15:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12890.933478 secs 2024-07-26T12:15:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12890.933478 secs 2024-07-26T12:15:46.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:46.998+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:15:47.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:46.998+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:15:47.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:46.999+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:15:47.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:15:46.999+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:15:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:47 smithi044 ceph-mon[32404]: pgmap v7275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:47 smithi116 ceph-mon[37097]: pgmap v7275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:47 smithi171 ceph-mon[37789]: pgmap v7275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:49 smithi044 ceph-mon[32404]: pgmap v7276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:49 smithi116 ceph-mon[37097]: pgmap v7276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:49 smithi171 ceph-mon[37789]: pgmap v7276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:51 smithi044 ceph-mon[32404]: pgmap v7277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12895.933687 secs 2024-07-26T12:15:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:51 smithi116 ceph-mon[37097]: pgmap v7277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12895.933687 secs 2024-07-26T12:15:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:51 smithi171 ceph-mon[37789]: pgmap v7277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12895.933687 secs 2024-07-26T12:15:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:53 smithi044 ceph-mon[32404]: pgmap v7278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:53 smithi116 ceph-mon[37097]: pgmap v7278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:53 smithi171 ceph-mon[37789]: pgmap v7278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:55 smithi044 ceph-mon[32404]: pgmap v7279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:55 smithi116 ceph-mon[37097]: pgmap v7279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:55 smithi171 ceph-mon[37789]: pgmap v7279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:15:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12900.933846 secs 2024-07-26T12:15:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12900.933846 secs 2024-07-26T12:15:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12900.933846 secs 2024-07-26T12:15:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:57 smithi044 ceph-mon[32404]: pgmap v7280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:57 smithi116 ceph-mon[37097]: pgmap v7280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:57 smithi171 ceph-mon[37789]: pgmap v7280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:15:59 smithi044 ceph-mon[32404]: pgmap v7281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:15:59 smithi116 ceph-mon[37097]: pgmap v7281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:15:59 smithi171 ceph-mon[37789]: pgmap v7281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:00.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:00.006+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:16:00.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:00.006+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:16:00.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:00.006+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:01 smithi044 ceph-mon[32404]: pgmap v7282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12905.934014 secs 2024-07-26T12:16:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:01 smithi116 ceph-mon[37097]: pgmap v7282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12905.934014 secs 2024-07-26T12:16:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:01 smithi171 ceph-mon[37789]: pgmap v7282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12905.934014 secs 2024-07-26T12:16:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:02 smithi044 ceph-mon[32404]: pgmap v7283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:02 smithi116 ceph-mon[37097]: pgmap v7283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:02 smithi171 ceph-mon[37789]: pgmap v7283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:05 smithi171 ceph-mon[37789]: pgmap v7284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:05 smithi044 ceph-mon[32404]: pgmap v7284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:05 smithi116 ceph-mon[37097]: pgmap v7284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12910.934160 secs 2024-07-26T12:16:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12910.934160 secs 2024-07-26T12:16:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12910.934160 secs 2024-07-26T12:16:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:07 smithi171 ceph-mon[37789]: pgmap v7285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:07 smithi044 ceph-mon[32404]: pgmap v7285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:07 smithi116 ceph-mon[37097]: pgmap v7285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:09 smithi171 ceph-mon[37789]: pgmap v7286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:09 smithi044 ceph-mon[32404]: pgmap v7286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:09 smithi116 ceph-mon[37097]: pgmap v7286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:10 smithi171 ceph-mon[37789]: pgmap v7287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:10 smithi171 ceph-mon[37789]: pgmap v7288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:10.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:16:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:10 smithi044 ceph-mon[32404]: pgmap v7287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:10 smithi044 ceph-mon[32404]: pgmap v7288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:16:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:10 smithi116 ceph-mon[37097]: pgmap v7287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:10 smithi116 ceph-mon[37097]: pgmap v7288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:10.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:16:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12915.934270 secs 2024-07-26T12:16:11.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12915.934270 secs 2024-07-26T12:16:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12915.934270 secs 2024-07-26T12:16:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:12 smithi171 ceph-mon[37789]: pgmap v7289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:12 smithi044 ceph-mon[32404]: pgmap v7289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:12 smithi116 ceph-mon[37097]: pgmap v7289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:13 smithi044 ceph-mon[32404]: pgmap v7290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:13 smithi116 ceph-mon[37097]: pgmap v7290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:13 smithi171 ceph-mon[37789]: pgmap v7290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:15.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:15.010+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:16:15.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:15.010+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:16:15.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:15.010+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:15 smithi044 ceph-mon[32404]: pgmap v7291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:15 smithi116 ceph-mon[37097]: pgmap v7291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:15 smithi171 ceph-mon[37789]: pgmap v7291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:16:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12920.934436 secs 2024-07-26T12:16:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12920.934436 secs 2024-07-26T12:16:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12920.934436 secs 2024-07-26T12:16:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:17 smithi044 ceph-mon[32404]: pgmap v7292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:17 smithi116 ceph-mon[37097]: pgmap v7292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:17 smithi171 ceph-mon[37789]: pgmap v7292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:19 smithi044 ceph-mon[32404]: pgmap v7293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:16:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:19 smithi116 ceph-mon[37097]: pgmap v7293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:16:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:19 smithi171 ceph-mon[37789]: pgmap v7293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:16:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12925.934589 secs 2024-07-26T12:16:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12925.934589 secs 2024-07-26T12:16:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12925.934589 secs 2024-07-26T12:16:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:21 smithi044 ceph-mon[32404]: pgmap v7294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:21 smithi116 ceph-mon[37097]: pgmap v7294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:21 smithi171 ceph-mon[37789]: pgmap v7294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:23 smithi044 ceph-mon[32404]: pgmap v7295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:23 smithi116 ceph-mon[37097]: pgmap v7295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:23 smithi171 ceph-mon[37789]: pgmap v7295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:25 smithi044 ceph-mon[32404]: pgmap v7296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:25 smithi116 ceph-mon[37097]: pgmap v7296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:25 smithi171 ceph-mon[37789]: pgmap v7296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12930.934727 secs 2024-07-26T12:16:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12930.934727 secs 2024-07-26T12:16:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12930.934727 secs 2024-07-26T12:16:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:27 smithi044 ceph-mon[32404]: pgmap v7297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:27 smithi116 ceph-mon[37097]: pgmap v7297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:27 smithi171 ceph-mon[37789]: pgmap v7297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:29 smithi044 ceph-mon[32404]: pgmap v7298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:29 smithi116 ceph-mon[37097]: pgmap v7298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:29 smithi171 ceph-mon[37789]: pgmap v7298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:30.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:30.011+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:16:30.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:30.011+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:16:30.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:30.011+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12935.934863 secs 2024-07-26T12:16:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12935.934863 secs 2024-07-26T12:16:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12935.934863 secs 2024-07-26T12:16:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:31 smithi044 ceph-mon[32404]: pgmap v7299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:31 smithi116 ceph-mon[37097]: pgmap v7299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:31 smithi171 ceph-mon[37789]: pgmap v7299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:33 smithi044 ceph-mon[32404]: pgmap v7300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:33 smithi116 ceph-mon[37097]: pgmap v7300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:33 smithi171 ceph-mon[37789]: pgmap v7300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:35 smithi044 ceph-mon[32404]: pgmap v7301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:35 smithi116 ceph-mon[37097]: pgmap v7301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:35 smithi171 ceph-mon[37789]: pgmap v7301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12940.935008 secs 2024-07-26T12:16:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12940.935008 secs 2024-07-26T12:16:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12940.935008 secs 2024-07-26T12:16:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:37 smithi044 ceph-mon[32404]: pgmap v7302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:37 smithi116 ceph-mon[37097]: pgmap v7302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:37 smithi171 ceph-mon[37789]: pgmap v7302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:39 smithi044 ceph-mon[32404]: pgmap v7303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:39 smithi116 ceph-mon[37097]: pgmap v7303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:39 smithi171 ceph-mon[37789]: pgmap v7303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12945.935191 secs 2024-07-26T12:16:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12945.935191 secs 2024-07-26T12:16:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12945.935191 secs 2024-07-26T12:16:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:41 smithi044 ceph-mon[32404]: pgmap v7304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:41 smithi116 ceph-mon[37097]: pgmap v7304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:41 smithi171 ceph-mon[37789]: pgmap v7304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:44 smithi044 ceph-mon[32404]: pgmap v7305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:44 smithi116 ceph-mon[37097]: pgmap v7305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:44 smithi171 ceph-mon[37789]: pgmap v7305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:45.017+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:16:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:45.017+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:16:45.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:16:45.017+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:46 smithi044 ceph-mon[32404]: pgmap v7306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12950.935368 secs 2024-07-26T12:16:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:46 smithi116 ceph-mon[37097]: pgmap v7306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12950.935368 secs 2024-07-26T12:16:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:46 smithi171 ceph-mon[37789]: pgmap v7306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12950.935368 secs 2024-07-26T12:16:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:48 smithi044 ceph-mon[32404]: pgmap v7307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:48 smithi116 ceph-mon[37097]: pgmap v7307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:48 smithi171 ceph-mon[37789]: pgmap v7307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:50 smithi044 ceph-mon[32404]: pgmap v7308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:50 smithi116 ceph-mon[37097]: pgmap v7308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:50 smithi171 ceph-mon[37789]: pgmap v7308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12955.935535 secs 2024-07-26T12:16:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12955.935535 secs 2024-07-26T12:16:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12955.935535 secs 2024-07-26T12:16:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:52 smithi171 ceph-mon[37789]: pgmap v7309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:52 smithi044 ceph-mon[32404]: pgmap v7309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:52 smithi116 ceph-mon[37097]: pgmap v7309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:54 smithi171 ceph-mon[37789]: pgmap v7310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:54 smithi044 ceph-mon[32404]: pgmap v7310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:54 smithi116 ceph-mon[37097]: pgmap v7310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:16:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:56 smithi171 ceph-mon[37789]: pgmap v7311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12960.935693 secs 2024-07-26T12:16:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:56 smithi044 ceph-mon[32404]: pgmap v7311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12960.935693 secs 2024-07-26T12:16:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:56 smithi116 ceph-mon[37097]: pgmap v7311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12960.935693 secs 2024-07-26T12:16:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:16:58 smithi171 ceph-mon[37789]: pgmap v7312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:16:58 smithi044 ceph-mon[32404]: pgmap v7312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:16:58 smithi116 ceph-mon[37097]: pgmap v7312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:00.017+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:17:00.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:00.017+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:17:00.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:00.017+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:00 smithi171 ceph-mon[37789]: pgmap v7313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:00 smithi044 ceph-mon[32404]: pgmap v7313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:00 smithi116 ceph-mon[37097]: pgmap v7313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12965.935860 secs 2024-07-26T12:17:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12965.935860 secs 2024-07-26T12:17:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12965.935860 secs 2024-07-26T12:17:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:02 smithi171 ceph-mon[37789]: pgmap v7314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:02.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:02 smithi044 ceph-mon[32404]: pgmap v7314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:02 smithi116 ceph-mon[37097]: pgmap v7314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:04 smithi171 ceph-mon[37789]: pgmap v7315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:17:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:04 smithi044 ceph-mon[32404]: pgmap v7315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:17:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:04 smithi116 ceph-mon[37097]: pgmap v7315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:17:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:06 smithi171 ceph-mon[37789]: pgmap v7316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12970.936009 secs 2024-07-26T12:17:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:06 smithi044 ceph-mon[32404]: pgmap v7316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12970.936009 secs 2024-07-26T12:17:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:06 smithi116 ceph-mon[37097]: pgmap v7316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12970.936009 secs 2024-07-26T12:17:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:08 smithi171 ceph-mon[37789]: pgmap v7317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:08 smithi044 ceph-mon[32404]: pgmap v7317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:08 smithi116 ceph-mon[37097]: pgmap v7317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:10 smithi044 ceph-mon[32404]: pgmap v7318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:10 smithi116 ceph-mon[37097]: pgmap v7318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:10 smithi171 ceph-mon[37789]: pgmap v7318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:11.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12975.936114 secs 2024-07-26T12:17:11.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:11.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:11.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:11 smithi044 ceph-mon[32404]: pgmap v7319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:17:11.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:11 smithi044 ceph-mon[32404]: pgmap v7320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:11.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:17:11.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12975.936114 secs 2024-07-26T12:17:11.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:11.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:11.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:11 smithi171 ceph-mon[37789]: pgmap v7319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:17:11.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:11 smithi171 ceph-mon[37789]: pgmap v7320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:11.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:17:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12975.936114 secs 2024-07-26T12:17:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:11 smithi116 ceph-mon[37097]: pgmap v7319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:17:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:11 smithi116 ceph-mon[37097]: pgmap v7320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:17:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:13 smithi044 ceph-mon[32404]: pgmap v7321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:17:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:13 smithi116 ceph-mon[37097]: pgmap v7321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:17:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:13 smithi171 ceph-mon[37789]: pgmap v7321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:17:15.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:15.020+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:17:15.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:15.020+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:17:15.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:15.020+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:15 smithi044 ceph-mon[32404]: pgmap v7322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:15 smithi116 ceph-mon[37097]: pgmap v7322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:15 smithi171 ceph-mon[37789]: pgmap v7322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:17.014+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:17:17.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:17.014+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:17:17.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:17.015+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:17:17.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:17.016+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:17:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12980.936260 secs 2024-07-26T12:17:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12980.936260 secs 2024-07-26T12:17:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12980.936260 secs 2024-07-26T12:17:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:17 smithi171 ceph-mon[37789]: pgmap v7323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:17:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:17 smithi044 ceph-mon[32404]: pgmap v7323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:17:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:17 smithi116 ceph-mon[37097]: pgmap v7323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:17:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:19 smithi171 ceph-mon[37789]: pgmap v7324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:19 smithi044 ceph-mon[32404]: pgmap v7324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:19 smithi116 ceph-mon[37097]: pgmap v7324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:17:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12985.936424 secs 2024-07-26T12:17:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:21 smithi171 ceph-mon[37789]: pgmap v7325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12985.936424 secs 2024-07-26T12:17:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:21 smithi044 ceph-mon[32404]: pgmap v7325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12985.936424 secs 2024-07-26T12:17:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:21 smithi116 ceph-mon[37097]: pgmap v7325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:23 smithi171 ceph-mon[37789]: pgmap v7326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:23 smithi044 ceph-mon[32404]: pgmap v7326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:23 smithi116 ceph-mon[37097]: pgmap v7326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:25 smithi171 ceph-mon[37789]: pgmap v7327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:25 smithi044 ceph-mon[32404]: pgmap v7327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:25 smithi116 ceph-mon[37097]: pgmap v7327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12990.936587 secs 2024-07-26T12:17:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12990.936587 secs 2024-07-26T12:17:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12990.936587 secs 2024-07-26T12:17:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:27 smithi171 ceph-mon[37789]: pgmap v7328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:27 smithi044 ceph-mon[32404]: pgmap v7328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:27 smithi116 ceph-mon[37097]: pgmap v7328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:30.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:30.021+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:17:30.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:30.021+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:30.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:30.021+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:17:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:29 smithi171 ceph-mon[37789]: pgmap v7329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:29 smithi044 ceph-mon[32404]: pgmap v7329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:29 smithi116 ceph-mon[37097]: pgmap v7329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 12995.936738 secs 2024-07-26T12:17:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:31 smithi171 ceph-mon[37789]: pgmap v7330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 12995.936738 secs 2024-07-26T12:17:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:31 smithi044 ceph-mon[32404]: pgmap v7330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 12995.936738 secs 2024-07-26T12:17:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:31 smithi116 ceph-mon[37097]: pgmap v7330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:33 smithi044 ceph-mon[32404]: pgmap v7331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:33 smithi171 ceph-mon[37789]: pgmap v7331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:33 smithi116 ceph-mon[37097]: pgmap v7331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:35 smithi044 ceph-mon[32404]: pgmap v7332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:35 smithi116 ceph-mon[37097]: pgmap v7332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:35 smithi171 ceph-mon[37789]: pgmap v7332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13000.936902 secs 2024-07-26T12:17:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13000.936902 secs 2024-07-26T12:17:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13000.936902 secs 2024-07-26T12:17:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:37 smithi044 ceph-mon[32404]: pgmap v7333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:37 smithi116 ceph-mon[37097]: pgmap v7333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:37 smithi171 ceph-mon[37789]: pgmap v7333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:39 smithi044 ceph-mon[32404]: pgmap v7334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:39 smithi116 ceph-mon[37097]: pgmap v7334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:39 smithi171 ceph-mon[37789]: pgmap v7334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:42.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13005.937066 secs 2024-07-26T12:17:42.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:41 smithi171 ceph-mon[37789]: pgmap v7335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13005.937066 secs 2024-07-26T12:17:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:41 smithi044 ceph-mon[32404]: pgmap v7335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13005.937066 secs 2024-07-26T12:17:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:41 smithi116 ceph-mon[37097]: pgmap v7335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:43 smithi044 ceph-mon[32404]: pgmap v7336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:43 smithi116 ceph-mon[37097]: pgmap v7336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:43 smithi171 ceph-mon[37789]: pgmap v7336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:45.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:45.027+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:17:45.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:45.027+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:17:45.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:17:45.027+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:45 smithi044 ceph-mon[32404]: pgmap v7337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13010.937230 secs 2024-07-26T12:17:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:45 smithi116 ceph-mon[37097]: pgmap v7337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13010.937230 secs 2024-07-26T12:17:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:45 smithi171 ceph-mon[37789]: pgmap v7337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13010.937230 secs 2024-07-26T12:17:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:47 smithi044 ceph-mon[32404]: pgmap v7338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:47 smithi116 ceph-mon[37097]: pgmap v7338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:47 smithi171 ceph-mon[37789]: pgmap v7338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:49 smithi044 ceph-mon[32404]: pgmap v7339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:49 smithi116 ceph-mon[37097]: pgmap v7339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:49 smithi171 ceph-mon[37789]: pgmap v7339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:17:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13015.937374 secs 2024-07-26T12:17:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13015.937374 secs 2024-07-26T12:17:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13015.937374 secs 2024-07-26T12:17:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:51 smithi044 ceph-mon[32404]: pgmap v7340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:51 smithi116 ceph-mon[37097]: pgmap v7340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:51 smithi171 ceph-mon[37789]: pgmap v7340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:53 smithi044 ceph-mon[32404]: pgmap v7341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:53 smithi116 ceph-mon[37097]: pgmap v7341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:53 smithi171 ceph-mon[37789]: pgmap v7341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:55 smithi044 ceph-mon[32404]: pgmap v7342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13020.937478 secs 2024-07-26T12:17:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:55 smithi116 ceph-mon[37097]: pgmap v7342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13020.937478 secs 2024-07-26T12:17:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:55 smithi171 ceph-mon[37789]: pgmap v7342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:17:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13020.937478 secs 2024-07-26T12:17:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:57 smithi044 ceph-mon[32404]: pgmap v7343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:57 smithi116 ceph-mon[37097]: pgmap v7343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:17:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:57 smithi171 ceph-mon[37789]: pgmap v7343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:18:00.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:00.028+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:18:00.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:00.028+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:00.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:00.028+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:18:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:17:59 smithi044 ceph-mon[32404]: pgmap v7344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:18:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:17:59 smithi116 ceph-mon[37097]: pgmap v7344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:18:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:17:59 smithi171 ceph-mon[37789]: pgmap v7344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:18:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13025.937697 secs 2024-07-26T12:18:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13025.937697 secs 2024-07-26T12:18:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13025.937697 secs 2024-07-26T12:18:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:01 smithi044 ceph-mon[32404]: pgmap v7345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:01 smithi116 ceph-mon[37097]: pgmap v7345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:01 smithi171 ceph-mon[37789]: pgmap v7345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:03 smithi044 ceph-mon[32404]: pgmap v7346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:03 smithi116 ceph-mon[37097]: pgmap v7346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:03 smithi171 ceph-mon[37789]: pgmap v7346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:05 smithi044 ceph-mon[32404]: pgmap v7347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13030.937839 secs 2024-07-26T12:18:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:05 smithi116 ceph-mon[37097]: pgmap v7347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13030.937839 secs 2024-07-26T12:18:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:05 smithi171 ceph-mon[37789]: pgmap v7347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:18:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13030.937839 secs 2024-07-26T12:18:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:07 smithi044 ceph-mon[32404]: pgmap v7348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:18:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:07 smithi116 ceph-mon[37097]: pgmap v7348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:18:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:07 smithi171 ceph-mon[37789]: pgmap v7348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:18:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:09 smithi044 ceph-mon[32404]: pgmap v7349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:18:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:09 smithi116 ceph-mon[37097]: pgmap v7349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:18:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:09 smithi171 ceph-mon[37789]: pgmap v7349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:18:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13035.938008 secs 2024-07-26T12:18:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13035.938008 secs 2024-07-26T12:18:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13035.938008 secs 2024-07-26T12:18:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:12.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: pgmap v7350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:12.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: pgmap v7350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: pgmap v7350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:18:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:18:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:18:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: pgmap v7351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:18:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: pgmap v7352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:13.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:18:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:18:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:18:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: pgmap v7351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:18:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: pgmap v7352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:13.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:18:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:18:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:18:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: pgmap v7351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:18:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: pgmap v7352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:18:15.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:15.033+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:18:15.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:15.033+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:15.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:15.033+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:18:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:15 smithi171 ceph-mon[37789]: pgmap v7353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:15 smithi044 ceph-mon[32404]: pgmap v7353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:15 smithi116 ceph-mon[37097]: pgmap v7353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13040.938177 secs 2024-07-26T12:18:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13040.938177 secs 2024-07-26T12:18:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13040.938177 secs 2024-07-26T12:18:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:17 smithi171 ceph-mon[37789]: pgmap v7354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:18:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:17 smithi044 ceph-mon[32404]: pgmap v7354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:18:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:17 smithi116 ceph-mon[37097]: pgmap v7354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:18:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:19 smithi171 ceph-mon[37789]: pgmap v7355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:19 smithi044 ceph-mon[32404]: pgmap v7355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:19 smithi116 ceph-mon[37097]: pgmap v7355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:21 smithi171 ceph-mon[37789]: pgmap v7356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13045.938344 secs 2024-07-26T12:18:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:21 smithi044 ceph-mon[32404]: pgmap v7356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13045.938344 secs 2024-07-26T12:18:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:21 smithi116 ceph-mon[37097]: pgmap v7356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13045.938344 secs 2024-07-26T12:18:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:23 smithi044 ceph-mon[32404]: pgmap v7357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:23 smithi116 ceph-mon[37097]: pgmap v7357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:23 smithi171 ceph-mon[37789]: pgmap v7357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:25 smithi044 ceph-mon[32404]: pgmap v7358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:25 smithi116 ceph-mon[37097]: pgmap v7358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:25 smithi171 ceph-mon[37789]: pgmap v7358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13050.938493 secs 2024-07-26T12:18:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13050.938493 secs 2024-07-26T12:18:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13050.938493 secs 2024-07-26T12:18:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:27 smithi044 ceph-mon[32404]: pgmap v7359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:27 smithi116 ceph-mon[37097]: pgmap v7359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:27 smithi171 ceph-mon[37789]: pgmap v7359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:29 smithi044 ceph-mon[32404]: pgmap v7360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:29 smithi116 ceph-mon[37097]: pgmap v7360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:29 smithi171 ceph-mon[37789]: pgmap v7360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:30.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:30.035+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:18:30.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:30.035+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:30.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:30.035+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:18:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:31 smithi044 ceph-mon[32404]: pgmap v7361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13055.938696 secs 2024-07-26T12:18:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:31 smithi116 ceph-mon[37097]: pgmap v7361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13055.938696 secs 2024-07-26T12:18:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:31 smithi171 ceph-mon[37789]: pgmap v7361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13055.938696 secs 2024-07-26T12:18:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:33 smithi044 ceph-mon[32404]: pgmap v7362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:33 smithi116 ceph-mon[37097]: pgmap v7362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:33 smithi171 ceph-mon[37789]: pgmap v7362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:35 smithi044 ceph-mon[32404]: pgmap v7363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:35 smithi116 ceph-mon[37097]: pgmap v7363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:35 smithi171 ceph-mon[37789]: pgmap v7363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:36.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13060.938859 secs 2024-07-26T12:18:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13060.938859 secs 2024-07-26T12:18:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13060.938859 secs 2024-07-26T12:18:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:37 smithi044 ceph-mon[32404]: pgmap v7364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:37 smithi116 ceph-mon[37097]: pgmap v7364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:37 smithi171 ceph-mon[37789]: pgmap v7364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:39 smithi044 ceph-mon[32404]: pgmap v7365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:39 smithi116 ceph-mon[37097]: pgmap v7365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:39 smithi171 ceph-mon[37789]: pgmap v7365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:41 smithi044 ceph-mon[32404]: pgmap v7366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13065.939014 secs 2024-07-26T12:18:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:41 smithi116 ceph-mon[37097]: pgmap v7366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13065.939014 secs 2024-07-26T12:18:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:41 smithi171 ceph-mon[37789]: pgmap v7366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13065.939014 secs 2024-07-26T12:18:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:43 smithi044 ceph-mon[32404]: pgmap v7367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:43 smithi116 ceph-mon[37097]: pgmap v7367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:43 smithi171 ceph-mon[37789]: pgmap v7367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:45.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:45.041+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:18:45.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:45.041+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:18:45.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:18:45.041+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:45 smithi044 ceph-mon[32404]: pgmap v7368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:45 smithi116 ceph-mon[37097]: pgmap v7368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:45 smithi171 ceph-mon[37789]: pgmap v7368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13070.939184 secs 2024-07-26T12:18:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13070.939184 secs 2024-07-26T12:18:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13070.939184 secs 2024-07-26T12:18:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:47 smithi044 ceph-mon[32404]: pgmap v7369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:47 smithi116 ceph-mon[37097]: pgmap v7369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:47 smithi171 ceph-mon[37789]: pgmap v7369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:49 smithi044 ceph-mon[32404]: pgmap v7370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:49 smithi116 ceph-mon[37097]: pgmap v7370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:49 smithi171 ceph-mon[37789]: pgmap v7370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:51 smithi044 ceph-mon[32404]: pgmap v7371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13075.939336 secs 2024-07-26T12:18:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:51 smithi116 ceph-mon[37097]: pgmap v7371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13075.939336 secs 2024-07-26T12:18:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:51 smithi171 ceph-mon[37789]: pgmap v7371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13075.939336 secs 2024-07-26T12:18:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:53 smithi171 ceph-mon[37789]: pgmap v7372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:53 smithi044 ceph-mon[32404]: pgmap v7372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:53 smithi116 ceph-mon[37097]: pgmap v7372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:55 smithi171 ceph-mon[37789]: pgmap v7373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:55 smithi044 ceph-mon[32404]: pgmap v7373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:55 smithi116 ceph-mon[37097]: pgmap v7373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:18:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13080.939433 secs 2024-07-26T12:18:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13080.939433 secs 2024-07-26T12:18:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13080.939433 secs 2024-07-26T12:18:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:57 smithi171 ceph-mon[37789]: pgmap v7374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:57 smithi044 ceph-mon[32404]: pgmap v7374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:57 smithi116 ceph-mon[37097]: pgmap v7374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:18:59 smithi171 ceph-mon[37789]: pgmap v7375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:18:59 smithi044 ceph-mon[32404]: pgmap v7375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:18:59 smithi116 ceph-mon[37097]: pgmap v7375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:00.042+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:19:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:00.042+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:00.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:00.042+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:19:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:01 smithi044 ceph-mon[32404]: pgmap v7376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13085.939601 secs 2024-07-26T12:19:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:01 smithi116 ceph-mon[37097]: pgmap v7376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13085.939601 secs 2024-07-26T12:19:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:01 smithi171 ceph-mon[37789]: pgmap v7376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13085.939601 secs 2024-07-26T12:19:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:03 smithi044 ceph-mon[32404]: pgmap v7377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:03 smithi116 ceph-mon[37097]: pgmap v7377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:03 smithi171 ceph-mon[37789]: pgmap v7377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:05 smithi044 ceph-mon[32404]: pgmap v7378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:05 smithi116 ceph-mon[37097]: pgmap v7378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:05 smithi171 ceph-mon[37789]: pgmap v7378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:06.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13090.939747 secs 2024-07-26T12:19:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13090.939747 secs 2024-07-26T12:19:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13090.939747 secs 2024-07-26T12:19:07.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:07 smithi044 ceph-mon[32404]: pgmap v7379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:07 smithi116 ceph-mon[37097]: pgmap v7379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:07 smithi171 ceph-mon[37789]: pgmap v7379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:09 smithi044 ceph-mon[32404]: pgmap v7380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:19:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:09 smithi116 ceph-mon[37097]: pgmap v7380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:19:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:09 smithi171 ceph-mon[37789]: pgmap v7380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:19:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:11 smithi044 ceph-mon[32404]: pgmap v7381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:19:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13095.939913 secs 2024-07-26T12:19:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:11 smithi116 ceph-mon[37097]: pgmap v7381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:19:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13095.939913 secs 2024-07-26T12:19:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:11 smithi171 ceph-mon[37789]: pgmap v7381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:19:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13095.939913 secs 2024-07-26T12:19:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:13.668 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: pgmap v7382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:19:13.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.669 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:13.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:13.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:19:13.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: pgmap v7382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:19:13.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.739 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:13.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:13.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:19:13.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: pgmap v7382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:19:13.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:19:13.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:13.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:13.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:19:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:14 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:19:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:14 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:19:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:14 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:19:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:14 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:19:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:14 smithi044 ceph-mon[32404]: pgmap v7383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:14 smithi044 ceph-mon[32404]: pgmap v7384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:14 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:19:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:14 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:19:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:14 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:19:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:14 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:19:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:14 smithi116 ceph-mon[37097]: pgmap v7383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:14 smithi116 ceph-mon[37097]: pgmap v7384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:14 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:19:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:14 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:19:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:14 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:19:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:14 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:19:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:14 smithi171 ceph-mon[37789]: pgmap v7383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:14 smithi171 ceph-mon[37789]: pgmap v7384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:15.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:15.047+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:19:15.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:15.047+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:19:15.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:15.047+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:16 smithi044 ceph-mon[32404]: pgmap v7385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13100.940076 secs 2024-07-26T12:19:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:16 smithi116 ceph-mon[37097]: pgmap v7385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13100.940076 secs 2024-07-26T12:19:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:16 smithi171 ceph-mon[37789]: pgmap v7385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13100.940076 secs 2024-07-26T12:19:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:18 smithi044 ceph-mon[32404]: pgmap v7386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:18 smithi116 ceph-mon[37097]: pgmap v7386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:18 smithi171 ceph-mon[37789]: pgmap v7386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:20 smithi044 ceph-mon[32404]: pgmap v7387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:19:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:20 smithi116 ceph-mon[37097]: pgmap v7387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:19:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:20 smithi171 ceph-mon[37789]: pgmap v7387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:19:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13105.940225 secs 2024-07-26T12:19:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13105.940225 secs 2024-07-26T12:19:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13105.940225 secs 2024-07-26T12:19:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:22 smithi044 ceph-mon[32404]: pgmap v7388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:22 smithi116 ceph-mon[37097]: pgmap v7388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:22 smithi171 ceph-mon[37789]: pgmap v7388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:24 smithi044 ceph-mon[32404]: pgmap v7389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:24 smithi116 ceph-mon[37097]: pgmap v7389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:24 smithi171 ceph-mon[37789]: pgmap v7389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:26 smithi044 ceph-mon[32404]: pgmap v7390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13110.940389 secs 2024-07-26T12:19:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:26 smithi116 ceph-mon[37097]: pgmap v7390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13110.940389 secs 2024-07-26T12:19:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:26 smithi171 ceph-mon[37789]: pgmap v7390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13110.940389 secs 2024-07-26T12:19:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:28 smithi044 ceph-mon[32404]: pgmap v7391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:28 smithi116 ceph-mon[37097]: pgmap v7391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:28 smithi171 ceph-mon[37789]: pgmap v7391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:30.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:30.048+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:19:30.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:30.048+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:30.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:30.048+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:19:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:30 smithi044 ceph-mon[32404]: pgmap v7392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:30 smithi116 ceph-mon[37097]: pgmap v7392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:30 smithi171 ceph-mon[37789]: pgmap v7392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13115.940534 secs 2024-07-26T12:19:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13115.940534 secs 2024-07-26T12:19:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13115.940534 secs 2024-07-26T12:19:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:32 smithi044 ceph-mon[32404]: pgmap v7393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:32 smithi116 ceph-mon[37097]: pgmap v7393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:32 smithi171 ceph-mon[37789]: pgmap v7393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:34 smithi044 ceph-mon[32404]: pgmap v7394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:34 smithi116 ceph-mon[37097]: pgmap v7394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:34 smithi171 ceph-mon[37789]: pgmap v7394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:36 smithi044 ceph-mon[32404]: pgmap v7395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13120.940703 secs 2024-07-26T12:19:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:36 smithi116 ceph-mon[37097]: pgmap v7395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13120.940703 secs 2024-07-26T12:19:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:36 smithi171 ceph-mon[37789]: pgmap v7395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13120.940703 secs 2024-07-26T12:19:37.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:37 smithi044 ceph-mon[32404]: pgmap v7396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:37 smithi116 ceph-mon[37097]: pgmap v7396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:37 smithi171 ceph-mon[37789]: pgmap v7396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:40 smithi171 ceph-mon[37789]: pgmap v7397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:40 smithi044 ceph-mon[32404]: pgmap v7397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:40 smithi116 ceph-mon[37097]: pgmap v7397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13125.940877 secs 2024-07-26T12:19:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13125.940877 secs 2024-07-26T12:19:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13125.940877 secs 2024-07-26T12:19:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:42 smithi171 ceph-mon[37789]: pgmap v7398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:42 smithi044 ceph-mon[32404]: pgmap v7398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:42.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:42 smithi116 ceph-mon[37097]: pgmap v7398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:44 smithi171 ceph-mon[37789]: pgmap v7399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:44 smithi044 ceph-mon[32404]: pgmap v7399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:44 smithi116 ceph-mon[37097]: pgmap v7399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:45.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:45.054+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:19:45.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:45.054+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:19:45.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:19:45.054+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:46 smithi044 ceph-mon[32404]: pgmap v7400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13130.941043 secs 2024-07-26T12:19:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:46 smithi116 ceph-mon[37097]: pgmap v7400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13130.941043 secs 2024-07-26T12:19:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:46 smithi171 ceph-mon[37789]: pgmap v7400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13130.941043 secs 2024-07-26T12:19:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:48 smithi044 ceph-mon[32404]: pgmap v7401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:48 smithi116 ceph-mon[37097]: pgmap v7401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:48 smithi171 ceph-mon[37789]: pgmap v7401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:50 smithi044 ceph-mon[32404]: pgmap v7402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:50 smithi116 ceph-mon[37097]: pgmap v7402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:50 smithi171 ceph-mon[37789]: pgmap v7402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13135.941211 secs 2024-07-26T12:19:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13135.941211 secs 2024-07-26T12:19:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13135.941211 secs 2024-07-26T12:19:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:52 smithi044 ceph-mon[32404]: pgmap v7403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:52 smithi116 ceph-mon[37097]: pgmap v7403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:52 smithi171 ceph-mon[37789]: pgmap v7403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:54 smithi044 ceph-mon[32404]: pgmap v7404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:54 smithi116 ceph-mon[37097]: pgmap v7404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:54.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:54 smithi171 ceph-mon[37789]: pgmap v7404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:56 smithi044 ceph-mon[32404]: pgmap v7405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13140.941378 secs 2024-07-26T12:19:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:56 smithi116 ceph-mon[37097]: pgmap v7405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13140.941378 secs 2024-07-26T12:19:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:56 smithi171 ceph-mon[37789]: pgmap v7405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13140.941378 secs 2024-07-26T12:19:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:19:58 smithi044 ceph-mon[32404]: pgmap v7406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:19:58 smithi116 ceph-mon[37097]: pgmap v7406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:19:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:19:58 smithi171 ceph-mon[37789]: pgmap v7406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:00.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:00.055+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:20:00.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:00.056+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:20:00.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:00.056+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:00 smithi044 ceph-mon[32404]: pgmap v7407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:00 smithi116 ceph-mon[37097]: pgmap v7407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:00 smithi171 ceph-mon[37789]: pgmap v7407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:20:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13145.941549 secs 2024-07-26T12:20:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13145.941549 secs 2024-07-26T12:20:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13145.941549 secs 2024-07-26T12:20:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:02 smithi044 ceph-mon[32404]: pgmap v7408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:02 smithi116 ceph-mon[37097]: pgmap v7408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:02 smithi171 ceph-mon[37789]: pgmap v7408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:04 smithi044 ceph-mon[32404]: pgmap v7409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:20:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:04 smithi116 ceph-mon[37097]: pgmap v7409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:20:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:04 smithi171 ceph-mon[37789]: pgmap v7409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:20:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:06 smithi044 ceph-mon[32404]: pgmap v7410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:20:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13150.941719 secs 2024-07-26T12:20:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:06 smithi116 ceph-mon[37097]: pgmap v7410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:20:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13150.941719 secs 2024-07-26T12:20:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:06 smithi171 ceph-mon[37789]: pgmap v7410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:20:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13150.941719 secs 2024-07-26T12:20:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:08 smithi044 ceph-mon[32404]: pgmap v7411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:08 smithi116 ceph-mon[37097]: pgmap v7411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:08 smithi171 ceph-mon[37789]: pgmap v7411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:10 smithi044 ceph-mon[32404]: pgmap v7412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:10 smithi116 ceph-mon[37097]: pgmap v7412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:10 smithi171 ceph-mon[37789]: pgmap v7412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:20:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13155.941876 secs 2024-07-26T12:20:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13155.941876 secs 2024-07-26T12:20:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13155.941876 secs 2024-07-26T12:20:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:12 smithi044 ceph-mon[32404]: pgmap v7413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:20:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:12 smithi116 ceph-mon[37097]: pgmap v7413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:20:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:12 smithi171 ceph-mon[37789]: pgmap v7413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:20:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:14 smithi044 ceph-mon[32404]: pgmap v7414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:20:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:14 smithi116 ceph-mon[37097]: pgmap v7414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:20:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:14 smithi171 ceph-mon[37789]: pgmap v7414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:20:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:15.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:15.061+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:20:15.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:15.061+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:20:15.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:15.061+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:15 smithi044 ceph-mon[32404]: pgmap v7415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:20:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:15 smithi044 ceph-mon[32404]: pgmap v7416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:20:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:15 smithi116 ceph-mon[37097]: pgmap v7415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:20:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:15 smithi116 ceph-mon[37097]: pgmap v7416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:20:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:15 smithi171 ceph-mon[37789]: pgmap v7415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:20:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:15 smithi171 ceph-mon[37789]: pgmap v7416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:15.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:20:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13160.942033 secs 2024-07-26T12:20:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13160.942033 secs 2024-07-26T12:20:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13160.942033 secs 2024-07-26T12:20:17.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:17.048+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:20:17.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:17.048+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:20:17.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:17.048+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:20:17.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:17.048+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:20:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:17 smithi044 ceph-mon[32404]: pgmap v7417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:17 smithi116 ceph-mon[37097]: pgmap v7417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:17 smithi171 ceph-mon[37789]: pgmap v7417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:18 smithi171 ceph-mon[37789]: pgmap v7418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:18 smithi044 ceph-mon[32404]: pgmap v7418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:18 smithi116 ceph-mon[37097]: pgmap v7418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:20 smithi171 ceph-mon[37789]: pgmap v7419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:20 smithi044 ceph-mon[32404]: pgmap v7419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:20 smithi116 ceph-mon[37097]: pgmap v7419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:20:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13165.942172 secs 2024-07-26T12:20:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13165.942172 secs 2024-07-26T12:20:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13165.942172 secs 2024-07-26T12:20:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:22 smithi171 ceph-mon[37789]: pgmap v7420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:22 smithi044 ceph-mon[32404]: pgmap v7420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:22 smithi116 ceph-mon[37097]: pgmap v7420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:24 smithi171 ceph-mon[37789]: pgmap v7421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:24 smithi044 ceph-mon[32404]: pgmap v7421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:24 smithi116 ceph-mon[37097]: pgmap v7421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13170.942330 secs 2024-07-26T12:20:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13170.942330 secs 2024-07-26T12:20:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13170.942330 secs 2024-07-26T12:20:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:26 smithi044 ceph-mon[32404]: pgmap v7422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:26 smithi116 ceph-mon[37097]: pgmap v7422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:26 smithi171 ceph-mon[37789]: pgmap v7422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:28 smithi171 ceph-mon[37789]: pgmap v7423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:28 smithi044 ceph-mon[32404]: pgmap v7423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:28 smithi116 ceph-mon[37097]: pgmap v7423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:30.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:30.063+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:20:30.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:30.063+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:30.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:30.063+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:20:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:30 smithi171 ceph-mon[37789]: pgmap v7424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:30 smithi044 ceph-mon[32404]: pgmap v7424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:30 smithi116 ceph-mon[37097]: pgmap v7424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13175.942447 secs 2024-07-26T12:20:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13175.942447 secs 2024-07-26T12:20:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13175.942447 secs 2024-07-26T12:20:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:32 smithi171 ceph-mon[37789]: pgmap v7425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:32 smithi044 ceph-mon[32404]: pgmap v7425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:32 smithi116 ceph-mon[37097]: pgmap v7425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:34 smithi171 ceph-mon[37789]: pgmap v7426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:34 smithi044 ceph-mon[32404]: pgmap v7426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:34 smithi116 ceph-mon[37097]: pgmap v7426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13180.942567 secs 2024-07-26T12:20:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13180.942567 secs 2024-07-26T12:20:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13180.942567 secs 2024-07-26T12:20:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:37 smithi044 ceph-mon[32404]: pgmap v7427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:37 smithi116 ceph-mon[37097]: pgmap v7427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:37 smithi171 ceph-mon[37789]: pgmap v7427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:39 smithi044 ceph-mon[32404]: pgmap v7428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:39 smithi116 ceph-mon[37097]: pgmap v7428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:39 smithi171 ceph-mon[37789]: pgmap v7428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13185.942715 secs 2024-07-26T12:20:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13185.942715 secs 2024-07-26T12:20:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13185.942715 secs 2024-07-26T12:20:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:41 smithi044 ceph-mon[32404]: pgmap v7429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:41 smithi116 ceph-mon[37097]: pgmap v7429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:41 smithi171 ceph-mon[37789]: pgmap v7429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:43 smithi044 ceph-mon[32404]: pgmap v7430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:43 smithi116 ceph-mon[37097]: pgmap v7430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:43 smithi171 ceph-mon[37789]: pgmap v7430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:45.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:45.068+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:20:45.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:45.068+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:20:45.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:20:45.068+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:45 smithi044 ceph-mon[32404]: pgmap v7431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13190.942809 secs 2024-07-26T12:20:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:45 smithi116 ceph-mon[37097]: pgmap v7431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13190.942809 secs 2024-07-26T12:20:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:45 smithi171 ceph-mon[37789]: pgmap v7431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13190.942809 secs 2024-07-26T12:20:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:47 smithi044 ceph-mon[32404]: pgmap v7432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:47 smithi116 ceph-mon[37097]: pgmap v7432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:47 smithi171 ceph-mon[37789]: pgmap v7432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:50 smithi044 ceph-mon[32404]: pgmap v7433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:50 smithi116 ceph-mon[37097]: pgmap v7433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:50 smithi171 ceph-mon[37789]: pgmap v7433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:20:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13195.942971 secs 2024-07-26T12:20:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13195.942971 secs 2024-07-26T12:20:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13195.942971 secs 2024-07-26T12:20:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:52 smithi116 ceph-mon[37097]: pgmap v7434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:52 smithi171 ceph-mon[37789]: pgmap v7434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:52 smithi044 ceph-mon[32404]: pgmap v7434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:54 smithi171 ceph-mon[37789]: pgmap v7435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:54 smithi044 ceph-mon[32404]: pgmap v7435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:54 smithi116 ceph-mon[37097]: pgmap v7435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:20:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:56 smithi171 ceph-mon[37789]: pgmap v7436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13200.943114 secs 2024-07-26T12:20:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:56 smithi044 ceph-mon[32404]: pgmap v7436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13200.943114 secs 2024-07-26T12:20:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:56 smithi116 ceph-mon[37097]: pgmap v7436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:20:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13200.943114 secs 2024-07-26T12:20:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:20:58 smithi171 ceph-mon[37789]: pgmap v7437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:20:58 smithi044 ceph-mon[32404]: pgmap v7437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:20:58 smithi116 ceph-mon[37097]: pgmap v7437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:00.069+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:21:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:00.069+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:00.069+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:21:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:00 smithi171 ceph-mon[37789]: pgmap v7438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:00 smithi044 ceph-mon[32404]: pgmap v7438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:00 smithi116 ceph-mon[37097]: pgmap v7438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13205.943269 secs 2024-07-26T12:21:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13205.943269 secs 2024-07-26T12:21:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13205.943269 secs 2024-07-26T12:21:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:02 smithi171 ceph-mon[37789]: pgmap v7439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:02 smithi044 ceph-mon[32404]: pgmap v7439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:02 smithi116 ceph-mon[37097]: pgmap v7439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:04.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:04 smithi171 ceph-mon[37789]: pgmap v7440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:04 smithi044 ceph-mon[32404]: pgmap v7440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:04 smithi116 ceph-mon[37097]: pgmap v7440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:06.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:06 smithi116 ceph-mon[37097]: pgmap v7441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:06.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13210.943438 secs 2024-07-26T12:21:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:06 smithi171 ceph-mon[37789]: pgmap v7441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13210.943438 secs 2024-07-26T12:21:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:06 smithi044 ceph-mon[32404]: pgmap v7441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13210.943438 secs 2024-07-26T12:21:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:08 smithi171 ceph-mon[37789]: pgmap v7442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:08 smithi044 ceph-mon[32404]: pgmap v7442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:08 smithi116 ceph-mon[37097]: pgmap v7442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:10 smithi171 ceph-mon[37789]: pgmap v7443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:10 smithi044 ceph-mon[32404]: pgmap v7443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:10 smithi116 ceph-mon[37097]: pgmap v7443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13215.943589 secs 2024-07-26T12:21:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13215.943589 secs 2024-07-26T12:21:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13215.943589 secs 2024-07-26T12:21:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:12 smithi044 ceph-mon[32404]: pgmap v7444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:12 smithi116 ceph-mon[37097]: pgmap v7444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:12 smithi171 ceph-mon[37789]: pgmap v7444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:14.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:14 smithi171 ceph-mon[37789]: pgmap v7445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:14 smithi044 ceph-mon[32404]: pgmap v7445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:14 smithi116 ceph-mon[37097]: pgmap v7445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:15.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:15.074+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:21:15.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:15.074+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:15.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:15.074+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:21:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:16 smithi044 ceph-mon[32404]: pgmap v7446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13220.943673 secs 2024-07-26T12:21:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:16 smithi116 ceph-mon[37097]: pgmap v7446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13220.943673 secs 2024-07-26T12:21:16.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:16 smithi171 ceph-mon[37789]: pgmap v7446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:16.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:16.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:16.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13220.943673 secs 2024-07-26T12:21:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:17 smithi044 ceph-mon[32404]: pgmap v7447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:17 smithi044 ceph-mon[32404]: pgmap v7448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:17 smithi116 ceph-mon[37097]: pgmap v7447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:17 smithi116 ceph-mon[37097]: pgmap v7448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:17 smithi171 ceph-mon[37789]: pgmap v7447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:17 smithi171 ceph-mon[37789]: pgmap v7448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:21:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:18 smithi171 ceph-mon[37789]: pgmap v7449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:18 smithi044 ceph-mon[32404]: pgmap v7449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:18 smithi116 ceph-mon[37097]: pgmap v7449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:20 smithi171 ceph-mon[37789]: pgmap v7450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:20 smithi044 ceph-mon[32404]: pgmap v7450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:20 smithi116 ceph-mon[37097]: pgmap v7450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13225.943846 secs 2024-07-26T12:21:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13225.943846 secs 2024-07-26T12:21:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13225.943846 secs 2024-07-26T12:21:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:22 smithi171 ceph-mon[37789]: pgmap v7451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:22 smithi044 ceph-mon[32404]: pgmap v7451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:22 smithi116 ceph-mon[37097]: pgmap v7451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:24 smithi171 ceph-mon[37789]: pgmap v7452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:24 smithi044 ceph-mon[32404]: pgmap v7452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:24 smithi116 ceph-mon[37097]: pgmap v7452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:21:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13230.943954 secs 2024-07-26T12:21:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13230.943954 secs 2024-07-26T12:21:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13230.943954 secs 2024-07-26T12:21:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:26 smithi044 ceph-mon[32404]: pgmap v7453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:26 smithi116 ceph-mon[37097]: pgmap v7453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:26 smithi171 ceph-mon[37789]: pgmap v7453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:28 smithi171 ceph-mon[37789]: pgmap v7454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:28 smithi044 ceph-mon[32404]: pgmap v7454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:28 smithi116 ceph-mon[37097]: pgmap v7454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:30.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:30.074+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:21:30.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:30.074+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:30.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:30.074+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:21:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:30 smithi171 ceph-mon[37789]: pgmap v7455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:30 smithi044 ceph-mon[32404]: pgmap v7455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:30 smithi116 ceph-mon[37097]: pgmap v7455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13235.944042 secs 2024-07-26T12:21:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13235.944042 secs 2024-07-26T12:21:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13235.944042 secs 2024-07-26T12:21:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:32 smithi171 ceph-mon[37789]: pgmap v7456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:32 smithi044 ceph-mon[32404]: pgmap v7456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:32 smithi116 ceph-mon[37097]: pgmap v7456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:34 smithi171 ceph-mon[37789]: pgmap v7457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:34 smithi044 ceph-mon[32404]: pgmap v7457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:34 smithi116 ceph-mon[37097]: pgmap v7457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13240.944203 secs 2024-07-26T12:21:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13240.944203 secs 2024-07-26T12:21:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13240.944203 secs 2024-07-26T12:21:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:36 smithi044 ceph-mon[32404]: pgmap v7458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:36 smithi116 ceph-mon[37097]: pgmap v7458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:36 smithi171 ceph-mon[37789]: pgmap v7458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:38.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:38 smithi171 ceph-mon[37789]: pgmap v7459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:38 smithi044 ceph-mon[32404]: pgmap v7459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:38 smithi116 ceph-mon[37097]: pgmap v7459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:40 smithi171 ceph-mon[37789]: pgmap v7460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:40 smithi044 ceph-mon[32404]: pgmap v7460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:40 smithi116 ceph-mon[37097]: pgmap v7460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:41.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13245.944369 secs 2024-07-26T12:21:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13245.944369 secs 2024-07-26T12:21:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13245.944369 secs 2024-07-26T12:21:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:42 smithi171 ceph-mon[37789]: pgmap v7461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:42.945 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:42 smithi044 ceph-mon[32404]: pgmap v7461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:42 smithi116 ceph-mon[37097]: pgmap v7461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:44 smithi171 ceph-mon[37789]: pgmap v7462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:44 smithi044 ceph-mon[32404]: pgmap v7462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:44 smithi116 ceph-mon[37097]: pgmap v7462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:45.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:45.079+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:21:45.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:45.079+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:45.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:21:45.079+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:21:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13250.944536 secs 2024-07-26T12:21:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13250.944536 secs 2024-07-26T12:21:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13250.944536 secs 2024-07-26T12:21:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:46 smithi044 ceph-mon[32404]: pgmap v7463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:46 smithi116 ceph-mon[37097]: pgmap v7463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:46 smithi171 ceph-mon[37789]: pgmap v7463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:48 smithi171 ceph-mon[37789]: pgmap v7464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:48 smithi044 ceph-mon[32404]: pgmap v7464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:48 smithi116 ceph-mon[37097]: pgmap v7464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:50 smithi171 ceph-mon[37789]: pgmap v7465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:50 smithi044 ceph-mon[32404]: pgmap v7465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:50 smithi116 ceph-mon[37097]: pgmap v7465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13255.944706 secs 2024-07-26T12:21:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13255.944706 secs 2024-07-26T12:21:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13255.944706 secs 2024-07-26T12:21:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:52 smithi171 ceph-mon[37789]: pgmap v7466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:52 smithi044 ceph-mon[32404]: pgmap v7466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:52 smithi116 ceph-mon[37097]: pgmap v7466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:21:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:54 smithi171 ceph-mon[37789]: pgmap v7467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:54 smithi044 ceph-mon[32404]: pgmap v7467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:54 smithi116 ceph-mon[37097]: pgmap v7467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:21:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13260.944864 secs 2024-07-26T12:21:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13260.944864 secs 2024-07-26T12:21:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13260.944864 secs 2024-07-26T12:21:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:56 smithi044 ceph-mon[32404]: pgmap v7468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:56 smithi116 ceph-mon[37097]: pgmap v7468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:56 smithi171 ceph-mon[37789]: pgmap v7468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:58.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:21:58 smithi044 ceph-mon[32404]: pgmap v7469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:21:58 smithi171 ceph-mon[37789]: pgmap v7469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:21:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:21:58 smithi116 ceph-mon[37097]: pgmap v7469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:00.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:00.081+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:22:00.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:00.081+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:00.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:00.081+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:22:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:00 smithi171 ceph-mon[37789]: pgmap v7470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:00 smithi044 ceph-mon[32404]: pgmap v7470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:00 smithi116 ceph-mon[37097]: pgmap v7470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13265.944965 secs 2024-07-26T12:22:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13265.944965 secs 2024-07-26T12:22:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13265.944965 secs 2024-07-26T12:22:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:02 smithi171 ceph-mon[37789]: pgmap v7471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:02 smithi044 ceph-mon[32404]: pgmap v7471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:02 smithi116 ceph-mon[37097]: pgmap v7471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:04 smithi171 ceph-mon[37789]: pgmap v7472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:04 smithi044 ceph-mon[32404]: pgmap v7472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:04 smithi116 ceph-mon[37097]: pgmap v7472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13270.945108 secs 2024-07-26T12:22:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13270.945108 secs 2024-07-26T12:22:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13270.945108 secs 2024-07-26T12:22:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:06 smithi044 ceph-mon[32404]: pgmap v7473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:06 smithi116 ceph-mon[37097]: pgmap v7473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:06 smithi171 ceph-mon[37789]: pgmap v7473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:08 smithi171 ceph-mon[37789]: pgmap v7474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:08 smithi044 ceph-mon[32404]: pgmap v7474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:08 smithi116 ceph-mon[37097]: pgmap v7474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:11 smithi171 ceph-mon[37789]: pgmap v7475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13275.945256 secs 2024-07-26T12:22:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:11 smithi044 ceph-mon[32404]: pgmap v7475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13275.945256 secs 2024-07-26T12:22:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:11 smithi116 ceph-mon[37097]: pgmap v7475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13275.945256 secs 2024-07-26T12:22:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:13 smithi171 ceph-mon[37789]: pgmap v7476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:13 smithi044 ceph-mon[32404]: pgmap v7476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:13 smithi116 ceph-mon[37097]: pgmap v7476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:15.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:15.087+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:22:15.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:15.087+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:15.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:15.087+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:22:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:15 smithi171 ceph-mon[37789]: pgmap v7477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:15 smithi044 ceph-mon[32404]: pgmap v7477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:15 smithi116 ceph-mon[37097]: pgmap v7477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:17.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13280.945395 secs 2024-07-26T12:22:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13280.945395 secs 2024-07-26T12:22:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13280.945395 secs 2024-07-26T12:22:17.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:17 smithi044 ceph-mon[32404]: pgmap v7478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:17.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:17.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.940 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:17 smithi171 ceph-mon[37789]: pgmap v7478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:17.940 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:17.941 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.941 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.941 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.941 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:17 smithi116 ceph-mon[37097]: pgmap v7478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:17.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:17.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:17.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:19 smithi044 ceph-mon[32404]: pgmap v7479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:19 smithi116 ceph-mon[37097]: pgmap v7479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:19 smithi171 ceph-mon[37789]: pgmap v7479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13285.945499 secs 2024-07-26T12:22:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13285.945499 secs 2024-07-26T12:22:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13285.945499 secs 2024-07-26T12:22:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:22 smithi044 ceph-mon[32404]: pgmap v7480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:22 smithi116 ceph-mon[37097]: pgmap v7480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:22 smithi171 ceph-mon[37789]: pgmap v7480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:22:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:23 smithi044 ceph-mon[32404]: pgmap v7481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:22:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:23 smithi044 ceph-mon[32404]: pgmap v7482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:23 smithi116 ceph-mon[37097]: pgmap v7481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:22:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:23 smithi116 ceph-mon[37097]: pgmap v7482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:23 smithi171 ceph-mon[37789]: pgmap v7481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:22:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:23 smithi171 ceph-mon[37789]: pgmap v7482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:25 smithi044 ceph-mon[32404]: pgmap v7483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:25 smithi116 ceph-mon[37097]: pgmap v7483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:25 smithi171 ceph-mon[37789]: pgmap v7483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13290.945664 secs 2024-07-26T12:22:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13290.945664 secs 2024-07-26T12:22:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13290.945664 secs 2024-07-26T12:22:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:27 smithi044 ceph-mon[32404]: pgmap v7484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:27 smithi116 ceph-mon[37097]: pgmap v7484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:27 smithi171 ceph-mon[37789]: pgmap v7484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:29 smithi044 ceph-mon[32404]: pgmap v7485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:29 smithi116 ceph-mon[37097]: pgmap v7485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:29 smithi171 ceph-mon[37789]: pgmap v7485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:30.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:30.087+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:30.087+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:30.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:30.087+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:22:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:31 smithi044 ceph-mon[32404]: pgmap v7486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13295.945832 secs 2024-07-26T12:22:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:31 smithi116 ceph-mon[37097]: pgmap v7486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13295.945832 secs 2024-07-26T12:22:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:31 smithi171 ceph-mon[37789]: pgmap v7486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13295.945832 secs 2024-07-26T12:22:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:33 smithi044 ceph-mon[32404]: pgmap v7487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:33 smithi116 ceph-mon[37097]: pgmap v7487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:33 smithi171 ceph-mon[37789]: pgmap v7487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:35 smithi044 ceph-mon[32404]: pgmap v7488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:35 smithi116 ceph-mon[37097]: pgmap v7488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:35 smithi171 ceph-mon[37789]: pgmap v7488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13300.945968 secs 2024-07-26T12:22:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13300.945968 secs 2024-07-26T12:22:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13300.945968 secs 2024-07-26T12:22:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:37 smithi044 ceph-mon[32404]: pgmap v7489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:37 smithi116 ceph-mon[37097]: pgmap v7489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:37 smithi171 ceph-mon[37789]: pgmap v7489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:39 smithi044 ceph-mon[32404]: pgmap v7490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:39 smithi116 ceph-mon[37097]: pgmap v7490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:39 smithi171 ceph-mon[37789]: pgmap v7490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:41 smithi044 ceph-mon[32404]: pgmap v7491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13305.946115 secs 2024-07-26T12:22:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:41 smithi116 ceph-mon[37097]: pgmap v7491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13305.946115 secs 2024-07-26T12:22:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:41 smithi171 ceph-mon[37789]: pgmap v7491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13305.946115 secs 2024-07-26T12:22:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:43 smithi044 ceph-mon[32404]: pgmap v7492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:43 smithi116 ceph-mon[37097]: pgmap v7492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:43 smithi171 ceph-mon[37789]: pgmap v7492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:45.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:45.092+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:22:45.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:45.092+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:22:45.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:22:45.093+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:45 smithi044 ceph-mon[32404]: pgmap v7493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:45 smithi116 ceph-mon[37097]: pgmap v7493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:45 smithi171 ceph-mon[37789]: pgmap v7493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13310.946279 secs 2024-07-26T12:22:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13310.946279 secs 2024-07-26T12:22:47.043 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13310.946279 secs 2024-07-26T12:22:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:47 smithi171 ceph-mon[37789]: pgmap v7494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:47 smithi044 ceph-mon[32404]: pgmap v7494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:47 smithi116 ceph-mon[37097]: pgmap v7494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:48 smithi171 ceph-mon[37789]: pgmap v7495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:48 smithi044 ceph-mon[32404]: pgmap v7495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:48 smithi116 ceph-mon[37097]: pgmap v7495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:50 smithi171 ceph-mon[37789]: pgmap v7496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:50 smithi044 ceph-mon[32404]: pgmap v7496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:50 smithi116 ceph-mon[37097]: pgmap v7496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13315.946425 secs 2024-07-26T12:22:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13315.946425 secs 2024-07-26T12:22:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13315.946425 secs 2024-07-26T12:22:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:53 smithi044 ceph-mon[32404]: pgmap v7497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:53 smithi116 ceph-mon[37097]: pgmap v7497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:53 smithi171 ceph-mon[37789]: pgmap v7497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:55 smithi044 ceph-mon[32404]: pgmap v7498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:55 smithi116 ceph-mon[37097]: pgmap v7498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:55 smithi171 ceph-mon[37789]: pgmap v7498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:22:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13320.946582 secs 2024-07-26T12:22:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13320.946582 secs 2024-07-26T12:22:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13320.946582 secs 2024-07-26T12:22:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:57 smithi044 ceph-mon[32404]: pgmap v7499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:57 smithi116 ceph-mon[37097]: pgmap v7499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:57 smithi171 ceph-mon[37789]: pgmap v7499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:22:59 smithi044 ceph-mon[32404]: pgmap v7500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:22:59 smithi116 ceph-mon[37097]: pgmap v7500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:22:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:22:59 smithi171 ceph-mon[37789]: pgmap v7500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:00.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:00.094+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:23:00.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:00.094+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:23:00.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:00.094+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:01 smithi044 ceph-mon[32404]: pgmap v7501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13325.946744 secs 2024-07-26T12:23:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:01 smithi116 ceph-mon[37097]: pgmap v7501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13325.946744 secs 2024-07-26T12:23:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:01 smithi171 ceph-mon[37789]: pgmap v7501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13325.946744 secs 2024-07-26T12:23:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:03 smithi044 ceph-mon[32404]: pgmap v7502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:03 smithi116 ceph-mon[37097]: pgmap v7502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:03 smithi171 ceph-mon[37789]: pgmap v7502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:05 smithi044 ceph-mon[32404]: pgmap v7503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:23:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:05 smithi116 ceph-mon[37097]: pgmap v7503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:23:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:05 smithi171 ceph-mon[37789]: pgmap v7503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:23:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13330.946906 secs 2024-07-26T12:23:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13330.946906 secs 2024-07-26T12:23:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13330.946906 secs 2024-07-26T12:23:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:07 smithi044 ceph-mon[32404]: pgmap v7504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:07 smithi116 ceph-mon[37097]: pgmap v7504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:07 smithi171 ceph-mon[37789]: pgmap v7504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:09.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:09 smithi044 ceph-mon[32404]: pgmap v7505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:09 smithi116 ceph-mon[37097]: pgmap v7505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:09 smithi171 ceph-mon[37789]: pgmap v7505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:11 smithi044 ceph-mon[32404]: pgmap v7506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13335.947022 secs 2024-07-26T12:23:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:11 smithi116 ceph-mon[37097]: pgmap v7506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13335.947022 secs 2024-07-26T12:23:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:11 smithi171 ceph-mon[37789]: pgmap v7506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13335.947022 secs 2024-07-26T12:23:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:13 smithi044 ceph-mon[32404]: pgmap v7507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:13 smithi116 ceph-mon[37097]: pgmap v7507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:13 smithi171 ceph-mon[37789]: pgmap v7507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:15.099+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:23:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:15.099+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:15.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:15.099+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:23:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:15 smithi044 ceph-mon[32404]: pgmap v7508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:23:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:15 smithi116 ceph-mon[37097]: pgmap v7508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:23:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:15 smithi171 ceph-mon[37789]: pgmap v7508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:23:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13340.947168 secs 2024-07-26T12:23:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13340.947168 secs 2024-07-26T12:23:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13340.947168 secs 2024-07-26T12:23:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:17.080+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:23:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:17.080+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:23:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:17.081+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:23:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:17.081+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:23:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:17 smithi171 ceph-mon[37789]: pgmap v7509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:17 smithi044 ceph-mon[32404]: pgmap v7509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:17 smithi116 ceph-mon[37097]: pgmap v7509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:18 smithi171 ceph-mon[37789]: pgmap v7510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:18 smithi044 ceph-mon[32404]: pgmap v7510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:18 smithi116 ceph-mon[37097]: pgmap v7510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:20 smithi044 ceph-mon[32404]: pgmap v7511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:20 smithi116 ceph-mon[37097]: pgmap v7511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:20 smithi171 ceph-mon[37789]: pgmap v7511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13345.947331 secs 2024-07-26T12:23:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13345.947331 secs 2024-07-26T12:23:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13345.947331 secs 2024-07-26T12:23:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:22 smithi044 ceph-mon[32404]: pgmap v7512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:22 smithi116 ceph-mon[37097]: pgmap v7512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:22 smithi171 ceph-mon[37789]: pgmap v7512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:23 smithi044 ceph-mon[32404]: pgmap v7513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:23 smithi044 ceph-mon[32404]: pgmap v7514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:23:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:23 smithi116 ceph-mon[37097]: pgmap v7513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:23 smithi116 ceph-mon[37097]: pgmap v7514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:24.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:23:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:23 smithi171 ceph-mon[37789]: pgmap v7513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:23 smithi171 ceph-mon[37789]: pgmap v7514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:23:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:25 smithi044 ceph-mon[32404]: pgmap v7515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:25 smithi116 ceph-mon[37097]: pgmap v7515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:25 smithi171 ceph-mon[37789]: pgmap v7515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13350.947481 secs 2024-07-26T12:23:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13350.947481 secs 2024-07-26T12:23:27.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13350.947481 secs 2024-07-26T12:23:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:27 smithi044 ceph-mon[32404]: pgmap v7516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:27 smithi116 ceph-mon[37097]: pgmap v7516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:27 smithi171 ceph-mon[37789]: pgmap v7516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:30.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:30.099+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:23:30.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:30.099+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:23:30.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:30.099+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:29 smithi044 ceph-mon[32404]: pgmap v7517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:29 smithi116 ceph-mon[37097]: pgmap v7517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:29 smithi171 ceph-mon[37789]: pgmap v7517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13355.947654 secs 2024-07-26T12:23:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:31 smithi044 ceph-mon[32404]: pgmap v7518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13355.947654 secs 2024-07-26T12:23:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:31 smithi116 ceph-mon[37097]: pgmap v7518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13355.947654 secs 2024-07-26T12:23:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:31 smithi171 ceph-mon[37789]: pgmap v7518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:23:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:33 smithi044 ceph-mon[32404]: pgmap v7519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:33 smithi116 ceph-mon[37097]: pgmap v7519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:33 smithi171 ceph-mon[37789]: pgmap v7519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:35 smithi044 ceph-mon[32404]: pgmap v7520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13360.947797 secs 2024-07-26T12:23:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:35 smithi116 ceph-mon[37097]: pgmap v7520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13360.947797 secs 2024-07-26T12:23:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:35 smithi171 ceph-mon[37789]: pgmap v7520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13360.947797 secs 2024-07-26T12:23:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:37 smithi044 ceph-mon[32404]: pgmap v7521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:37 smithi116 ceph-mon[37097]: pgmap v7521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:37 smithi171 ceph-mon[37789]: pgmap v7521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:39 smithi044 ceph-mon[32404]: pgmap v7522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:39 smithi116 ceph-mon[37097]: pgmap v7522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:39 smithi171 ceph-mon[37789]: pgmap v7522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13365.947949 secs 2024-07-26T12:23:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13365.947949 secs 2024-07-26T12:23:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13365.947949 secs 2024-07-26T12:23:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:41 smithi044 ceph-mon[32404]: pgmap v7523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:41 smithi116 ceph-mon[37097]: pgmap v7523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:41 smithi171 ceph-mon[37789]: pgmap v7523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:43 smithi044 ceph-mon[32404]: pgmap v7524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:43 smithi116 ceph-mon[37097]: pgmap v7524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:44.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:43 smithi171 ceph-mon[37789]: pgmap v7524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:45.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:45.102+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:23:45.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:45.102+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:23:45.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:23:45.102+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:45 smithi044 ceph-mon[32404]: pgmap v7525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13370.948110 secs 2024-07-26T12:23:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:45 smithi116 ceph-mon[37097]: pgmap v7525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:46.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13370.948110 secs 2024-07-26T12:23:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:45 smithi171 ceph-mon[37789]: pgmap v7525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13370.948110 secs 2024-07-26T12:23:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:47 smithi044 ceph-mon[32404]: pgmap v7526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:47 smithi116 ceph-mon[37097]: pgmap v7526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:47 smithi171 ceph-mon[37789]: pgmap v7526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:49 smithi044 ceph-mon[32404]: pgmap v7527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:50.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:49 smithi116 ceph-mon[37097]: pgmap v7527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:49 smithi171 ceph-mon[37789]: pgmap v7527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13375.948282 secs 2024-07-26T12:23:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13375.948282 secs 2024-07-26T12:23:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13375.948282 secs 2024-07-26T12:23:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:52 smithi044 ceph-mon[32404]: pgmap v7528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:52 smithi116 ceph-mon[37097]: pgmap v7528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:52 smithi171 ceph-mon[37789]: pgmap v7528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:54 smithi044 ceph-mon[32404]: pgmap v7529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:54 smithi116 ceph-mon[37097]: pgmap v7529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:54 smithi171 ceph-mon[37789]: pgmap v7529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:56 smithi044 ceph-mon[32404]: pgmap v7530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13380.948445 secs 2024-07-26T12:23:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:56 smithi116 ceph-mon[37097]: pgmap v7530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13380.948445 secs 2024-07-26T12:23:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:56 smithi171 ceph-mon[37789]: pgmap v7530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:23:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13380.948445 secs 2024-07-26T12:23:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:23:58 smithi116 ceph-mon[37097]: pgmap v7531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:23:58 smithi171 ceph-mon[37789]: pgmap v7531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:23:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:23:58 smithi044 ceph-mon[32404]: pgmap v7531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:00.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:00.103+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:24:00.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:00.103+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:00.106 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:00.103+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:24:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:00 smithi171 ceph-mon[37789]: pgmap v7532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:00.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:00 smithi044 ceph-mon[32404]: pgmap v7532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:00 smithi116 ceph-mon[37097]: pgmap v7532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:01.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13385.948611 secs 2024-07-26T12:24:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13385.948611 secs 2024-07-26T12:24:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13385.948611 secs 2024-07-26T12:24:02.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:02.089+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:24:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:02.089+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:24:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:02.089+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:24:02.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:02.090+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:24:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:02 smithi171 ceph-mon[37789]: pgmap v7533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:02 smithi044 ceph-mon[32404]: pgmap v7533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:02 smithi116 ceph-mon[37097]: pgmap v7533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:04 smithi171 ceph-mon[37789]: pgmap v7534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:04 smithi044 ceph-mon[32404]: pgmap v7534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:04 smithi116 ceph-mon[37097]: pgmap v7534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:06.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:06 smithi116 ceph-mon[37097]: pgmap v7535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:06.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13390.948748 secs 2024-07-26T12:24:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:06 smithi171 ceph-mon[37789]: pgmap v7535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13390.948748 secs 2024-07-26T12:24:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:06 smithi044 ceph-mon[32404]: pgmap v7535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13390.948748 secs 2024-07-26T12:24:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:08 smithi171 ceph-mon[37789]: pgmap v7536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:08 smithi044 ceph-mon[32404]: pgmap v7536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:08 smithi116 ceph-mon[37097]: pgmap v7536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:10 smithi171 ceph-mon[37789]: pgmap v7537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:10 smithi044 ceph-mon[32404]: pgmap v7537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:10 smithi116 ceph-mon[37097]: pgmap v7537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13395.948892 secs 2024-07-26T12:24:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13395.948892 secs 2024-07-26T12:24:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13395.948892 secs 2024-07-26T12:24:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:12 smithi171 ceph-mon[37789]: pgmap v7538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:12 smithi044 ceph-mon[32404]: pgmap v7538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:12 smithi116 ceph-mon[37097]: pgmap v7538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:14 smithi171 ceph-mon[37789]: pgmap v7539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:14 smithi044 ceph-mon[32404]: pgmap v7539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:14 smithi116 ceph-mon[37097]: pgmap v7539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:15.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:15.108+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:24:15.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:15.108+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:24:15.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:15.108+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:16 smithi044 ceph-mon[32404]: pgmap v7540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13400.949057 secs 2024-07-26T12:24:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:16 smithi116 ceph-mon[37097]: pgmap v7540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13400.949057 secs 2024-07-26T12:24:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:16 smithi171 ceph-mon[37789]: pgmap v7540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13400.949057 secs 2024-07-26T12:24:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:18 smithi044 ceph-mon[32404]: pgmap v7541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:18 smithi116 ceph-mon[37097]: pgmap v7541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:18 smithi171 ceph-mon[37789]: pgmap v7541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:20 smithi044 ceph-mon[32404]: pgmap v7542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:20 smithi116 ceph-mon[37097]: pgmap v7542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:20 smithi171 ceph-mon[37789]: pgmap v7542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13405.949208 secs 2024-07-26T12:24:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13405.949208 secs 2024-07-26T12:24:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13405.949208 secs 2024-07-26T12:24:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:22 smithi044 ceph-mon[32404]: pgmap v7543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:22 smithi116 ceph-mon[37097]: pgmap v7543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:22 smithi171 ceph-mon[37789]: pgmap v7543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:24 smithi044 ceph-mon[32404]: pgmap v7544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:24 smithi116 ceph-mon[37097]: pgmap v7544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:24 smithi171 ceph-mon[37789]: pgmap v7544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:25 smithi044 ceph-mon[32404]: pgmap v7545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:24:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:25 smithi044 ceph-mon[32404]: pgmap v7546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:24:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:25 smithi116 ceph-mon[37097]: pgmap v7545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:24:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:25 smithi116 ceph-mon[37097]: pgmap v7546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:24:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:25 smithi171 ceph-mon[37789]: pgmap v7545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:24:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:25 smithi171 ceph-mon[37789]: pgmap v7546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:24:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13410.949358 secs 2024-07-26T12:24:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13410.949358 secs 2024-07-26T12:24:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13410.949358 secs 2024-07-26T12:24:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:27 smithi044 ceph-mon[32404]: pgmap v7547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:24:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:27 smithi116 ceph-mon[37097]: pgmap v7547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:24:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:27 smithi171 ceph-mon[37789]: pgmap v7547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:24:29.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:29 smithi044 ceph-mon[32404]: pgmap v7548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:29 smithi116 ceph-mon[37097]: pgmap v7548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:29 smithi171 ceph-mon[37789]: pgmap v7548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:30.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:30.110+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:24:30.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:30.110+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:30.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:30.110+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:24:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:31 smithi044 ceph-mon[32404]: pgmap v7549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13415.949522 secs 2024-07-26T12:24:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:31 smithi116 ceph-mon[37097]: pgmap v7549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13415.949522 secs 2024-07-26T12:24:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:31 smithi171 ceph-mon[37789]: pgmap v7549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13415.949522 secs 2024-07-26T12:24:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:33 smithi044 ceph-mon[32404]: pgmap v7550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:33 smithi116 ceph-mon[37097]: pgmap v7550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:33 smithi171 ceph-mon[37789]: pgmap v7550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:24:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:35 smithi044 ceph-mon[32404]: pgmap v7551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:24:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:35 smithi116 ceph-mon[37097]: pgmap v7551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:24:35.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:35 smithi171 ceph-mon[37789]: pgmap v7551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:24:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13420.949673 secs 2024-07-26T12:24:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13420.949673 secs 2024-07-26T12:24:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13420.949673 secs 2024-07-26T12:24:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:37 smithi044 ceph-mon[32404]: pgmap v7552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:37 smithi116 ceph-mon[37097]: pgmap v7552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:37 smithi171 ceph-mon[37789]: pgmap v7552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:39 smithi116 ceph-mon[37097]: pgmap v7553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:39 smithi171 ceph-mon[37789]: pgmap v7553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:39 smithi044 ceph-mon[32404]: pgmap v7553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:41 smithi171 ceph-mon[37789]: pgmap v7554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13425.949916 secs 2024-07-26T12:24:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:41 smithi044 ceph-mon[32404]: pgmap v7554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13425.949916 secs 2024-07-26T12:24:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:41 smithi116 ceph-mon[37097]: pgmap v7554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13425.949916 secs 2024-07-26T12:24:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:43 smithi171 ceph-mon[37789]: pgmap v7555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:43 smithi044 ceph-mon[32404]: pgmap v7555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:43 smithi116 ceph-mon[37097]: pgmap v7555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:45.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:45.113+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:24:45.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:45.113+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:45.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:24:45.113+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:24:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:45 smithi171 ceph-mon[37789]: pgmap v7556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:45 smithi044 ceph-mon[32404]: pgmap v7556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:45 smithi116 ceph-mon[37097]: pgmap v7556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13430.950022 secs 2024-07-26T12:24:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13430.950022 secs 2024-07-26T12:24:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13430.950022 secs 2024-07-26T12:24:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:47 smithi171 ceph-mon[37789]: pgmap v7557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:47 smithi044 ceph-mon[32404]: pgmap v7557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:47 smithi116 ceph-mon[37097]: pgmap v7557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:48 smithi171 ceph-mon[37789]: pgmap v7558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:48 smithi044 ceph-mon[32404]: pgmap v7558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:48 smithi116 ceph-mon[37097]: pgmap v7558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:51 smithi044 ceph-mon[32404]: pgmap v7559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13435.950161 secs 2024-07-26T12:24:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:51 smithi116 ceph-mon[37097]: pgmap v7559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13435.950161 secs 2024-07-26T12:24:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:51 smithi171 ceph-mon[37789]: pgmap v7559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13435.950161 secs 2024-07-26T12:24:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:53 smithi044 ceph-mon[32404]: pgmap v7560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:53 smithi116 ceph-mon[37097]: pgmap v7560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:53 smithi171 ceph-mon[37789]: pgmap v7560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:55 smithi044 ceph-mon[32404]: pgmap v7561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:55 smithi116 ceph-mon[37097]: pgmap v7561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:55 smithi171 ceph-mon[37789]: pgmap v7561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13440.950310 secs 2024-07-26T12:24:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13440.950310 secs 2024-07-26T12:24:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13440.950310 secs 2024-07-26T12:24:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:57 smithi044 ceph-mon[32404]: pgmap v7562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:57 smithi116 ceph-mon[37097]: pgmap v7562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:57 smithi171 ceph-mon[37789]: pgmap v7562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:24:59 smithi044 ceph-mon[32404]: pgmap v7563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:24:59 smithi116 ceph-mon[37097]: pgmap v7563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:24:59 smithi171 ceph-mon[37789]: pgmap v7563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:00.114+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:25:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:00.114+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:00.114+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:25:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:01 smithi116 ceph-mon[37097]: pgmap v7564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13445.950472 secs 2024-07-26T12:25:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:01 smithi171 ceph-mon[37789]: pgmap v7564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13445.950472 secs 2024-07-26T12:25:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:01 smithi044 ceph-mon[32404]: pgmap v7564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13445.950472 secs 2024-07-26T12:25:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:03 smithi171 ceph-mon[37789]: pgmap v7565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:03 smithi044 ceph-mon[32404]: pgmap v7565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:03 smithi116 ceph-mon[37097]: pgmap v7565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:05.819 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:05 smithi044 ceph-mon[32404]: pgmap v7566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:05 smithi171 ceph-mon[37789]: pgmap v7566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:05 smithi116 ceph-mon[37097]: pgmap v7566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13450.950681 secs 2024-07-26T12:25:06.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13450.950681 secs 2024-07-26T12:25:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13450.950681 secs 2024-07-26T12:25:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:07 smithi171 ceph-mon[37789]: pgmap v7567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:07 smithi044 ceph-mon[32404]: pgmap v7567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:07 smithi116 ceph-mon[37097]: pgmap v7567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:09 smithi171 ceph-mon[37789]: pgmap v7568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:09 smithi044 ceph-mon[32404]: pgmap v7568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:10.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:09 smithi116 ceph-mon[37097]: pgmap v7568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:10 smithi044 ceph-mon[32404]: pgmap v7569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:10 smithi116 ceph-mon[37097]: pgmap v7569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:10 smithi171 ceph-mon[37789]: pgmap v7569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13455.950784 secs 2024-07-26T12:25:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13455.950784 secs 2024-07-26T12:25:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13455.950784 secs 2024-07-26T12:25:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:12 smithi044 ceph-mon[32404]: pgmap v7570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:12 smithi116 ceph-mon[37097]: pgmap v7570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:12 smithi171 ceph-mon[37789]: pgmap v7570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:15.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:15.120+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:25:15.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:15.120+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:15.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:15.120+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:25:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:15 smithi044 ceph-mon[32404]: pgmap v7571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:15 smithi116 ceph-mon[37097]: pgmap v7571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:15.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:15 smithi171 ceph-mon[37789]: pgmap v7571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13460.950918 secs 2024-07-26T12:25:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13460.950918 secs 2024-07-26T12:25:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13460.950918 secs 2024-07-26T12:25:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:17 smithi116 ceph-mon[37097]: pgmap v7572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:17 smithi171 ceph-mon[37789]: pgmap v7572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:17 smithi044 ceph-mon[32404]: pgmap v7572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:19 smithi116 ceph-mon[37097]: pgmap v7573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:25:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:19 smithi171 ceph-mon[37789]: pgmap v7573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:25:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:19 smithi044 ceph-mon[32404]: pgmap v7573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:25:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:21 smithi044 ceph-mon[32404]: pgmap v7574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13465.951090 secs 2024-07-26T12:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:21 smithi171 ceph-mon[37789]: pgmap v7574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13465.951090 secs 2024-07-26T12:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:21 smithi116 ceph-mon[37097]: pgmap v7574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13465.951090 secs 2024-07-26T12:25:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:23 smithi171 ceph-mon[37789]: pgmap v7575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:25:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:23 smithi044 ceph-mon[32404]: pgmap v7575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:25:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:23 smithi116 ceph-mon[37097]: pgmap v7575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:25:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:25 smithi171 ceph-mon[37789]: pgmap v7576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:25:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:25 smithi044 ceph-mon[32404]: pgmap v7576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:25:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:25 smithi116 ceph-mon[37097]: pgmap v7576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:25:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:26 smithi171 ceph-mon[37789]: pgmap v7577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:25:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:26 smithi171 ceph-mon[37789]: pgmap v7578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:26.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:25:26.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13470.951206 secs 2024-07-26T12:25:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:26 smithi044 ceph-mon[32404]: pgmap v7577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:25:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:26 smithi044 ceph-mon[32404]: pgmap v7578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:25:26.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13470.951206 secs 2024-07-26T12:25:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:26 smithi116 ceph-mon[37097]: pgmap v7577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:25:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:26 smithi116 ceph-mon[37097]: pgmap v7578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:25:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:25:26.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13470.951206 secs 2024-07-26T12:25:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:28 smithi044 ceph-mon[32404]: pgmap v7579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:28 smithi116 ceph-mon[37097]: pgmap v7579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:28 smithi171 ceph-mon[37789]: pgmap v7579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:30.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:30.121+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:25:30.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:30.121+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:30.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:30.121+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:25:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:30 smithi171 ceph-mon[37789]: pgmap v7580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:25:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:30 smithi044 ceph-mon[32404]: pgmap v7580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:25:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:30 smithi116 ceph-mon[37097]: pgmap v7580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:25:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13475.951373 secs 2024-07-26T12:25:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13475.951373 secs 2024-07-26T12:25:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13475.951373 secs 2024-07-26T12:25:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:32 smithi171 ceph-mon[37789]: pgmap v7581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:32 smithi044 ceph-mon[32404]: pgmap v7581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:33.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:32 smithi116 ceph-mon[37097]: pgmap v7581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:25:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:34 smithi171 ceph-mon[37789]: pgmap v7582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:25:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:34 smithi044 ceph-mon[32404]: pgmap v7582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:25:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:34 smithi116 ceph-mon[37097]: pgmap v7582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:25:36.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:36 smithi116 ceph-mon[37097]: pgmap v7583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:36.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13480.951508 secs 2024-07-26T12:25:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:36 smithi171 ceph-mon[37789]: pgmap v7583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13480.951508 secs 2024-07-26T12:25:36.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:36 smithi044 ceph-mon[32404]: pgmap v7583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:36.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13480.951508 secs 2024-07-26T12:25:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:38 smithi171 ceph-mon[37789]: pgmap v7584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:38 smithi044 ceph-mon[32404]: pgmap v7584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:38 smithi116 ceph-mon[37097]: pgmap v7584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:39 smithi171 ceph-mon[37789]: pgmap v7585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:39 smithi044 ceph-mon[32404]: pgmap v7585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:39 smithi116 ceph-mon[37097]: pgmap v7585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13485.951609 secs 2024-07-26T12:25:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13485.951609 secs 2024-07-26T12:25:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13485.951609 secs 2024-07-26T12:25:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:42 smithi171 ceph-mon[37789]: pgmap v7586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:42 smithi044 ceph-mon[32404]: pgmap v7586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:42 smithi116 ceph-mon[37097]: pgmap v7586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:44 smithi171 ceph-mon[37789]: pgmap v7587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:44 smithi044 ceph-mon[32404]: pgmap v7587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:44 smithi116 ceph-mon[37097]: pgmap v7587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:45.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:45.126+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:25:45.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:45.126+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:25:45.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:25:45.126+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:46 smithi171 ceph-mon[37789]: pgmap v7588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13490.951760 secs 2024-07-26T12:25:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:46 smithi044 ceph-mon[32404]: pgmap v7588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13490.951760 secs 2024-07-26T12:25:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:46 smithi116 ceph-mon[37097]: pgmap v7588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13490.951760 secs 2024-07-26T12:25:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:48 smithi171 ceph-mon[37789]: pgmap v7589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:48 smithi044 ceph-mon[32404]: pgmap v7589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:48 smithi116 ceph-mon[37097]: pgmap v7589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:50 smithi171 ceph-mon[37789]: pgmap v7590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:50 smithi044 ceph-mon[32404]: pgmap v7590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:50 smithi116 ceph-mon[37097]: pgmap v7590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:25:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13495.951926 secs 2024-07-26T12:25:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13495.951926 secs 2024-07-26T12:25:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13495.951926 secs 2024-07-26T12:25:52.808 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:52 smithi044 ceph-mon[32404]: pgmap v7591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:52 smithi171 ceph-mon[37789]: pgmap v7591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:52 smithi116 ceph-mon[37097]: pgmap v7591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:54 smithi171 ceph-mon[37789]: pgmap v7592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:25:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:54 smithi044 ceph-mon[32404]: pgmap v7592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:25:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:54 smithi116 ceph-mon[37097]: pgmap v7592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:25:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:55 smithi044 ceph-mon[32404]: pgmap v7593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:25:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:55 smithi116 ceph-mon[37097]: pgmap v7593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:25:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:55 smithi171 ceph-mon[37789]: pgmap v7593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:25:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13500.952067 secs 2024-07-26T12:25:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13500.952067 secs 2024-07-26T12:25:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13500.952067 secs 2024-07-26T12:25:58.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:25:57 smithi044 ceph-mon[32404]: pgmap v7594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:25:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:25:57 smithi116 ceph-mon[37097]: pgmap v7594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:25:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:25:57 smithi171 ceph-mon[37789]: pgmap v7594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:26:00.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:00.127+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:26:00.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:00.127+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:26:00.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:00.127+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:00 smithi171 ceph-mon[37789]: pgmap v7595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:26:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:00 smithi044 ceph-mon[32404]: pgmap v7595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:26:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:00 smithi116 ceph-mon[37097]: pgmap v7595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:26:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13505.952231 secs 2024-07-26T12:26:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13505.952231 secs 2024-07-26T12:26:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13505.952231 secs 2024-07-26T12:26:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:02 smithi171 ceph-mon[37789]: pgmap v7596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:26:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:02 smithi044 ceph-mon[32404]: pgmap v7596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:26:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:02 smithi116 ceph-mon[37097]: pgmap v7596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:26:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:04 smithi171 ceph-mon[37789]: pgmap v7597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:26:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:04 smithi044 ceph-mon[32404]: pgmap v7597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:26:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:04 smithi116 ceph-mon[37097]: pgmap v7597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:26:06.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:06 smithi116 ceph-mon[37097]: pgmap v7598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:06.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13510.952382 secs 2024-07-26T12:26:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:06 smithi171 ceph-mon[37789]: pgmap v7598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13510.952382 secs 2024-07-26T12:26:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:06 smithi044 ceph-mon[32404]: pgmap v7598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13510.952382 secs 2024-07-26T12:26:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:08 smithi171 ceph-mon[37789]: pgmap v7599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:08 smithi044 ceph-mon[32404]: pgmap v7599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:08 smithi116 ceph-mon[37097]: pgmap v7599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:09.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:09 smithi171 ceph-mon[37789]: pgmap v7600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:09 smithi044 ceph-mon[32404]: pgmap v7600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:09 smithi116 ceph-mon[37097]: pgmap v7600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13515.952503 secs 2024-07-26T12:26:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13515.952503 secs 2024-07-26T12:26:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13515.952503 secs 2024-07-26T12:26:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:12 smithi171 ceph-mon[37789]: pgmap v7601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:12 smithi044 ceph-mon[32404]: pgmap v7601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:12 smithi116 ceph-mon[37097]: pgmap v7601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:14.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T12:26:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:26:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58086000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:26:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58086000 comp rx=0 tx=0).stop 2024-07-26T12:26:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T12:26:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:26:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:26:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.730+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:26:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.731+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:26:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:14.733+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf5808d000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:26:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:14 smithi171 ceph-mon[37789]: pgmap v7602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:14 smithi044 ceph-mon[32404]: pgmap v7602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:14 smithi116 ceph-mon[37097]: pgmap v7602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:15.128+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:26:15.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:15.128+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:26:15.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:15.128+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:16 smithi171 ceph-mon[37789]: pgmap v7603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13520.952682 secs 2024-07-26T12:26:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:16 smithi044 ceph-mon[32404]: pgmap v7603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13520.952682 secs 2024-07-26T12:26:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:16 smithi116 ceph-mon[37097]: pgmap v7603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13520.952682 secs 2024-07-26T12:26:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:17.115+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:26:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:17.115+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:26:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:17.115+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:26:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:17.115+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:26:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:18 smithi171 ceph-mon[37789]: pgmap v7604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:18 smithi044 ceph-mon[32404]: pgmap v7604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:18 smithi116 ceph-mon[37097]: pgmap v7604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:20 smithi171 ceph-mon[37789]: pgmap v7605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:20 smithi044 ceph-mon[32404]: pgmap v7605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:20 smithi116 ceph-mon[37097]: pgmap v7605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13525.952839 secs 2024-07-26T12:26:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13525.952839 secs 2024-07-26T12:26:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13525.952839 secs 2024-07-26T12:26:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:22 smithi171 ceph-mon[37789]: pgmap v7606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:22 smithi044 ceph-mon[32404]: pgmap v7606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:22 smithi116 ceph-mon[37097]: pgmap v7606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:23 smithi044 ceph-mon[32404]: pgmap v7607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:26:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:23 smithi116 ceph-mon[37097]: pgmap v7607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:26:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:23 smithi171 ceph-mon[37789]: pgmap v7607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:26:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:26 smithi171 ceph-mon[37789]: pgmap v7608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13530.953017 secs 2024-07-26T12:26:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:26 smithi044 ceph-mon[32404]: pgmap v7608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13530.953017 secs 2024-07-26T12:26:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:26 smithi116 ceph-mon[37097]: pgmap v7608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13530.953017 secs 2024-07-26T12:26:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:26:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:26:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:28 smithi171 ceph-mon[37789]: pgmap v7609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:28 smithi171 ceph-mon[37789]: pgmap v7610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:26:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:28 smithi044 ceph-mon[32404]: pgmap v7609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:28 smithi044 ceph-mon[32404]: pgmap v7610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:26:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:28 smithi116 ceph-mon[37097]: pgmap v7609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:26:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:28 smithi116 ceph-mon[37097]: pgmap v7610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:26:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:29 smithi044 ceph-mon[32404]: pgmap v7611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:26:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:29 smithi116 ceph-mon[37097]: pgmap v7611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:26:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:29 smithi171 ceph-mon[37789]: pgmap v7611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:26:30.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:30.129+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:26:30.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:30.129+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:26:30.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:30.129+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13535.953168 secs 2024-07-26T12:26:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:31 smithi171 ceph-mon[37789]: pgmap v7612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:26:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13535.953168 secs 2024-07-26T12:26:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:31 smithi044 ceph-mon[32404]: pgmap v7612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:26:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13535.953168 secs 2024-07-26T12:26:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:31 smithi116 ceph-mon[37097]: pgmap v7612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:26:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:33 smithi171 ceph-mon[37789]: pgmap v7613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:26:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:33 smithi044 ceph-mon[32404]: pgmap v7613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:26:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:33 smithi116 ceph-mon[37097]: pgmap v7613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:26:36.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:35 smithi116 ceph-mon[37097]: pgmap v7614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:35 smithi171 ceph-mon[37789]: pgmap v7614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:35 smithi044 ceph-mon[32404]: pgmap v7614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13540.953345 secs 2024-07-26T12:26:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13540.953345 secs 2024-07-26T12:26:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13540.953345 secs 2024-07-26T12:26:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:37 smithi171 ceph-mon[37789]: pgmap v7615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:26:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:37 smithi044 ceph-mon[32404]: pgmap v7615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:26:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:37 smithi116 ceph-mon[37097]: pgmap v7615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:26:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:39 smithi171 ceph-mon[37789]: pgmap v7616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:26:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:39 smithi044 ceph-mon[32404]: pgmap v7616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:26:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:39 smithi116 ceph-mon[37097]: pgmap v7616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:26:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13545.953506 secs 2024-07-26T12:26:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:41 smithi171 ceph-mon[37789]: pgmap v7617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13545.953506 secs 2024-07-26T12:26:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:41 smithi044 ceph-mon[32404]: pgmap v7617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13545.953506 secs 2024-07-26T12:26:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:41 smithi116 ceph-mon[37097]: pgmap v7617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:43 smithi171 ceph-mon[37789]: pgmap v7618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:26:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:43 smithi044 ceph-mon[32404]: pgmap v7618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:26:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:43 smithi116 ceph-mon[37097]: pgmap v7618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:26:45.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:45.134+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:26:45.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:45.134+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:26:45.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:26:45.134+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:45 smithi171 ceph-mon[37789]: pgmap v7619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:45 smithi044 ceph-mon[32404]: pgmap v7619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:45 smithi116 ceph-mon[37097]: pgmap v7619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13550.953672 secs 2024-07-26T12:26:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13550.953672 secs 2024-07-26T12:26:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13550.953672 secs 2024-07-26T12:26:48.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:47 smithi044 ceph-mon[32404]: pgmap v7620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:47 smithi116 ceph-mon[37097]: pgmap v7620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:47 smithi171 ceph-mon[37789]: pgmap v7620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:49 smithi044 ceph-mon[32404]: pgmap v7621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:26:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:49 smithi116 ceph-mon[37097]: pgmap v7621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:26:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:49 smithi171 ceph-mon[37789]: pgmap v7621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:26:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13555.953857 secs 2024-07-26T12:26:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:51 smithi044 ceph-mon[32404]: pgmap v7622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13555.953857 secs 2024-07-26T12:26:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:51 smithi116 ceph-mon[37097]: pgmap v7622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13555.953857 secs 2024-07-26T12:26:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:51 smithi171 ceph-mon[37789]: pgmap v7622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:53 smithi044 ceph-mon[32404]: pgmap v7623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:53 smithi116 ceph-mon[37097]: pgmap v7623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:53 smithi171 ceph-mon[37789]: pgmap v7623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:26:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:55 smithi044 ceph-mon[32404]: pgmap v7624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:55 smithi116 ceph-mon[37097]: pgmap v7624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:55 smithi171 ceph-mon[37789]: pgmap v7624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:26:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13560.953965 secs 2024-07-26T12:26:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13560.953965 secs 2024-07-26T12:26:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13560.953965 secs 2024-07-26T12:26:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:57 smithi044 ceph-mon[32404]: pgmap v7625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:57 smithi116 ceph-mon[37097]: pgmap v7625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:26:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:57 smithi171 ceph-mon[37789]: pgmap v7625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:00.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:00.135+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:27:00.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:00.135+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:00.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:00.135+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:27:00.138 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:26:59 smithi044 ceph-mon[32404]: pgmap v7626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:26:59 smithi116 ceph-mon[37097]: pgmap v7626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:26:59 smithi171 ceph-mon[37789]: pgmap v7626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13565.954147 secs 2024-07-26T12:27:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:01 smithi044 ceph-mon[32404]: pgmap v7627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13565.954147 secs 2024-07-26T12:27:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:01 smithi116 ceph-mon[37097]: pgmap v7627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13565.954147 secs 2024-07-26T12:27:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:01 smithi171 ceph-mon[37789]: pgmap v7627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:03 smithi044 ceph-mon[32404]: pgmap v7628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:03 smithi116 ceph-mon[37097]: pgmap v7628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:03 smithi171 ceph-mon[37789]: pgmap v7628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:05 smithi044 ceph-mon[32404]: pgmap v7629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:05 smithi116 ceph-mon[37097]: pgmap v7629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:05 smithi171 ceph-mon[37789]: pgmap v7629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13570.954326 secs 2024-07-26T12:27:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13570.954326 secs 2024-07-26T12:27:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13570.954326 secs 2024-07-26T12:27:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:07 smithi044 ceph-mon[32404]: pgmap v7630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:07 smithi116 ceph-mon[37097]: pgmap v7630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:07 smithi171 ceph-mon[37789]: pgmap v7630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:09 smithi044 ceph-mon[32404]: pgmap v7631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:09 smithi116 ceph-mon[37097]: pgmap v7631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:09 smithi171 ceph-mon[37789]: pgmap v7631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13575.954505 secs 2024-07-26T12:27:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:11 smithi044 ceph-mon[32404]: pgmap v7632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13575.954505 secs 2024-07-26T12:27:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:11 smithi116 ceph-mon[37097]: pgmap v7632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13575.954505 secs 2024-07-26T12:27:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:11 smithi171 ceph-mon[37789]: pgmap v7632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:13 smithi044 ceph-mon[32404]: pgmap v7633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:13 smithi116 ceph-mon[37097]: pgmap v7633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:13 smithi171 ceph-mon[37789]: pgmap v7633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:15.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:15.140+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:27:15.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:15.140+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:27:15.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:15.140+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:15 smithi044 ceph-mon[32404]: pgmap v7634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:15 smithi116 ceph-mon[37097]: pgmap v7634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:15 smithi171 ceph-mon[37789]: pgmap v7634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13580.954602 secs 2024-07-26T12:27:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13580.954602 secs 2024-07-26T12:27:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13580.954602 secs 2024-07-26T12:27:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:17 smithi044 ceph-mon[32404]: pgmap v7635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:17 smithi116 ceph-mon[37097]: pgmap v7635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:17 smithi171 ceph-mon[37789]: pgmap v7635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:19 smithi044 ceph-mon[32404]: pgmap v7636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:19 smithi116 ceph-mon[37097]: pgmap v7636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:19 smithi171 ceph-mon[37789]: pgmap v7636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13585.954763 secs 2024-07-26T12:27:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:21 smithi044 ceph-mon[32404]: pgmap v7637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13585.954763 secs 2024-07-26T12:27:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:21 smithi116 ceph-mon[37097]: pgmap v7637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13585.954763 secs 2024-07-26T12:27:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:21 smithi171 ceph-mon[37789]: pgmap v7637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:23 smithi044 ceph-mon[32404]: pgmap v7638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:23 smithi116 ceph-mon[37097]: pgmap v7638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:23 smithi171 ceph-mon[37789]: pgmap v7638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:25 smithi044 ceph-mon[32404]: pgmap v7639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:25 smithi116 ceph-mon[37097]: pgmap v7639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:25 smithi171 ceph-mon[37789]: pgmap v7639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13590.954903 secs 2024-07-26T12:27:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13590.954903 secs 2024-07-26T12:27:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13590.954903 secs 2024-07-26T12:27:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:27 smithi044 ceph-mon[32404]: pgmap v7640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:27:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:28 smithi116 ceph-mon[37097]: pgmap v7640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:27:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:27 smithi171 ceph-mon[37789]: pgmap v7640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:28.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:27:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:29 smithi044 ceph-mon[32404]: pgmap v7641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:27:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:29 smithi044 ceph-mon[32404]: pgmap v7642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:29 smithi116 ceph-mon[37097]: pgmap v7641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:27:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:29 smithi116 ceph-mon[37097]: pgmap v7642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:29 smithi171 ceph-mon[37789]: pgmap v7641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:27:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:29 smithi171 ceph-mon[37789]: pgmap v7642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:30.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:30.141+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:27:30.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:30.141+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:27:30.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:30.141+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:31 smithi044 ceph-mon[32404]: pgmap v7643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13595.955064 secs 2024-07-26T12:27:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:31 smithi116 ceph-mon[37097]: pgmap v7643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13595.955064 secs 2024-07-26T12:27:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:31 smithi171 ceph-mon[37789]: pgmap v7643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13595.955064 secs 2024-07-26T12:27:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:33 smithi044 ceph-mon[32404]: pgmap v7644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:33 smithi116 ceph-mon[37097]: pgmap v7644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:33 smithi171 ceph-mon[37789]: pgmap v7644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:35 smithi116 ceph-mon[37097]: pgmap v7645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:35 smithi171 ceph-mon[37789]: pgmap v7645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:35 smithi044 ceph-mon[32404]: pgmap v7645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:27:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13600.955186 secs 2024-07-26T12:27:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13600.955186 secs 2024-07-26T12:27:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13600.955186 secs 2024-07-26T12:27:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:37 smithi171 ceph-mon[37789]: pgmap v7646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:27:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:37 smithi044 ceph-mon[32404]: pgmap v7646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:27:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:37 smithi116 ceph-mon[37097]: pgmap v7646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:27:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:39 smithi171 ceph-mon[37789]: pgmap v7647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:39 smithi044 ceph-mon[32404]: pgmap v7647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:39 smithi116 ceph-mon[37097]: pgmap v7647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:41 smithi044 ceph-mon[32404]: pgmap v7648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13605.955332 secs 2024-07-26T12:27:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:41 smithi171 ceph-mon[37789]: pgmap v7648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13605.955332 secs 2024-07-26T12:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:41 smithi116 ceph-mon[37097]: pgmap v7648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13605.955332 secs 2024-07-26T12:27:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:43 smithi171 ceph-mon[37789]: pgmap v7649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:43 smithi044 ceph-mon[32404]: pgmap v7649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:43 smithi116 ceph-mon[37097]: pgmap v7649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:45.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:45.146+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:27:45.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:45.146+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:27:45.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:27:45.146+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:45.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:45 smithi044 ceph-mon[32404]: pgmap v7650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:45 smithi171 ceph-mon[37789]: pgmap v7650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:45 smithi116 ceph-mon[37097]: pgmap v7650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13610.955494 secs 2024-07-26T12:27:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13610.955494 secs 2024-07-26T12:27:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13610.955494 secs 2024-07-26T12:27:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:47 smithi171 ceph-mon[37789]: pgmap v7651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:47 smithi044 ceph-mon[32404]: pgmap v7651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:47 smithi116 ceph-mon[37097]: pgmap v7651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:49 smithi171 ceph-mon[37789]: pgmap v7652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:49 smithi044 ceph-mon[32404]: pgmap v7652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:49 smithi116 ceph-mon[37097]: pgmap v7652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:51 smithi171 ceph-mon[37789]: pgmap v7653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13615.955706 secs 2024-07-26T12:27:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:51 smithi044 ceph-mon[32404]: pgmap v7653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13615.955706 secs 2024-07-26T12:27:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:51 smithi116 ceph-mon[37097]: pgmap v7653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:27:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13615.955706 secs 2024-07-26T12:27:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:53 smithi044 ceph-mon[32404]: pgmap v7654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:53 smithi116 ceph-mon[37097]: pgmap v7654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:53 smithi171 ceph-mon[37789]: pgmap v7654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:55 smithi044 ceph-mon[32404]: pgmap v7655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:55 smithi116 ceph-mon[37097]: pgmap v7655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:55 smithi171 ceph-mon[37789]: pgmap v7655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13620.955813 secs 2024-07-26T12:27:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13620.955813 secs 2024-07-26T12:27:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13620.955813 secs 2024-07-26T12:27:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:57 smithi044 ceph-mon[32404]: pgmap v7656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:57 smithi116 ceph-mon[37097]: pgmap v7656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:57 smithi171 ceph-mon[37789]: pgmap v7656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:27:59 smithi044 ceph-mon[32404]: pgmap v7657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:27:59 smithi116 ceph-mon[37097]: pgmap v7657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:27:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:27:59 smithi171 ceph-mon[37789]: pgmap v7657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:00.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:00.148+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:28:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:00.148+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:00.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:00.148+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:28:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:01 smithi044 ceph-mon[32404]: pgmap v7658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13625.955978 secs 2024-07-26T12:28:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:01 smithi116 ceph-mon[37097]: pgmap v7658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13625.955978 secs 2024-07-26T12:28:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:01 smithi171 ceph-mon[37789]: pgmap v7658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13625.955978 secs 2024-07-26T12:28:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:03 smithi044 ceph-mon[32404]: pgmap v7659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:03 smithi116 ceph-mon[37097]: pgmap v7659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:03 smithi171 ceph-mon[37789]: pgmap v7659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:05 smithi044 ceph-mon[32404]: pgmap v7660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:05 smithi116 ceph-mon[37097]: pgmap v7660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:05 smithi171 ceph-mon[37789]: pgmap v7660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:06.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13630.956142 secs 2024-07-26T12:28:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13630.956142 secs 2024-07-26T12:28:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13630.956142 secs 2024-07-26T12:28:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:07 smithi044 ceph-mon[32404]: pgmap v7661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:07 smithi116 ceph-mon[37097]: pgmap v7661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:07 smithi171 ceph-mon[37789]: pgmap v7661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:09 smithi044 ceph-mon[32404]: pgmap v7662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:09 smithi116 ceph-mon[37097]: pgmap v7662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:09 smithi171 ceph-mon[37789]: pgmap v7662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:11 smithi044 ceph-mon[32404]: pgmap v7663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13635.956303 secs 2024-07-26T12:28:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:11 smithi116 ceph-mon[37097]: pgmap v7663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13635.956303 secs 2024-07-26T12:28:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:11 smithi171 ceph-mon[37789]: pgmap v7663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13635.956303 secs 2024-07-26T12:28:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:13 smithi044 ceph-mon[32404]: pgmap v7664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:13 smithi116 ceph-mon[37097]: pgmap v7664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:13 smithi171 ceph-mon[37789]: pgmap v7664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:15.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:15.153+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:28:15.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:15.153+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:15.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:15.153+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:28:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:15 smithi044 ceph-mon[32404]: pgmap v7665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:15 smithi116 ceph-mon[37097]: pgmap v7665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:15 smithi171 ceph-mon[37789]: pgmap v7665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:16.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13640.956482 secs 2024-07-26T12:28:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13640.956482 secs 2024-07-26T12:28:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13640.956482 secs 2024-07-26T12:28:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:17 smithi044 ceph-mon[32404]: pgmap v7666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:17 smithi116 ceph-mon[37097]: pgmap v7666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:17 smithi171 ceph-mon[37789]: pgmap v7666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:19 smithi044 ceph-mon[32404]: pgmap v7667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:19 smithi116 ceph-mon[37097]: pgmap v7667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:19 smithi171 ceph-mon[37789]: pgmap v7667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:21 smithi044 ceph-mon[32404]: pgmap v7668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13645.956687 secs 2024-07-26T12:28:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:21 smithi116 ceph-mon[37097]: pgmap v7668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13645.956687 secs 2024-07-26T12:28:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:21 smithi171 ceph-mon[37789]: pgmap v7668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13645.956687 secs 2024-07-26T12:28:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:23 smithi044 ceph-mon[32404]: pgmap v7669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:23 smithi116 ceph-mon[37097]: pgmap v7669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:23 smithi171 ceph-mon[37789]: pgmap v7669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:25 smithi044 ceph-mon[32404]: pgmap v7670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:25 smithi116 ceph-mon[37097]: pgmap v7670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:25 smithi171 ceph-mon[37789]: pgmap v7670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13650.956865 secs 2024-07-26T12:28:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13650.956865 secs 2024-07-26T12:28:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13650.956865 secs 2024-07-26T12:28:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:27 smithi044 ceph-mon[32404]: pgmap v7671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:27 smithi116 ceph-mon[37097]: pgmap v7671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:27 smithi171 ceph-mon[37789]: pgmap v7671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:29 smithi044 ceph-mon[32404]: pgmap v7672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:29 smithi116 ceph-mon[37097]: pgmap v7672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:29 smithi171 ceph-mon[37789]: pgmap v7672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:30.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:30.155+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:28:30.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:30.155+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:30.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:30.155+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:28:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:28:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:28:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:30.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: pgmap v7673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:28:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: pgmap v7674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:30.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:28:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:28:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:28:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: pgmap v7673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:28:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: pgmap v7674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:30.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:28:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:28:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:28:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:28:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:30.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: pgmap v7673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:28:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: pgmap v7674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:30.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:28:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13655.957048 secs 2024-07-26T12:28:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13655.957048 secs 2024-07-26T12:28:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13655.957048 secs 2024-07-26T12:28:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:32 smithi044 ceph-mon[32404]: pgmap v7675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:28:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:32 smithi116 ceph-mon[37097]: pgmap v7675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:28:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:32 smithi171 ceph-mon[37789]: pgmap v7675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:28:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:34 smithi044 ceph-mon[32404]: pgmap v7676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:34 smithi116 ceph-mon[37097]: pgmap v7676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:34 smithi171 ceph-mon[37789]: pgmap v7676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:36 smithi044 ceph-mon[32404]: pgmap v7677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13660.957218 secs 2024-07-26T12:28:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:36 smithi116 ceph-mon[37097]: pgmap v7677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13660.957218 secs 2024-07-26T12:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:36 smithi171 ceph-mon[37789]: pgmap v7677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13660.957218 secs 2024-07-26T12:28:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:38 smithi044 ceph-mon[32404]: pgmap v7678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:38 smithi116 ceph-mon[37097]: pgmap v7678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:38 smithi171 ceph-mon[37789]: pgmap v7678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:40.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:40 smithi044 ceph-mon[32404]: pgmap v7679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:40 smithi116 ceph-mon[37097]: pgmap v7679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:40 smithi171 ceph-mon[37789]: pgmap v7679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13665.957396 secs 2024-07-26T12:28:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13665.957396 secs 2024-07-26T12:28:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13665.957396 secs 2024-07-26T12:28:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:42 smithi044 ceph-mon[32404]: pgmap v7680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:42 smithi116 ceph-mon[37097]: pgmap v7680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:42 smithi171 ceph-mon[37789]: pgmap v7680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:44 smithi044 ceph-mon[32404]: pgmap v7681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:44 smithi116 ceph-mon[37097]: pgmap v7681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:44 smithi171 ceph-mon[37789]: pgmap v7681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:45.156+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:28:45.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:45.156+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:28:45.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:28:45.157+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:46.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:46 smithi171 ceph-mon[37789]: pgmap v7682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:46.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13670.957575 secs 2024-07-26T12:28:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:46 smithi044 ceph-mon[32404]: pgmap v7682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13670.957575 secs 2024-07-26T12:28:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:46 smithi116 ceph-mon[37097]: pgmap v7682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13670.957575 secs 2024-07-26T12:28:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:48 smithi044 ceph-mon[32404]: pgmap v7683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:48 smithi116 ceph-mon[37097]: pgmap v7683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:48 smithi171 ceph-mon[37789]: pgmap v7683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:50 smithi044 ceph-mon[32404]: pgmap v7684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:50 smithi116 ceph-mon[37097]: pgmap v7684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:50 smithi171 ceph-mon[37789]: pgmap v7684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:28:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13675.957721 secs 2024-07-26T12:28:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13675.957721 secs 2024-07-26T12:28:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13675.957721 secs 2024-07-26T12:28:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:52 smithi044 ceph-mon[32404]: pgmap v7685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:52 smithi116 ceph-mon[37097]: pgmap v7685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:52 smithi171 ceph-mon[37789]: pgmap v7685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:54 smithi044 ceph-mon[32404]: pgmap v7686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:54 smithi116 ceph-mon[37097]: pgmap v7686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:54 smithi171 ceph-mon[37789]: pgmap v7686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:56 smithi044 ceph-mon[32404]: pgmap v7687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13680.957890 secs 2024-07-26T12:28:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:56 smithi116 ceph-mon[37097]: pgmap v7687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13680.957890 secs 2024-07-26T12:28:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:56 smithi171 ceph-mon[37789]: pgmap v7687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13680.957890 secs 2024-07-26T12:28:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:28:58 smithi044 ceph-mon[32404]: pgmap v7688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:28:58 smithi116 ceph-mon[37097]: pgmap v7688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:28:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:28:58 smithi171 ceph-mon[37789]: pgmap v7688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:00.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:00.157+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:29:00.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:00.157+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:29:00.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:00.157+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:00 smithi044 ceph-mon[32404]: pgmap v7689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:00 smithi116 ceph-mon[37097]: pgmap v7689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:00 smithi171 ceph-mon[37789]: pgmap v7689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13685.958039 secs 2024-07-26T12:29:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13685.958039 secs 2024-07-26T12:29:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13685.958039 secs 2024-07-26T12:29:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:02 smithi044 ceph-mon[32404]: pgmap v7690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:02 smithi116 ceph-mon[37097]: pgmap v7690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:02.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:02 smithi171 ceph-mon[37789]: pgmap v7690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:04 smithi044 ceph-mon[32404]: pgmap v7691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:04 smithi116 ceph-mon[37097]: pgmap v7691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:04 smithi171 ceph-mon[37789]: pgmap v7691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:06 smithi044 ceph-mon[32404]: pgmap v7692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13690.958161 secs 2024-07-26T12:29:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:06 smithi116 ceph-mon[37097]: pgmap v7692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13690.958161 secs 2024-07-26T12:29:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:06 smithi171 ceph-mon[37789]: pgmap v7692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13690.958161 secs 2024-07-26T12:29:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:08 smithi044 ceph-mon[32404]: pgmap v7693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:08 smithi116 ceph-mon[37097]: pgmap v7693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:08 smithi171 ceph-mon[37789]: pgmap v7693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:10 smithi044 ceph-mon[32404]: pgmap v7694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:10 smithi116 ceph-mon[37097]: pgmap v7694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:10 smithi171 ceph-mon[37789]: pgmap v7694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13695.958315 secs 2024-07-26T12:29:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13695.958315 secs 2024-07-26T12:29:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13695.958315 secs 2024-07-26T12:29:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:12 smithi044 ceph-mon[32404]: pgmap v7695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:12 smithi116 ceph-mon[37097]: pgmap v7695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:12 smithi171 ceph-mon[37789]: pgmap v7695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:14 smithi044 ceph-mon[32404]: pgmap v7696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:14 smithi116 ceph-mon[37097]: pgmap v7696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:14 smithi171 ceph-mon[37789]: pgmap v7696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:15.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:15.161+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:29:15.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:15.161+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:15.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:15.161+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:29:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:16 smithi044 ceph-mon[32404]: pgmap v7697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13700.958414 secs 2024-07-26T12:29:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:16 smithi116 ceph-mon[37097]: pgmap v7697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13700.958414 secs 2024-07-26T12:29:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:16 smithi171 ceph-mon[37789]: pgmap v7697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13700.958414 secs 2024-07-26T12:29:17.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:17.147+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:29:17.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:17.147+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:29:17.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:17.148+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:29:17.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:17.148+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:29:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:18 smithi044 ceph-mon[32404]: pgmap v7698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:18 smithi116 ceph-mon[37097]: pgmap v7698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:18 smithi171 ceph-mon[37789]: pgmap v7698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:20 smithi116 ceph-mon[37097]: pgmap v7699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:20 smithi171 ceph-mon[37789]: pgmap v7699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:21.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:20 smithi044 ceph-mon[32404]: pgmap v7699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13705.958548 secs 2024-07-26T12:29:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13705.958548 secs 2024-07-26T12:29:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13705.958548 secs 2024-07-26T12:29:22.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:22 smithi171 ceph-mon[37789]: pgmap v7700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:22 smithi044 ceph-mon[32404]: pgmap v7700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:22 smithi116 ceph-mon[37097]: pgmap v7700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:24 smithi171 ceph-mon[37789]: pgmap v7701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:24 smithi044 ceph-mon[32404]: pgmap v7701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:24 smithi116 ceph-mon[37097]: pgmap v7701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:26 smithi171 ceph-mon[37789]: pgmap v7702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13710.958703 secs 2024-07-26T12:29:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:26 smithi044 ceph-mon[32404]: pgmap v7702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13710.958703 secs 2024-07-26T12:29:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:26 smithi116 ceph-mon[37097]: pgmap v7702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13710.958703 secs 2024-07-26T12:29:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:28 smithi171 ceph-mon[37789]: pgmap v7703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:28 smithi044 ceph-mon[32404]: pgmap v7703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:28 smithi116 ceph-mon[37097]: pgmap v7703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:29.891 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:30.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:30.163+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:29:30.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:30.163+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:29:30.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:30.163+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: pgmap v7704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:30.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:30.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:29:31.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: pgmap v7704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:31.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:31.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:29:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: pgmap v7704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:31.025 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:29:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:29:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:29:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:29:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:29:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: pgmap v7705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: pgmap v7706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:29:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13715.958779 secs 2024-07-26T12:29:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:29:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:29:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:29:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:29:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: pgmap v7705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: pgmap v7706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:29:32.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13715.958779 secs 2024-07-26T12:29:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:29:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:29:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:29:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:29:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: pgmap v7705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: pgmap v7706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:29:32.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13715.958779 secs 2024-07-26T12:29:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:32 smithi044 ceph-mon[32404]: pgmap v7707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:29:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:32 smithi116 ceph-mon[37097]: pgmap v7707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:29:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:32 smithi171 ceph-mon[37789]: pgmap v7707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:29:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:35 smithi044 ceph-mon[32404]: pgmap v7708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:29:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:35 smithi116 ceph-mon[37097]: pgmap v7708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:29:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:35 smithi171 ceph-mon[37789]: pgmap v7708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:29:36.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13720.958903 secs 2024-07-26T12:29:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13720.958903 secs 2024-07-26T12:29:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13720.958903 secs 2024-07-26T12:29:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:37 smithi044 ceph-mon[32404]: pgmap v7709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:29:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:37 smithi116 ceph-mon[37097]: pgmap v7709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:29:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:37 smithi171 ceph-mon[37789]: pgmap v7709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:29:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:38 smithi044 ceph-mon[32404]: pgmap v7710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:38 smithi116 ceph-mon[37097]: pgmap v7710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:38 smithi171 ceph-mon[37789]: pgmap v7710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:41 smithi044 ceph-mon[32404]: pgmap v7711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:29:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13725.959024 secs 2024-07-26T12:29:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:41 smithi116 ceph-mon[37097]: pgmap v7711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:29:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13725.959024 secs 2024-07-26T12:29:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:41 smithi171 ceph-mon[37789]: pgmap v7711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:29:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13725.959024 secs 2024-07-26T12:29:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:42 smithi044 ceph-mon[32404]: pgmap v7712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:42 smithi116 ceph-mon[37097]: pgmap v7712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:42 smithi171 ceph-mon[37789]: pgmap v7712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:29:45.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:45.169+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:29:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:45.169+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:29:45.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:29:45.169+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:45.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:45 smithi044 ceph-mon[32404]: pgmap v7713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:29:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:45 smithi116 ceph-mon[37097]: pgmap v7713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:29:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:45 smithi171 ceph-mon[37789]: pgmap v7713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:29:46.987 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13730.959152 secs 2024-07-26T12:29:46.987 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:46 smithi171 ceph-mon[37789]: pgmap v7714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13730.959152 secs 2024-07-26T12:29:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:46 smithi044 ceph-mon[32404]: pgmap v7714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13730.959152 secs 2024-07-26T12:29:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:46 smithi116 ceph-mon[37097]: pgmap v7714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:49 smithi044 ceph-mon[32404]: pgmap v7715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:49 smithi116 ceph-mon[37097]: pgmap v7715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:49 smithi171 ceph-mon[37789]: pgmap v7715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:50 smithi044 ceph-mon[32404]: pgmap v7716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:50 smithi116 ceph-mon[37097]: pgmap v7716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:50 smithi171 ceph-mon[37789]: pgmap v7716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13735.959307 secs 2024-07-26T12:29:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13735.959307 secs 2024-07-26T12:29:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13735.959307 secs 2024-07-26T12:29:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:52 smithi044 ceph-mon[32404]: pgmap v7717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:52 smithi116 ceph-mon[37097]: pgmap v7717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:52 smithi171 ceph-mon[37789]: pgmap v7717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:29:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:54 smithi044 ceph-mon[32404]: pgmap v7718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:54 smithi116 ceph-mon[37097]: pgmap v7718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:54 smithi171 ceph-mon[37789]: pgmap v7718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:29:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13740.959459 secs 2024-07-26T12:29:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13740.959459 secs 2024-07-26T12:29:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13740.959459 secs 2024-07-26T12:29:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:57 smithi171 ceph-mon[37789]: pgmap v7719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:57 smithi044 ceph-mon[32404]: pgmap v7719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:57 smithi116 ceph-mon[37097]: pgmap v7719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:29:58 smithi044 ceph-mon[32404]: pgmap v7720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:29:58 smithi116 ceph-mon[37097]: pgmap v7720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:29:58 smithi171 ceph-mon[37789]: pgmap v7720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:00.169+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:30:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:00.169+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:30:00.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:00.169+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:30:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:01 smithi171 ceph-mon[37789]: pgmap v7721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13745.959597 secs 2024-07-26T12:30:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:01 smithi044 ceph-mon[32404]: pgmap v7721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13745.959597 secs 2024-07-26T12:30:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:01 smithi116 ceph-mon[37097]: pgmap v7721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13745.959597 secs 2024-07-26T12:30:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:02 smithi044 ceph-mon[32404]: pgmap v7722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:02 smithi116 ceph-mon[37097]: pgmap v7722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:02 smithi171 ceph-mon[37789]: pgmap v7722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:04 smithi044 ceph-mon[32404]: pgmap v7723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:04 smithi116 ceph-mon[37097]: pgmap v7723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:04 smithi171 ceph-mon[37789]: pgmap v7723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13750.959759 secs 2024-07-26T12:30:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13750.959759 secs 2024-07-26T12:30:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13750.959759 secs 2024-07-26T12:30:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:07 smithi171 ceph-mon[37789]: pgmap v7724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:07 smithi044 ceph-mon[32404]: pgmap v7724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:07 smithi116 ceph-mon[37097]: pgmap v7724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:08 smithi044 ceph-mon[32404]: pgmap v7725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:08 smithi116 ceph-mon[37097]: pgmap v7725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:08 smithi171 ceph-mon[37789]: pgmap v7725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:10 smithi044 ceph-mon[32404]: pgmap v7726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:10 smithi116 ceph-mon[37097]: pgmap v7726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:10 smithi171 ceph-mon[37789]: pgmap v7726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13755.959900 secs 2024-07-26T12:30:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13755.959900 secs 2024-07-26T12:30:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13755.959900 secs 2024-07-26T12:30:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:12 smithi044 ceph-mon[32404]: pgmap v7727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:12 smithi116 ceph-mon[37097]: pgmap v7727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:12 smithi171 ceph-mon[37789]: pgmap v7727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:14 smithi044 ceph-mon[32404]: pgmap v7728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:14 smithi116 ceph-mon[37097]: pgmap v7728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:14 smithi171 ceph-mon[37789]: pgmap v7728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:15.174+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:30:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:15.174+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:30:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:15.174+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13760.960056 secs 2024-07-26T12:30:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13760.960056 secs 2024-07-26T12:30:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13760.960056 secs 2024-07-26T12:30:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:17 smithi171 ceph-mon[37789]: pgmap v7729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:17 smithi044 ceph-mon[32404]: pgmap v7729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:17 smithi116 ceph-mon[37097]: pgmap v7729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:18 smithi044 ceph-mon[32404]: pgmap v7730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:18 smithi116 ceph-mon[37097]: pgmap v7730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:18 smithi171 ceph-mon[37789]: pgmap v7730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:20 smithi044 ceph-mon[32404]: pgmap v7731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:20 smithi116 ceph-mon[37097]: pgmap v7731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:20 smithi171 ceph-mon[37789]: pgmap v7731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13765.960238 secs 2024-07-26T12:30:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13765.960238 secs 2024-07-26T12:30:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13765.960238 secs 2024-07-26T12:30:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:22 smithi044 ceph-mon[32404]: pgmap v7732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:22 smithi116 ceph-mon[37097]: pgmap v7732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:22 smithi171 ceph-mon[37789]: pgmap v7732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:24 smithi044 ceph-mon[32404]: pgmap v7733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:24 smithi116 ceph-mon[37097]: pgmap v7733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:25.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:24 smithi171 ceph-mon[37789]: pgmap v7733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13770.960420 secs 2024-07-26T12:30:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13770.960420 secs 2024-07-26T12:30:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13770.960420 secs 2024-07-26T12:30:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:27 smithi171 ceph-mon[37789]: pgmap v7734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:27 smithi044 ceph-mon[32404]: pgmap v7734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:27 smithi116 ceph-mon[37097]: pgmap v7734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:28 smithi044 ceph-mon[32404]: pgmap v7735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:28 smithi116 ceph-mon[37097]: pgmap v7735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:28 smithi171 ceph-mon[37789]: pgmap v7735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:30.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:30.176+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:30:30.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:30.176+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:30:30.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:30.176+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:30 smithi044 ceph-mon[32404]: pgmap v7736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:30 smithi116 ceph-mon[37097]: pgmap v7736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:30 smithi171 ceph-mon[37789]: pgmap v7736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13775.960586 secs 2024-07-26T12:30:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13775.960586 secs 2024-07-26T12:30:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13775.960586 secs 2024-07-26T12:30:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:32 smithi044 ceph-mon[32404]: pgmap v7737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:32 smithi044 ceph-mon[32404]: pgmap v7738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:33.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:30:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:32 smithi116 ceph-mon[37097]: pgmap v7737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:32 smithi116 ceph-mon[37097]: pgmap v7738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:30:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:32 smithi171 ceph-mon[37789]: pgmap v7737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:32 smithi171 ceph-mon[37789]: pgmap v7738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:30:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:34 smithi044 ceph-mon[32404]: pgmap v7739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:34 smithi116 ceph-mon[37097]: pgmap v7739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:34 smithi171 ceph-mon[37789]: pgmap v7739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13780.960764 secs 2024-07-26T12:30:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:36 smithi044 ceph-mon[32404]: pgmap v7740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:30:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13780.960764 secs 2024-07-26T12:30:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:36 smithi116 ceph-mon[37097]: pgmap v7740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:30:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13780.960764 secs 2024-07-26T12:30:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:36 smithi171 ceph-mon[37789]: pgmap v7740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:30:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:38 smithi044 ceph-mon[32404]: pgmap v7741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:30:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:38 smithi116 ceph-mon[37097]: pgmap v7741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:30:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:38 smithi171 ceph-mon[37789]: pgmap v7741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:30:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:40 smithi044 ceph-mon[32404]: pgmap v7742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:40 smithi116 ceph-mon[37097]: pgmap v7742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:40 smithi171 ceph-mon[37789]: pgmap v7742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:30:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13785.960907 secs 2024-07-26T12:30:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13785.960907 secs 2024-07-26T12:30:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13785.960907 secs 2024-07-26T12:30:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:42 smithi044 ceph-mon[32404]: pgmap v7743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:42 smithi116 ceph-mon[37097]: pgmap v7743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:42 smithi171 ceph-mon[37789]: pgmap v7743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:45.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:45.179+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:30:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:45.179+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:30:45.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:30:45.179+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:44 smithi044 ceph-mon[32404]: pgmap v7744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:44 smithi116 ceph-mon[37097]: pgmap v7744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:44 smithi171 ceph-mon[37789]: pgmap v7744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13790.961051 secs 2024-07-26T12:30:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:46 smithi044 ceph-mon[32404]: pgmap v7745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13790.961051 secs 2024-07-26T12:30:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:46 smithi116 ceph-mon[37097]: pgmap v7745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13790.961051 secs 2024-07-26T12:30:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:46 smithi171 ceph-mon[37789]: pgmap v7745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:48 smithi044 ceph-mon[32404]: pgmap v7746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:48 smithi116 ceph-mon[37097]: pgmap v7746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:48 smithi171 ceph-mon[37789]: pgmap v7746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:50 smithi044 ceph-mon[32404]: pgmap v7747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:50 smithi116 ceph-mon[37097]: pgmap v7747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:50 smithi171 ceph-mon[37789]: pgmap v7747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:30:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13795.961199 secs 2024-07-26T12:30:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13795.961199 secs 2024-07-26T12:30:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13795.961199 secs 2024-07-26T12:30:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:52 smithi044 ceph-mon[32404]: pgmap v7748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:52 smithi116 ceph-mon[37097]: pgmap v7748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:53.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:52 smithi171 ceph-mon[37789]: pgmap v7748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:54 smithi044 ceph-mon[32404]: pgmap v7749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:54 smithi116 ceph-mon[37097]: pgmap v7749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:54 smithi171 ceph-mon[37789]: pgmap v7749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13800.961360 secs 2024-07-26T12:30:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:56 smithi044 ceph-mon[32404]: pgmap v7750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13800.961360 secs 2024-07-26T12:30:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:56 smithi116 ceph-mon[37097]: pgmap v7750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13800.961360 secs 2024-07-26T12:30:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:56 smithi171 ceph-mon[37789]: pgmap v7750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:30:58 smithi044 ceph-mon[32404]: pgmap v7751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:30:58 smithi116 ceph-mon[37097]: pgmap v7751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:30:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:30:58 smithi171 ceph-mon[37789]: pgmap v7751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:00.179+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:31:00.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:00.179+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:31:00.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:00.179+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:00 smithi044 ceph-mon[32404]: pgmap v7752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:00 smithi116 ceph-mon[37097]: pgmap v7752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:00 smithi171 ceph-mon[37789]: pgmap v7752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13805.961510 secs 2024-07-26T12:31:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13805.961510 secs 2024-07-26T12:31:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13805.961510 secs 2024-07-26T12:31:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:02 smithi044 ceph-mon[32404]: pgmap v7753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:02 smithi116 ceph-mon[37097]: pgmap v7753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:02 smithi171 ceph-mon[37789]: pgmap v7753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:04 smithi044 ceph-mon[32404]: pgmap v7754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:04 smithi116 ceph-mon[37097]: pgmap v7754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:04 smithi171 ceph-mon[37789]: pgmap v7754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13810.961655 secs 2024-07-26T12:31:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:07 smithi044 ceph-mon[32404]: pgmap v7755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13810.961655 secs 2024-07-26T12:31:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:07 smithi116 ceph-mon[37097]: pgmap v7755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13810.961655 secs 2024-07-26T12:31:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:07 smithi171 ceph-mon[37789]: pgmap v7755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:09 smithi044 ceph-mon[32404]: pgmap v7756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:09 smithi116 ceph-mon[37097]: pgmap v7756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:09 smithi171 ceph-mon[37789]: pgmap v7756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:11 smithi116 ceph-mon[37097]: pgmap v7757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:11 smithi171 ceph-mon[37789]: pgmap v7757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:11 smithi044 ceph-mon[32404]: pgmap v7757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13815.961794 secs 2024-07-26T12:31:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13815.961794 secs 2024-07-26T12:31:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13815.961794 secs 2024-07-26T12:31:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:13 smithi171 ceph-mon[37789]: pgmap v7758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:13 smithi044 ceph-mon[32404]: pgmap v7758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:13 smithi116 ceph-mon[37097]: pgmap v7758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:15.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:15.185+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:31:15.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:15.185+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:31:15.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:15.185+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:15.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:15 smithi044 ceph-mon[32404]: pgmap v7759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:15 smithi171 ceph-mon[37789]: pgmap v7759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:15 smithi116 ceph-mon[37097]: pgmap v7759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13820.961934 secs 2024-07-26T12:31:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13820.961934 secs 2024-07-26T12:31:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13820.961934 secs 2024-07-26T12:31:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:17 smithi171 ceph-mon[37789]: pgmap v7760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:31:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:17 smithi044 ceph-mon[32404]: pgmap v7760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:31:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:17 smithi116 ceph-mon[37097]: pgmap v7760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:31:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:19 smithi044 ceph-mon[32404]: pgmap v7761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:19 smithi171 ceph-mon[37789]: pgmap v7761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:19 smithi116 ceph-mon[37097]: pgmap v7761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:21 smithi171 ceph-mon[37789]: pgmap v7762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:31:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13825.962107 secs 2024-07-26T12:31:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:21 smithi044 ceph-mon[32404]: pgmap v7762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:31:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13825.962107 secs 2024-07-26T12:31:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:21 smithi116 ceph-mon[37097]: pgmap v7762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:31:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13825.962107 secs 2024-07-26T12:31:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:23 smithi171 ceph-mon[37789]: pgmap v7763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:23 smithi044 ceph-mon[32404]: pgmap v7763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:23 smithi116 ceph-mon[37097]: pgmap v7763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:31:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:25 smithi171 ceph-mon[37789]: pgmap v7764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:25 smithi044 ceph-mon[32404]: pgmap v7764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:25 smithi116 ceph-mon[37097]: pgmap v7764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13830.962278 secs 2024-07-26T12:31:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13830.962278 secs 2024-07-26T12:31:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13830.962278 secs 2024-07-26T12:31:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:27 smithi044 ceph-mon[32404]: pgmap v7765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:27 smithi116 ceph-mon[37097]: pgmap v7765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:27 smithi171 ceph-mon[37789]: pgmap v7765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:29 smithi044 ceph-mon[32404]: pgmap v7766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:29 smithi116 ceph-mon[37097]: pgmap v7766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:29 smithi171 ceph-mon[37789]: pgmap v7766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:30.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:30.186+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:31:30.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:30.186+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:31:30.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:30.186+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:31 smithi044 ceph-mon[32404]: pgmap v7767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13835.962427 secs 2024-07-26T12:31:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:31 smithi116 ceph-mon[37097]: pgmap v7767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13835.962427 secs 2024-07-26T12:31:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:31 smithi171 ceph-mon[37789]: pgmap v7767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13835.962427 secs 2024-07-26T12:31:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:33 smithi044 ceph-mon[32404]: pgmap v7768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:33 smithi116 ceph-mon[37097]: pgmap v7768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:33 smithi171 ceph-mon[37789]: pgmap v7768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:31:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:35 smithi044 ceph-mon[32404]: pgmap v7769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:35 smithi044 ceph-mon[32404]: pgmap v7770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:35 smithi116 ceph-mon[37097]: pgmap v7769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:35 smithi116 ceph-mon[37097]: pgmap v7770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:35 smithi171 ceph-mon[37789]: pgmap v7769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:35 smithi171 ceph-mon[37789]: pgmap v7770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13840.962590 secs 2024-07-26T12:31:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13840.962590 secs 2024-07-26T12:31:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13840.962590 secs 2024-07-26T12:31:37.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:37 smithi116 ceph-mon[37097]: pgmap v7771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:37 smithi044 ceph-mon[32404]: pgmap v7771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:37 smithi171 ceph-mon[37789]: pgmap v7771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:39 smithi044 ceph-mon[32404]: pgmap v7772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T12:31:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:39 smithi116 ceph-mon[37097]: pgmap v7772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T12:31:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:39 smithi171 ceph-mon[37789]: pgmap v7772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T12:31:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:41 smithi044 ceph-mon[32404]: pgmap v7773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13845.962736 secs 2024-07-26T12:31:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:41 smithi116 ceph-mon[37097]: pgmap v7773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13845.962736 secs 2024-07-26T12:31:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:41 smithi171 ceph-mon[37789]: pgmap v7773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13845.962736 secs 2024-07-26T12:31:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:43 smithi044 ceph-mon[32404]: pgmap v7774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:43 smithi116 ceph-mon[37097]: pgmap v7774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:43 smithi171 ceph-mon[37789]: pgmap v7774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:31:45.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:45.191+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:31:45.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:45.191+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:31:45.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:31:45.191+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:45 smithi044 ceph-mon[32404]: pgmap v7775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:45 smithi116 ceph-mon[37097]: pgmap v7775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:45 smithi171 ceph-mon[37789]: pgmap v7775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:31:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13850.962885 secs 2024-07-26T12:31:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13850.962885 secs 2024-07-26T12:31:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13850.962885 secs 2024-07-26T12:31:47.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:47 smithi171 ceph-mon[37789]: pgmap v7776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:47 smithi044 ceph-mon[32404]: pgmap v7776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:47 smithi116 ceph-mon[37097]: pgmap v7776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:49 smithi044 ceph-mon[32404]: pgmap v7777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:49 smithi116 ceph-mon[37097]: pgmap v7777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:49 smithi171 ceph-mon[37789]: pgmap v7777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:51 smithi044 ceph-mon[32404]: pgmap v7778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13855.963031 secs 2024-07-26T12:31:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:51 smithi116 ceph-mon[37097]: pgmap v7778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13855.963031 secs 2024-07-26T12:31:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:51 smithi171 ceph-mon[37789]: pgmap v7778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:31:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13855.963031 secs 2024-07-26T12:31:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:53 smithi044 ceph-mon[32404]: pgmap v7779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:53 smithi116 ceph-mon[37097]: pgmap v7779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:53 smithi171 ceph-mon[37789]: pgmap v7779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:55 smithi044 ceph-mon[32404]: pgmap v7780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:55 smithi116 ceph-mon[37097]: pgmap v7780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:55 smithi171 ceph-mon[37789]: pgmap v7780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13860.963193 secs 2024-07-26T12:31:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13860.963193 secs 2024-07-26T12:31:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13860.963193 secs 2024-07-26T12:31:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:57 smithi044 ceph-mon[32404]: pgmap v7781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:57 smithi116 ceph-mon[37097]: pgmap v7781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:57 smithi171 ceph-mon[37789]: pgmap v7781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:31:59 smithi044 ceph-mon[32404]: pgmap v7782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:31:59 smithi116 ceph-mon[37097]: pgmap v7782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:31:59 smithi171 ceph-mon[37789]: pgmap v7782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:00.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:00.192+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:32:00.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:00.192+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:00.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:00.192+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:32:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:01 smithi044 ceph-mon[32404]: pgmap v7783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13865.963320 secs 2024-07-26T12:32:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:01 smithi116 ceph-mon[37097]: pgmap v7783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13865.963320 secs 2024-07-26T12:32:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:01 smithi171 ceph-mon[37789]: pgmap v7783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13865.963320 secs 2024-07-26T12:32:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:03 smithi044 ceph-mon[32404]: pgmap v7784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:03 smithi116 ceph-mon[37097]: pgmap v7784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:03 smithi171 ceph-mon[37789]: pgmap v7784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:05.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:05 smithi044 ceph-mon[32404]: pgmap v7785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:05 smithi116 ceph-mon[37097]: pgmap v7785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:05 smithi171 ceph-mon[37789]: pgmap v7785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13870.963456 secs 2024-07-26T12:32:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13870.963456 secs 2024-07-26T12:32:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13870.963456 secs 2024-07-26T12:32:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:07 smithi171 ceph-mon[37789]: pgmap v7786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:07.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:07 smithi116 ceph-mon[37097]: pgmap v7786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:07 smithi044 ceph-mon[32404]: pgmap v7786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:09 smithi171 ceph-mon[37789]: pgmap v7787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:09 smithi044 ceph-mon[32404]: pgmap v7787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:09 smithi116 ceph-mon[37097]: pgmap v7787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:11 smithi171 ceph-mon[37789]: pgmap v7788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13875.963652 secs 2024-07-26T12:32:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:11 smithi044 ceph-mon[32404]: pgmap v7788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13875.963652 secs 2024-07-26T12:32:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:11 smithi116 ceph-mon[37097]: pgmap v7788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13875.963652 secs 2024-07-26T12:32:13.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:13 smithi171 ceph-mon[37789]: pgmap v7789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:13 smithi044 ceph-mon[32404]: pgmap v7789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:13 smithi116 ceph-mon[37097]: pgmap v7789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:15.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:15.192+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:32:15.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:15.192+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:15.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:15.192+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:32:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:15 smithi044 ceph-mon[32404]: pgmap v7790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:15 smithi171 ceph-mon[37789]: pgmap v7790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:15 smithi116 ceph-mon[37097]: pgmap v7790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13880.963778 secs 2024-07-26T12:32:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13880.963778 secs 2024-07-26T12:32:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13880.963778 secs 2024-07-26T12:32:17.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.180+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:32:17.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.180+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:32:17.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.181+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T12:32:17.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.181+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T12:32:17.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.181+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T12:32:17.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.181+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:32:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.182+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T12:32:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:17.183+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:32:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:17 smithi044 ceph-mon[32404]: pgmap v7791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:17 smithi171 ceph-mon[37789]: pgmap v7791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:17 smithi116 ceph-mon[37097]: pgmap v7791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:19 smithi171 ceph-mon[37789]: pgmap v7792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:19 smithi044 ceph-mon[32404]: pgmap v7792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:19 smithi116 ceph-mon[37097]: pgmap v7792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:21 smithi171 ceph-mon[37789]: pgmap v7793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13885.963918 secs 2024-07-26T12:32:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:21 smithi044 ceph-mon[32404]: pgmap v7793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13885.963918 secs 2024-07-26T12:32:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:21 smithi116 ceph-mon[37097]: pgmap v7793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13885.963918 secs 2024-07-26T12:32:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:23 smithi171 ceph-mon[37789]: pgmap v7794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:23 smithi044 ceph-mon[32404]: pgmap v7794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:23 smithi116 ceph-mon[37097]: pgmap v7794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:25 smithi171 ceph-mon[37789]: pgmap v7795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:25 smithi044 ceph-mon[32404]: pgmap v7795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:25 smithi116 ceph-mon[37097]: pgmap v7795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13890.964067 secs 2024-07-26T12:32:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13890.964067 secs 2024-07-26T12:32:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13890.964067 secs 2024-07-26T12:32:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:27 smithi044 ceph-mon[32404]: pgmap v7796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:27 smithi116 ceph-mon[37097]: pgmap v7796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:27 smithi171 ceph-mon[37789]: pgmap v7796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:29 smithi044 ceph-mon[32404]: pgmap v7797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:29 smithi116 ceph-mon[37097]: pgmap v7797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:29.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:29 smithi171 ceph-mon[37789]: pgmap v7797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:30.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:30.194+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:32:30.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:30.194+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:30.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:30.194+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:32:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:31 smithi044 ceph-mon[32404]: pgmap v7798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13895.964205 secs 2024-07-26T12:32:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:31 smithi116 ceph-mon[37097]: pgmap v7798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13895.964205 secs 2024-07-26T12:32:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:31 smithi171 ceph-mon[37789]: pgmap v7798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13895.964205 secs 2024-07-26T12:32:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:33 smithi044 ceph-mon[32404]: pgmap v7799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:33 smithi116 ceph-mon[37097]: pgmap v7799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:33 smithi171 ceph-mon[37789]: pgmap v7799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:34.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:35 smithi171 ceph-mon[37789]: pgmap v7800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:35 smithi044 ceph-mon[32404]: pgmap v7800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:35 smithi116 ceph-mon[37097]: pgmap v7800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13900.964337 secs 2024-07-26T12:32:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: pgmap v7801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13900.964337 secs 2024-07-26T12:32:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: pgmap v7801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13900.964337 secs 2024-07-26T12:32:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: pgmap v7801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:37.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:39.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:39 smithi044 ceph-mon[32404]: pgmap v7802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:40.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:39 smithi116 ceph-mon[37097]: pgmap v7802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:40.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:39 smithi171 ceph-mon[37789]: pgmap v7802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:40.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:32:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:40 smithi044 ceph-mon[32404]: pgmap v7803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:40 smithi044 ceph-mon[32404]: pgmap v7804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:40 smithi116 ceph-mon[37097]: pgmap v7803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:40 smithi116 ceph-mon[37097]: pgmap v7804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:40 smithi171 ceph-mon[37789]: pgmap v7803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:40 smithi171 ceph-mon[37789]: pgmap v7804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13905.964496 secs 2024-07-26T12:32:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13905.964496 secs 2024-07-26T12:32:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13905.964496 secs 2024-07-26T12:32:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:42 smithi044 ceph-mon[32404]: pgmap v7805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:32:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:42 smithi116 ceph-mon[37097]: pgmap v7805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:32:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:42 smithi171 ceph-mon[37789]: pgmap v7805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:32:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:44 smithi044 ceph-mon[32404]: pgmap v7806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:44 smithi116 ceph-mon[37097]: pgmap v7806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:45.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:44 smithi171 ceph-mon[37789]: pgmap v7806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:45.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:45.196+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:32:45.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:45.196+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:45.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:32:45.196+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:32:46.922 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:46 smithi044 ceph-mon[32404]: pgmap v7807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:46.922 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13910.964664 secs 2024-07-26T12:32:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:46 smithi116 ceph-mon[37097]: pgmap v7807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13910.964664 secs 2024-07-26T12:32:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:46 smithi171 ceph-mon[37789]: pgmap v7807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:32:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13910.964664 secs 2024-07-26T12:32:47.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:47 smithi171 ceph-mon[37789]: pgmap v7808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:47 smithi044 ceph-mon[32404]: pgmap v7808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:47 smithi116 ceph-mon[37097]: pgmap v7808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:50 smithi171 ceph-mon[37789]: pgmap v7809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:50 smithi044 ceph-mon[32404]: pgmap v7809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:50 smithi116 ceph-mon[37097]: pgmap v7809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13915.964831 secs 2024-07-26T12:32:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13915.964831 secs 2024-07-26T12:32:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13915.964831 secs 2024-07-26T12:32:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:52 smithi171 ceph-mon[37789]: pgmap v7810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:52.642 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:52 smithi044 ceph-mon[32404]: pgmap v7810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:52 smithi116 ceph-mon[37097]: pgmap v7810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:54 smithi171 ceph-mon[37789]: pgmap v7811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:54 smithi044 ceph-mon[32404]: pgmap v7811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:54 smithi116 ceph-mon[37097]: pgmap v7811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:56 smithi044 ceph-mon[32404]: pgmap v7812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13920.964930 secs 2024-07-26T12:32:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:56 smithi116 ceph-mon[37097]: pgmap v7812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13920.964930 secs 2024-07-26T12:32:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:56 smithi171 ceph-mon[37789]: pgmap v7812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13920.964930 secs 2024-07-26T12:32:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:32:58 smithi044 ceph-mon[32404]: pgmap v7813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:32:58 smithi116 ceph-mon[37097]: pgmap v7813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:32:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:32:58 smithi171 ceph-mon[37789]: pgmap v7813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:00.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:00.197+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:33:00.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:00.197+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:00.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:00.197+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:33:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:00 smithi044 ceph-mon[32404]: pgmap v7814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:00 smithi116 ceph-mon[37097]: pgmap v7814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:00 smithi171 ceph-mon[37789]: pgmap v7814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13925.965105 secs 2024-07-26T12:33:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13925.965105 secs 2024-07-26T12:33:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13925.965105 secs 2024-07-26T12:33:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:02 smithi044 ceph-mon[32404]: pgmap v7815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:02 smithi116 ceph-mon[37097]: pgmap v7815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:02 smithi171 ceph-mon[37789]: pgmap v7815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:04 smithi044 ceph-mon[32404]: pgmap v7816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:04 smithi116 ceph-mon[37097]: pgmap v7816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:04 smithi171 ceph-mon[37789]: pgmap v7816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:06 smithi044 ceph-mon[32404]: pgmap v7817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13930.965256 secs 2024-07-26T12:33:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:06 smithi116 ceph-mon[37097]: pgmap v7817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13930.965256 secs 2024-07-26T12:33:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:06 smithi171 ceph-mon[37789]: pgmap v7817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13930.965256 secs 2024-07-26T12:33:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:08 smithi044 ceph-mon[32404]: pgmap v7818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:08 smithi116 ceph-mon[37097]: pgmap v7818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:08.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:08 smithi171 ceph-mon[37789]: pgmap v7818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:10 smithi044 ceph-mon[32404]: pgmap v7819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:10 smithi116 ceph-mon[37097]: pgmap v7819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:10 smithi171 ceph-mon[37789]: pgmap v7819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13935.965424 secs 2024-07-26T12:33:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13935.965424 secs 2024-07-26T12:33:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13935.965424 secs 2024-07-26T12:33:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:12 smithi044 ceph-mon[32404]: pgmap v7820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:12 smithi116 ceph-mon[37097]: pgmap v7820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:12 smithi171 ceph-mon[37789]: pgmap v7820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:14 smithi044 ceph-mon[32404]: pgmap v7821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:14 smithi116 ceph-mon[37097]: pgmap v7821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:14 smithi171 ceph-mon[37789]: pgmap v7821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:15.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:15.202+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:33:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:15.202+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:33:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:15.202+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:16 smithi044 ceph-mon[32404]: pgmap v7822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13940.965572 secs 2024-07-26T12:33:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:16 smithi116 ceph-mon[37097]: pgmap v7822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13940.965572 secs 2024-07-26T12:33:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:16 smithi171 ceph-mon[37789]: pgmap v7822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13940.965572 secs 2024-07-26T12:33:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:18 smithi044 ceph-mon[32404]: pgmap v7823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:18 smithi116 ceph-mon[37097]: pgmap v7823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:18 smithi171 ceph-mon[37789]: pgmap v7823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:20 smithi044 ceph-mon[32404]: pgmap v7824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:20 smithi116 ceph-mon[37097]: pgmap v7824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:20.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:20 smithi171 ceph-mon[37789]: pgmap v7824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13945.965736 secs 2024-07-26T12:33:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13945.965736 secs 2024-07-26T12:33:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13945.965736 secs 2024-07-26T12:33:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:22 smithi044 ceph-mon[32404]: pgmap v7825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:22 smithi116 ceph-mon[37097]: pgmap v7825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:22 smithi171 ceph-mon[37789]: pgmap v7825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:24 smithi044 ceph-mon[32404]: pgmap v7826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:24 smithi116 ceph-mon[37097]: pgmap v7826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:24 smithi171 ceph-mon[37789]: pgmap v7826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:26 smithi044 ceph-mon[32404]: pgmap v7827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13950.965875 secs 2024-07-26T12:33:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:26 smithi116 ceph-mon[37097]: pgmap v7827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13950.965875 secs 2024-07-26T12:33:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:26 smithi171 ceph-mon[37789]: pgmap v7827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13950.965875 secs 2024-07-26T12:33:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:28 smithi044 ceph-mon[32404]: pgmap v7828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:28 smithi116 ceph-mon[37097]: pgmap v7828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:28 smithi171 ceph-mon[37789]: pgmap v7828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:30.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:30.203+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:33:30.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:30.203+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:33:30.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:30.203+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:30 smithi044 ceph-mon[32404]: pgmap v7829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:30 smithi116 ceph-mon[37097]: pgmap v7829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:30 smithi171 ceph-mon[37789]: pgmap v7829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13955.966013 secs 2024-07-26T12:33:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13955.966013 secs 2024-07-26T12:33:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13955.966013 secs 2024-07-26T12:33:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:32 smithi044 ceph-mon[32404]: pgmap v7830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:32 smithi116 ceph-mon[37097]: pgmap v7830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:32 smithi171 ceph-mon[37789]: pgmap v7830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:34 smithi044 ceph-mon[32404]: pgmap v7831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:34 smithi116 ceph-mon[37097]: pgmap v7831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:34 smithi171 ceph-mon[37789]: pgmap v7831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:36 smithi044 ceph-mon[32404]: pgmap v7832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13960.966158 secs 2024-07-26T12:33:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:36 smithi116 ceph-mon[37097]: pgmap v7832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13960.966158 secs 2024-07-26T12:33:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:36 smithi171 ceph-mon[37789]: pgmap v7832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13960.966158 secs 2024-07-26T12:33:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:38 smithi044 ceph-mon[32404]: pgmap v7833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:38 smithi116 ceph-mon[37097]: pgmap v7833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:38 smithi171 ceph-mon[37789]: pgmap v7833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:40 smithi044 ceph-mon[32404]: pgmap v7834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:40 smithi116 ceph-mon[37097]: pgmap v7834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:40 smithi171 ceph-mon[37789]: pgmap v7834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:41 smithi044 ceph-mon[32404]: pgmap v7835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:33:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:41 smithi044 ceph-mon[32404]: pgmap v7836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:33:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13965.966280 secs 2024-07-26T12:33:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:41 smithi116 ceph-mon[37097]: pgmap v7835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:33:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:41 smithi116 ceph-mon[37097]: pgmap v7836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:33:41.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13965.966280 secs 2024-07-26T12:33:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:41 smithi171 ceph-mon[37789]: pgmap v7835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:33:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:41 smithi171 ceph-mon[37789]: pgmap v7836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:33:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13965.966280 secs 2024-07-26T12:33:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:43 smithi044 ceph-mon[32404]: pgmap v7837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:33:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:43 smithi116 ceph-mon[37097]: pgmap v7837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:33:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:43 smithi171 ceph-mon[37789]: pgmap v7837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:33:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:45.208+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:33:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:45.209+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:33:45.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:33:45.209+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:45 smithi044 ceph-mon[32404]: pgmap v7838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:45 smithi116 ceph-mon[37097]: pgmap v7838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:45 smithi171 ceph-mon[37789]: pgmap v7838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13970.966433 secs 2024-07-26T12:33:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13970.966433 secs 2024-07-26T12:33:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13970.966433 secs 2024-07-26T12:33:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:47 smithi044 ceph-mon[32404]: pgmap v7839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:33:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:47 smithi116 ceph-mon[37097]: pgmap v7839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:33:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:47 smithi171 ceph-mon[37789]: pgmap v7839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:33:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:49 smithi044 ceph-mon[32404]: pgmap v7840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:49 smithi116 ceph-mon[37097]: pgmap v7840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:49 smithi171 ceph-mon[37789]: pgmap v7840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:33:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:51 smithi044 ceph-mon[32404]: pgmap v7841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13975.966573 secs 2024-07-26T12:33:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:51 smithi116 ceph-mon[37097]: pgmap v7841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13975.966573 secs 2024-07-26T12:33:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:51 smithi171 ceph-mon[37789]: pgmap v7841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13975.966573 secs 2024-07-26T12:33:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:53 smithi044 ceph-mon[32404]: pgmap v7842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:53 smithi116 ceph-mon[37097]: pgmap v7842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:53 smithi171 ceph-mon[37789]: pgmap v7842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:33:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:55 smithi044 ceph-mon[32404]: pgmap v7843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:55 smithi116 ceph-mon[37097]: pgmap v7843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:55 smithi171 ceph-mon[37789]: pgmap v7843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:33:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13980.966731 secs 2024-07-26T12:33:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13980.966731 secs 2024-07-26T12:33:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13980.966731 secs 2024-07-26T12:33:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:57 smithi044 ceph-mon[32404]: pgmap v7844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:57 smithi116 ceph-mon[37097]: pgmap v7844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:57 smithi171 ceph-mon[37789]: pgmap v7844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:33:58 smithi044 ceph-mon[32404]: pgmap v7845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:33:58 smithi116 ceph-mon[37097]: pgmap v7845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:33:58 smithi171 ceph-mon[37789]: pgmap v7845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:34:00.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:00.210+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:34:00.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:00.210+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:00.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:00.210+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:34:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:01 smithi044 ceph-mon[32404]: pgmap v7846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13985.966840 secs 2024-07-26T12:34:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:01 smithi116 ceph-mon[37097]: pgmap v7846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13985.966840 secs 2024-07-26T12:34:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:01 smithi171 ceph-mon[37789]: pgmap v7846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13985.966840 secs 2024-07-26T12:34:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:03 smithi044 ceph-mon[32404]: pgmap v7847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:34:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:03 smithi116 ceph-mon[37097]: pgmap v7847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:34:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:03 smithi171 ceph-mon[37789]: pgmap v7847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:34:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:05 smithi044 ceph-mon[32404]: pgmap v7848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:05 smithi116 ceph-mon[37097]: pgmap v7848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:05 smithi171 ceph-mon[37789]: pgmap v7848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13990.967003 secs 2024-07-26T12:34:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13990.967003 secs 2024-07-26T12:34:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13990.967003 secs 2024-07-26T12:34:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:07 smithi044 ceph-mon[32404]: pgmap v7849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:34:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:07 smithi116 ceph-mon[37097]: pgmap v7849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:34:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:07 smithi171 ceph-mon[37789]: pgmap v7849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:34:09.954 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:09 smithi044 ceph-mon[32404]: pgmap v7850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:34:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:09 smithi116 ceph-mon[37097]: pgmap v7850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:34:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:09 smithi171 ceph-mon[37789]: pgmap v7850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:34:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:11 smithi044 ceph-mon[32404]: pgmap v7851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 13995.967171 secs 2024-07-26T12:34:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:11 smithi116 ceph-mon[37097]: pgmap v7851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 13995.967171 secs 2024-07-26T12:34:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:11 smithi171 ceph-mon[37789]: pgmap v7851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 13995.967171 secs 2024-07-26T12:34:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:13 smithi044 ceph-mon[32404]: pgmap v7852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:13 smithi116 ceph-mon[37097]: pgmap v7852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:13 smithi171 ceph-mon[37789]: pgmap v7852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:14 smithi044 ceph-mon[32404]: pgmap v7853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:14 smithi116 ceph-mon[37097]: pgmap v7853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:14 smithi171 ceph-mon[37789]: pgmap v7853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:15.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:15.215+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:34:15.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:15.215+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:34:15.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:15.215+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14000.967325 secs 2024-07-26T12:34:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14000.967325 secs 2024-07-26T12:34:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14000.967325 secs 2024-07-26T12:34:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:17 smithi044 ceph-mon[32404]: pgmap v7854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:17 smithi116 ceph-mon[37097]: pgmap v7854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:17 smithi171 ceph-mon[37789]: pgmap v7854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:19 smithi044 ceph-mon[32404]: pgmap v7855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:19 smithi116 ceph-mon[37097]: pgmap v7855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:19 smithi171 ceph-mon[37789]: pgmap v7855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:21 smithi044 ceph-mon[32404]: pgmap v7856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14005.967495 secs 2024-07-26T12:34:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:21 smithi116 ceph-mon[37097]: pgmap v7856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14005.967495 secs 2024-07-26T12:34:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:21 smithi171 ceph-mon[37789]: pgmap v7856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14005.967495 secs 2024-07-26T12:34:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:23 smithi044 ceph-mon[32404]: pgmap v7857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:23 smithi116 ceph-mon[37097]: pgmap v7857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:23 smithi171 ceph-mon[37789]: pgmap v7857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:25 smithi044 ceph-mon[32404]: pgmap v7858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:25 smithi116 ceph-mon[37097]: pgmap v7858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:25 smithi171 ceph-mon[37789]: pgmap v7858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14010.967659 secs 2024-07-26T12:34:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14010.967659 secs 2024-07-26T12:34:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14010.967659 secs 2024-07-26T12:34:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:27 smithi044 ceph-mon[32404]: pgmap v7859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:27 smithi116 ceph-mon[37097]: pgmap v7859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:27 smithi171 ceph-mon[37789]: pgmap v7859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:29 smithi044 ceph-mon[32404]: pgmap v7860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:29 smithi116 ceph-mon[37097]: pgmap v7860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:29 smithi171 ceph-mon[37789]: pgmap v7860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:30.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:30.217+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:34:30.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:30.217+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:34:30.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:30.217+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:30 smithi044 ceph-mon[32404]: pgmap v7861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:30 smithi116 ceph-mon[37097]: pgmap v7861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:30 smithi171 ceph-mon[37789]: pgmap v7861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14015.967801 secs 2024-07-26T12:34:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14015.967801 secs 2024-07-26T12:34:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14015.967801 secs 2024-07-26T12:34:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:32 smithi044 ceph-mon[32404]: pgmap v7862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:32 smithi116 ceph-mon[37097]: pgmap v7862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:32 smithi171 ceph-mon[37789]: pgmap v7862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:35 smithi044 ceph-mon[32404]: pgmap v7863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:35 smithi116 ceph-mon[37097]: pgmap v7863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:35 smithi171 ceph-mon[37789]: pgmap v7863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:34:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14020.967960 secs 2024-07-26T12:34:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14020.967960 secs 2024-07-26T12:34:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14020.967960 secs 2024-07-26T12:34:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:37 smithi044 ceph-mon[32404]: pgmap v7864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:37 smithi116 ceph-mon[37097]: pgmap v7864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:37 smithi171 ceph-mon[37789]: pgmap v7864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:39 smithi044 ceph-mon[32404]: pgmap v7865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:39 smithi116 ceph-mon[37097]: pgmap v7865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:39 smithi171 ceph-mon[37789]: pgmap v7865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:41 smithi044 ceph-mon[32404]: pgmap v7866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14025.968127 secs 2024-07-26T12:34:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:41 smithi116 ceph-mon[37097]: pgmap v7866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14025.968127 secs 2024-07-26T12:34:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:41 smithi171 ceph-mon[37789]: pgmap v7866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14025.968127 secs 2024-07-26T12:34:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:34:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:34:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:34:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:43 smithi044 ceph-mon[32404]: pgmap v7867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:34:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:43 smithi044 ceph-mon[32404]: pgmap v7868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:43 smithi116 ceph-mon[37097]: pgmap v7867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:34:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:43 smithi116 ceph-mon[37097]: pgmap v7868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:43 smithi171 ceph-mon[37789]: pgmap v7867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:34:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:43 smithi171 ceph-mon[37789]: pgmap v7868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:44 smithi044 ceph-mon[32404]: pgmap v7869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:44 smithi116 ceph-mon[37097]: pgmap v7869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:45.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:44 smithi171 ceph-mon[37789]: pgmap v7869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:45.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:45.222+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:34:45.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:45.222+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:45.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:34:45.222+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:34:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14030.968266 secs 2024-07-26T12:34:47.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:46 smithi044 ceph-mon[32404]: pgmap v7870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:34:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14030.968266 secs 2024-07-26T12:34:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:46 smithi116 ceph-mon[37097]: pgmap v7870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:34:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14030.968266 secs 2024-07-26T12:34:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:46 smithi171 ceph-mon[37789]: pgmap v7870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:34:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:48 smithi044 ceph-mon[32404]: pgmap v7871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:48 smithi116 ceph-mon[37097]: pgmap v7871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:48 smithi171 ceph-mon[37789]: pgmap v7871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:51.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:50 smithi171 ceph-mon[37789]: pgmap v7872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:50 smithi044 ceph-mon[32404]: pgmap v7872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:50 smithi116 ceph-mon[37097]: pgmap v7872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:34:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14035.968429 secs 2024-07-26T12:34:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14035.968429 secs 2024-07-26T12:34:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14035.968429 secs 2024-07-26T12:34:53.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:52 smithi044 ceph-mon[32404]: pgmap v7873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:52 smithi116 ceph-mon[37097]: pgmap v7873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:52 smithi171 ceph-mon[37789]: pgmap v7873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:54 smithi044 ceph-mon[32404]: pgmap v7874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:54 smithi116 ceph-mon[37097]: pgmap v7874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:54 smithi171 ceph-mon[37789]: pgmap v7874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14040.968600 secs 2024-07-26T12:34:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:56 smithi044 ceph-mon[32404]: pgmap v7875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14040.968600 secs 2024-07-26T12:34:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:56 smithi116 ceph-mon[37097]: pgmap v7875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14040.968600 secs 2024-07-26T12:34:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:56 smithi171 ceph-mon[37789]: pgmap v7875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:34:58 smithi044 ceph-mon[32404]: pgmap v7876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:34:58 smithi116 ceph-mon[37097]: pgmap v7876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:34:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:34:58 smithi171 ceph-mon[37789]: pgmap v7876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:00.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:00.223+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:35:00.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:00.224+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:35:00.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:00.224+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:00 smithi044 ceph-mon[32404]: pgmap v7877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:00 smithi116 ceph-mon[37097]: pgmap v7877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:00 smithi171 ceph-mon[37789]: pgmap v7877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14045.968744 secs 2024-07-26T12:35:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14045.968744 secs 2024-07-26T12:35:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14045.968744 secs 2024-07-26T12:35:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:02 smithi044 ceph-mon[32404]: pgmap v7878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:02 smithi116 ceph-mon[37097]: pgmap v7878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:02 smithi171 ceph-mon[37789]: pgmap v7878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:04 smithi044 ceph-mon[32404]: pgmap v7879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:04 smithi116 ceph-mon[37097]: pgmap v7879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:04 smithi171 ceph-mon[37789]: pgmap v7879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14050.968884 secs 2024-07-26T12:35:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:06 smithi044 ceph-mon[32404]: pgmap v7880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14050.968884 secs 2024-07-26T12:35:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:06 smithi116 ceph-mon[37097]: pgmap v7880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14050.968884 secs 2024-07-26T12:35:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:06 smithi171 ceph-mon[37789]: pgmap v7880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:08 smithi044 ceph-mon[32404]: pgmap v7881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:08 smithi116 ceph-mon[37097]: pgmap v7881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:09.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:08 smithi171 ceph-mon[37789]: pgmap v7881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:10 smithi044 ceph-mon[32404]: pgmap v7882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:10 smithi116 ceph-mon[37097]: pgmap v7882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:10 smithi171 ceph-mon[37789]: pgmap v7882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14055.969031 secs 2024-07-26T12:35:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14055.969031 secs 2024-07-26T12:35:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14055.969031 secs 2024-07-26T12:35:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:12 smithi044 ceph-mon[32404]: pgmap v7883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:12 smithi116 ceph-mon[37097]: pgmap v7883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:12 smithi171 ceph-mon[37789]: pgmap v7883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:15.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:15.228+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:35:15.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:15.228+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:15.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:15.228+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:35:15.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:14 smithi044 ceph-mon[32404]: pgmap v7884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:14 smithi116 ceph-mon[37097]: pgmap v7884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:14 smithi171 ceph-mon[37789]: pgmap v7884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:17.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:17.213+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T12:35:17.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:17.213+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:35:17.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:17.213+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T12:35:17.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:17.213+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:35:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14060.969124 secs 2024-07-26T12:35:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:16 smithi044 ceph-mon[32404]: pgmap v7885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14060.969124 secs 2024-07-26T12:35:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:16 smithi116 ceph-mon[37097]: pgmap v7885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14060.969124 secs 2024-07-26T12:35:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:16 smithi171 ceph-mon[37789]: pgmap v7885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:18 smithi044 ceph-mon[32404]: pgmap v7886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:18 smithi116 ceph-mon[37097]: pgmap v7886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:18 smithi171 ceph-mon[37789]: pgmap v7886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:20 smithi044 ceph-mon[32404]: pgmap v7887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:20 smithi116 ceph-mon[37097]: pgmap v7887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:20 smithi171 ceph-mon[37789]: pgmap v7887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14065.969258 secs 2024-07-26T12:35:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14065.969258 secs 2024-07-26T12:35:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14065.969258 secs 2024-07-26T12:35:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:23 smithi044 ceph-mon[32404]: pgmap v7888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:23 smithi116 ceph-mon[37097]: pgmap v7888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:23 smithi171 ceph-mon[37789]: pgmap v7888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:25 smithi044 ceph-mon[32404]: pgmap v7889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:25 smithi116 ceph-mon[37097]: pgmap v7889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:25 smithi171 ceph-mon[37789]: pgmap v7889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14070.969389 secs 2024-07-26T12:35:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:27 smithi116 ceph-mon[37097]: pgmap v7890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:35:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14070.969389 secs 2024-07-26T12:35:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:27 smithi171 ceph-mon[37789]: pgmap v7890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:35:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14070.969389 secs 2024-07-26T12:35:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:27 smithi044 ceph-mon[32404]: pgmap v7890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:35:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:29 smithi044 ceph-mon[32404]: pgmap v7891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:29 smithi171 ceph-mon[37789]: pgmap v7891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:29 smithi116 ceph-mon[37097]: pgmap v7891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:30.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:30.230+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:35:30.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:30.230+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:30.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:30.230+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:35:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:31 smithi171 ceph-mon[37789]: pgmap v7892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:31 smithi044 ceph-mon[32404]: pgmap v7892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:31 smithi116 ceph-mon[37097]: pgmap v7892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14075.969533 secs 2024-07-26T12:35:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14075.969533 secs 2024-07-26T12:35:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14075.969533 secs 2024-07-26T12:35:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:33 smithi171 ceph-mon[37789]: pgmap v7893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:33 smithi044 ceph-mon[32404]: pgmap v7893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:33 smithi116 ceph-mon[37097]: pgmap v7893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:35 smithi171 ceph-mon[37789]: pgmap v7894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:35 smithi044 ceph-mon[32404]: pgmap v7894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:35 smithi116 ceph-mon[37097]: pgmap v7894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14080.969692 secs 2024-07-26T12:35:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:37 smithi171 ceph-mon[37789]: pgmap v7895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14080.969692 secs 2024-07-26T12:35:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:37 smithi044 ceph-mon[32404]: pgmap v7895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14080.969692 secs 2024-07-26T12:35:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:37 smithi116 ceph-mon[37097]: pgmap v7895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:39.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:39 smithi116 ceph-mon[37097]: pgmap v7896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:39 smithi171 ceph-mon[37789]: pgmap v7896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:39 smithi044 ceph-mon[32404]: pgmap v7896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:41 smithi171 ceph-mon[37789]: pgmap v7897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:41 smithi044 ceph-mon[32404]: pgmap v7897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:41 smithi116 ceph-mon[37097]: pgmap v7897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:35:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14085.969853 secs 2024-07-26T12:35:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14085.969853 secs 2024-07-26T12:35:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14085.969853 secs 2024-07-26T12:35:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:43 smithi171 ceph-mon[37789]: pgmap v7898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:43 smithi044 ceph-mon[32404]: pgmap v7898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:43 smithi116 ceph-mon[37097]: pgmap v7898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:44 smithi044 ceph-mon[32404]: pgmap v7899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:35:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:44 smithi044 ceph-mon[32404]: pgmap v7900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:44.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:44 smithi116 ceph-mon[37097]: pgmap v7899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:35:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:44 smithi116 ceph-mon[37097]: pgmap v7900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:44.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:44 smithi171 ceph-mon[37789]: pgmap v7899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:35:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:44 smithi171 ceph-mon[37789]: pgmap v7900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:45.236+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:35:45.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:45.236+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:35:45.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:35:45.236+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:46 smithi044 ceph-mon[32404]: pgmap v7901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14090.970015 secs 2024-07-26T12:35:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:46 smithi116 ceph-mon[37097]: pgmap v7901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14090.970015 secs 2024-07-26T12:35:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:46 smithi171 ceph-mon[37789]: pgmap v7901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14090.970015 secs 2024-07-26T12:35:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:48 smithi044 ceph-mon[32404]: pgmap v7902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:48 smithi116 ceph-mon[37097]: pgmap v7902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:48 smithi171 ceph-mon[37789]: pgmap v7902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:50 smithi044 ceph-mon[32404]: pgmap v7903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:50 smithi116 ceph-mon[37097]: pgmap v7903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:50 smithi171 ceph-mon[37789]: pgmap v7903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:35:51.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14095.970177 secs 2024-07-26T12:35:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14095.970177 secs 2024-07-26T12:35:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14095.970177 secs 2024-07-26T12:35:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:52 smithi044 ceph-mon[32404]: pgmap v7904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:52 smithi116 ceph-mon[37097]: pgmap v7904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:52 smithi171 ceph-mon[37789]: pgmap v7904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:35:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:54 smithi044 ceph-mon[32404]: pgmap v7905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:54 smithi116 ceph-mon[37097]: pgmap v7905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:54 smithi171 ceph-mon[37789]: pgmap v7905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:56 smithi044 ceph-mon[32404]: pgmap v7906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14100.970315 secs 2024-07-26T12:35:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:56 smithi116 ceph-mon[37097]: pgmap v7906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14100.970315 secs 2024-07-26T12:35:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:56 smithi171 ceph-mon[37789]: pgmap v7906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14100.970315 secs 2024-07-26T12:35:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:35:58 smithi044 ceph-mon[32404]: pgmap v7907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:35:58 smithi116 ceph-mon[37097]: pgmap v7907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:35:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:35:58 smithi171 ceph-mon[37789]: pgmap v7907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:00.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:00.237+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:36:00.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:00.237+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:00.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:00.237+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:36:00.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:00 smithi044 ceph-mon[32404]: pgmap v7908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:00 smithi116 ceph-mon[37097]: pgmap v7908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:00 smithi171 ceph-mon[37789]: pgmap v7908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14105.970465 secs 2024-07-26T12:36:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14105.970465 secs 2024-07-26T12:36:01.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14105.970465 secs 2024-07-26T12:36:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:02 smithi044 ceph-mon[32404]: pgmap v7909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:02 smithi116 ceph-mon[37097]: pgmap v7909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:02 smithi171 ceph-mon[37789]: pgmap v7909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:04.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:04 smithi044 ceph-mon[32404]: pgmap v7910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:04 smithi116 ceph-mon[37097]: pgmap v7910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:04 smithi171 ceph-mon[37789]: pgmap v7910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:06 smithi044 ceph-mon[32404]: pgmap v7911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14110.970668 secs 2024-07-26T12:36:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:06 smithi116 ceph-mon[37097]: pgmap v7911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14110.970668 secs 2024-07-26T12:36:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:06 smithi171 ceph-mon[37789]: pgmap v7911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14110.970668 secs 2024-07-26T12:36:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:08 smithi044 ceph-mon[32404]: pgmap v7912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:08 smithi116 ceph-mon[37097]: pgmap v7912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:08 smithi171 ceph-mon[37789]: pgmap v7912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:10.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:10 smithi044 ceph-mon[32404]: pgmap v7913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:10 smithi116 ceph-mon[37097]: pgmap v7913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:10 smithi171 ceph-mon[37789]: pgmap v7913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14115.970802 secs 2024-07-26T12:36:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14115.970802 secs 2024-07-26T12:36:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14115.970802 secs 2024-07-26T12:36:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:12 smithi044 ceph-mon[32404]: pgmap v7914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:12 smithi116 ceph-mon[37097]: pgmap v7914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:12 smithi171 ceph-mon[37789]: pgmap v7914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:14 smithi044 ceph-mon[32404]: pgmap v7915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:14 smithi116 ceph-mon[37097]: pgmap v7915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:14 smithi171 ceph-mon[37789]: pgmap v7915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:15.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:15.239+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:36:15.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:15.239+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:36:15.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:15.239+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:16 smithi044 ceph-mon[32404]: pgmap v7916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14120.970963 secs 2024-07-26T12:36:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:16 smithi116 ceph-mon[37097]: pgmap v7916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14120.970963 secs 2024-07-26T12:36:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:16 smithi171 ceph-mon[37789]: pgmap v7916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14120.970963 secs 2024-07-26T12:36:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:18 smithi044 ceph-mon[32404]: pgmap v7917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:18 smithi116 ceph-mon[37097]: pgmap v7917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:18 smithi171 ceph-mon[37789]: pgmap v7917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:20 smithi044 ceph-mon[32404]: pgmap v7918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:20 smithi116 ceph-mon[37097]: pgmap v7918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:20 smithi171 ceph-mon[37789]: pgmap v7918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14125.971130 secs 2024-07-26T12:36:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14125.971130 secs 2024-07-26T12:36:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14125.971130 secs 2024-07-26T12:36:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:22 smithi044 ceph-mon[32404]: pgmap v7919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:22 smithi116 ceph-mon[37097]: pgmap v7919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:22 smithi171 ceph-mon[37789]: pgmap v7919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:24.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:24 smithi044 ceph-mon[32404]: pgmap v7920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:24 smithi116 ceph-mon[37097]: pgmap v7920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:24 smithi171 ceph-mon[37789]: pgmap v7920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:26 smithi116 ceph-mon[37097]: pgmap v7921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:26.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14130.971214 secs 2024-07-26T12:36:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:26 smithi171 ceph-mon[37789]: pgmap v7921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14130.971214 secs 2024-07-26T12:36:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:26 smithi044 ceph-mon[32404]: pgmap v7921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:26.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14130.971214 secs 2024-07-26T12:36:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:28 smithi171 ceph-mon[37789]: pgmap v7922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:28 smithi044 ceph-mon[32404]: pgmap v7922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:28 smithi116 ceph-mon[37097]: pgmap v7922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:30.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:30.240+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:36:30.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:30.240+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:36:30.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:30.240+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:30 smithi044 ceph-mon[32404]: pgmap v7923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:30 smithi171 ceph-mon[37789]: pgmap v7923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:30 smithi116 ceph-mon[37097]: pgmap v7923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14135.971358 secs 2024-07-26T12:36:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14135.971358 secs 2024-07-26T12:36:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14135.971358 secs 2024-07-26T12:36:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:32 smithi171 ceph-mon[37789]: pgmap v7924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:32 smithi044 ceph-mon[32404]: pgmap v7924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:32 smithi116 ceph-mon[37097]: pgmap v7924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:34 smithi171 ceph-mon[37789]: pgmap v7925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:34 smithi044 ceph-mon[32404]: pgmap v7925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:34 smithi116 ceph-mon[37097]: pgmap v7925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:36 smithi171 ceph-mon[37789]: pgmap v7926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14140.971451 secs 2024-07-26T12:36:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:36 smithi044 ceph-mon[32404]: pgmap v7926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14140.971451 secs 2024-07-26T12:36:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:36 smithi116 ceph-mon[37097]: pgmap v7926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14140.971451 secs 2024-07-26T12:36:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:38 smithi171 ceph-mon[37789]: pgmap v7927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:38 smithi044 ceph-mon[32404]: pgmap v7927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:38 smithi116 ceph-mon[37097]: pgmap v7927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:40.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:40 smithi116 ceph-mon[37097]: pgmap v7928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:40 smithi171 ceph-mon[37789]: pgmap v7928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:40.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:40 smithi044 ceph-mon[32404]: pgmap v7928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:36:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14145.971610 secs 2024-07-26T12:36:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14145.971610 secs 2024-07-26T12:36:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14145.971610 secs 2024-07-26T12:36:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:42 smithi171 ceph-mon[37789]: pgmap v7929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:42 smithi044 ceph-mon[32404]: pgmap v7929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:42 smithi116 ceph-mon[37097]: pgmap v7929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:44 smithi044 ceph-mon[32404]: pgmap v7930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:44 smithi116 ceph-mon[37097]: pgmap v7930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:44 smithi171 ceph-mon[37789]: pgmap v7930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:45.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:45.245+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:36:45.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:45.245+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:36:45.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:36:45.245+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:45 smithi044 ceph-mon[32404]: pgmap v7931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:36:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:45 smithi044 ceph-mon[32404]: pgmap v7932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:45.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:45 smithi116 ceph-mon[37097]: pgmap v7931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:36:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:45 smithi116 ceph-mon[37097]: pgmap v7932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:45.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:45 smithi171 ceph-mon[37789]: pgmap v7931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:36:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:45 smithi171 ceph-mon[37789]: pgmap v7932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14150.971746 secs 2024-07-26T12:36:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14150.971746 secs 2024-07-26T12:36:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14150.971746 secs 2024-07-26T12:36:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:47 smithi044 ceph-mon[32404]: pgmap v7933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:47 smithi116 ceph-mon[37097]: pgmap v7933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:47 smithi171 ceph-mon[37789]: pgmap v7933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:49.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:49 smithi044 ceph-mon[32404]: pgmap v7934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:49 smithi116 ceph-mon[37097]: pgmap v7934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:49 smithi171 ceph-mon[37789]: pgmap v7934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:51 smithi044 ceph-mon[32404]: pgmap v7935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14155.971850 secs 2024-07-26T12:36:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:51 smithi116 ceph-mon[37097]: pgmap v7935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14155.971850 secs 2024-07-26T12:36:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:51 smithi171 ceph-mon[37789]: pgmap v7935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14155.971850 secs 2024-07-26T12:36:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:53 smithi044 ceph-mon[32404]: pgmap v7936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:53.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:53 smithi116 ceph-mon[37097]: pgmap v7936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:53 smithi171 ceph-mon[37789]: pgmap v7936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:55 smithi044 ceph-mon[32404]: pgmap v7937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:55 smithi116 ceph-mon[37097]: pgmap v7937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:55 smithi171 ceph-mon[37789]: pgmap v7937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14160.971999 secs 2024-07-26T12:36:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14160.971999 secs 2024-07-26T12:36:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14160.971999 secs 2024-07-26T12:36:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:57 smithi044 ceph-mon[32404]: pgmap v7938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:57 smithi116 ceph-mon[37097]: pgmap v7938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:57 smithi171 ceph-mon[37789]: pgmap v7938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:36:59 smithi044 ceph-mon[32404]: pgmap v7939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:36:59 smithi116 ceph-mon[37097]: pgmap v7939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:36:59 smithi171 ceph-mon[37789]: pgmap v7939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:00.247+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:37:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:00.247+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:37:00.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:00.247+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:01 smithi044 ceph-mon[32404]: pgmap v7940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14165.972164 secs 2024-07-26T12:37:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:01 smithi116 ceph-mon[37097]: pgmap v7940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14165.972164 secs 2024-07-26T12:37:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:01 smithi171 ceph-mon[37789]: pgmap v7940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14165.972164 secs 2024-07-26T12:37:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:03 smithi044 ceph-mon[32404]: pgmap v7941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:03 smithi116 ceph-mon[37097]: pgmap v7941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:03 smithi171 ceph-mon[37789]: pgmap v7941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:05 smithi116 ceph-mon[37097]: pgmap v7942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:05.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:05 smithi171 ceph-mon[37789]: pgmap v7942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:05 smithi044 ceph-mon[32404]: pgmap v7942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14170.972312 secs 2024-07-26T12:37:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14170.972312 secs 2024-07-26T12:37:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14170.972312 secs 2024-07-26T12:37:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:07 smithi171 ceph-mon[37789]: pgmap v7943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:07 smithi044 ceph-mon[32404]: pgmap v7943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:07 smithi116 ceph-mon[37097]: pgmap v7943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:09 smithi171 ceph-mon[37789]: pgmap v7944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:09 smithi044 ceph-mon[32404]: pgmap v7944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:10.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:09 smithi116 ceph-mon[37097]: pgmap v7944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:11.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:11 smithi116 ceph-mon[37097]: pgmap v7945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:11.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14175.972481 secs 2024-07-26T12:37:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:11 smithi171 ceph-mon[37789]: pgmap v7945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14175.972481 secs 2024-07-26T12:37:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:11 smithi044 ceph-mon[32404]: pgmap v7945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14175.972481 secs 2024-07-26T12:37:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:13 smithi171 ceph-mon[37789]: pgmap v7946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:13 smithi044 ceph-mon[32404]: pgmap v7946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:13 smithi116 ceph-mon[37097]: pgmap v7946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:15.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:15.248+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:37:15.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:15.248+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:15.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:15.248+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:37:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:15 smithi171 ceph-mon[37789]: pgmap v7947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:15 smithi044 ceph-mon[32404]: pgmap v7947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:15 smithi116 ceph-mon[37097]: pgmap v7947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:16.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14180.972687 secs 2024-07-26T12:37:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14180.972687 secs 2024-07-26T12:37:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14180.972687 secs 2024-07-26T12:37:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:17 smithi171 ceph-mon[37789]: pgmap v7948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:17 smithi044 ceph-mon[32404]: pgmap v7948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:17 smithi116 ceph-mon[37097]: pgmap v7948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:19 smithi171 ceph-mon[37789]: pgmap v7949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:19 smithi044 ceph-mon[32404]: pgmap v7949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:19 smithi116 ceph-mon[37097]: pgmap v7949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:21 smithi171 ceph-mon[37789]: pgmap v7950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14185.972838 secs 2024-07-26T12:37:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:21 smithi044 ceph-mon[32404]: pgmap v7950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14185.972838 secs 2024-07-26T12:37:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:21 smithi116 ceph-mon[37097]: pgmap v7950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14185.972838 secs 2024-07-26T12:37:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:23 smithi171 ceph-mon[37789]: pgmap v7951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:23 smithi044 ceph-mon[32404]: pgmap v7951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:23 smithi116 ceph-mon[37097]: pgmap v7951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:25 smithi044 ceph-mon[32404]: pgmap v7952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:25 smithi116 ceph-mon[37097]: pgmap v7952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:26.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:25 smithi171 ceph-mon[37789]: pgmap v7952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14190.972934 secs 2024-07-26T12:37:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14190.972934 secs 2024-07-26T12:37:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14190.972934 secs 2024-07-26T12:37:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:27 smithi044 ceph-mon[32404]: pgmap v7953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:27 smithi116 ceph-mon[37097]: pgmap v7953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:27 smithi171 ceph-mon[37789]: pgmap v7953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:29 smithi044 ceph-mon[32404]: pgmap v7954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:29 smithi116 ceph-mon[37097]: pgmap v7954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:29 smithi171 ceph-mon[37789]: pgmap v7954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:30.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:30.248+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:37:30.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:30.248+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:37:30.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:30.248+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:31 smithi044 ceph-mon[32404]: pgmap v7955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14195.973097 secs 2024-07-26T12:37:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:31 smithi116 ceph-mon[37097]: pgmap v7955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14195.973097 secs 2024-07-26T12:37:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:31 smithi171 ceph-mon[37789]: pgmap v7955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14195.973097 secs 2024-07-26T12:37:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:33 smithi044 ceph-mon[32404]: pgmap v7956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:33 smithi116 ceph-mon[37097]: pgmap v7956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:33 smithi171 ceph-mon[37789]: pgmap v7956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:35 smithi044 ceph-mon[32404]: pgmap v7957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:35 smithi116 ceph-mon[37097]: pgmap v7957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:35 smithi171 ceph-mon[37789]: pgmap v7957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14200.973207 secs 2024-07-26T12:37:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14200.973207 secs 2024-07-26T12:37:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14200.973207 secs 2024-07-26T12:37:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:37 smithi044 ceph-mon[32404]: pgmap v7958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:37 smithi116 ceph-mon[37097]: pgmap v7958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:37 smithi171 ceph-mon[37789]: pgmap v7958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:39 smithi044 ceph-mon[32404]: pgmap v7959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:39 smithi116 ceph-mon[37097]: pgmap v7959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:39 smithi171 ceph-mon[37789]: pgmap v7959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:41 smithi044 ceph-mon[32404]: pgmap v7960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14205.973370 secs 2024-07-26T12:37:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:41 smithi116 ceph-mon[37097]: pgmap v7960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14205.973370 secs 2024-07-26T12:37:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:41 smithi171 ceph-mon[37789]: pgmap v7960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14205.973370 secs 2024-07-26T12:37:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:43 smithi044 ceph-mon[32404]: pgmap v7961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:43 smithi116 ceph-mon[37097]: pgmap v7961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:43 smithi171 ceph-mon[37789]: pgmap v7961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:37:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:45.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:45.254+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:37:45.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:45.254+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:37:45.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:37:45.254+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:45 smithi044 ceph-mon[32404]: pgmap v7962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:45 smithi116 ceph-mon[37097]: pgmap v7962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:45 smithi171 ceph-mon[37789]: pgmap v7962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:46.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14210.973507 secs 2024-07-26T12:37:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:46 smithi044 ceph-mon[32404]: pgmap v7963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:46 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14210.973507 secs 2024-07-26T12:37:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:46 smithi116 ceph-mon[37097]: pgmap v7963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:46 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14210.973507 secs 2024-07-26T12:37:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:46 smithi171 ceph-mon[37789]: pgmap v7963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:46 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:49.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:48 smithi044 ceph-mon[32404]: pgmap v7964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:37:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:48 smithi116 ceph-mon[37097]: pgmap v7964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:37:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:48 smithi171 ceph-mon[37789]: pgmap v7964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:37:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:50 smithi044 ceph-mon[32404]: pgmap v7965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:50 smithi116 ceph-mon[37097]: pgmap v7965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:50 smithi171 ceph-mon[37789]: pgmap v7965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:51.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14215.973706 secs 2024-07-26T12:37:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14215.973706 secs 2024-07-26T12:37:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14215.973706 secs 2024-07-26T12:37:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:52 smithi044 ceph-mon[32404]: pgmap v7966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:37:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:52 smithi116 ceph-mon[37097]: pgmap v7966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:37:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:52 smithi171 ceph-mon[37789]: pgmap v7966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:37:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:54 smithi044 ceph-mon[32404]: pgmap v7967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:37:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:54 smithi116 ceph-mon[37097]: pgmap v7967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:37:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:54 smithi171 ceph-mon[37789]: pgmap v7967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:37:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14220.973862 secs 2024-07-26T12:37:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:56 smithi044 ceph-mon[32404]: pgmap v7968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14220.973862 secs 2024-07-26T12:37:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:56 smithi116 ceph-mon[37097]: pgmap v7968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14220.973862 secs 2024-07-26T12:37:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:56 smithi171 ceph-mon[37789]: pgmap v7968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:37:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:57 smithi171 ceph-mon[37789]: pgmap v7969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:57 smithi044 ceph-mon[32404]: pgmap v7969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:37:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:57 smithi116 ceph-mon[37097]: pgmap v7969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:37:59 smithi171 ceph-mon[37789]: pgmap v7970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:00.256+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:38:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:00.256+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:38:00.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:00.256+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:00.259 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:37:59 smithi044 ceph-mon[32404]: pgmap v7970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:00.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:37:59 smithi116 ceph-mon[37097]: pgmap v7970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14225.974026 secs 2024-07-26T12:38:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14225.974026 secs 2024-07-26T12:38:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14225.974026 secs 2024-07-26T12:38:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:02 smithi044 ceph-mon[32404]: pgmap v7971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:02 smithi116 ceph-mon[37097]: pgmap v7971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:02 smithi171 ceph-mon[37789]: pgmap v7971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:03 smithi171 ceph-mon[37789]: pgmap v7972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:03 smithi044 ceph-mon[32404]: pgmap v7972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:03 smithi116 ceph-mon[37097]: pgmap v7972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:05 smithi171 ceph-mon[37789]: pgmap v7973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:05 smithi044 ceph-mon[32404]: pgmap v7973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:05 smithi116 ceph-mon[37097]: pgmap v7973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14230.974163 secs 2024-07-26T12:38:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14230.974163 secs 2024-07-26T12:38:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14230.974163 secs 2024-07-26T12:38:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:07 smithi171 ceph-mon[37789]: pgmap v7974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:07 smithi044 ceph-mon[32404]: pgmap v7974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:07 smithi116 ceph-mon[37097]: pgmap v7974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:09 smithi171 ceph-mon[37789]: pgmap v7975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:09 smithi044 ceph-mon[32404]: pgmap v7975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:09 smithi116 ceph-mon[37097]: pgmap v7975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14235.974309 secs 2024-07-26T12:38:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14235.974309 secs 2024-07-26T12:38:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14235.974309 secs 2024-07-26T12:38:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:12 smithi044 ceph-mon[32404]: pgmap v7976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:12 smithi116 ceph-mon[37097]: pgmap v7976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:12 smithi171 ceph-mon[37789]: pgmap v7976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:13 smithi171 ceph-mon[37789]: pgmap v7977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:13 smithi044 ceph-mon[32404]: pgmap v7977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:13 smithi116 ceph-mon[37097]: pgmap v7977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:15.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:15.262+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:38:15.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:15.262+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:38:15.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:15.262+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:15 smithi171 ceph-mon[37789]: pgmap v7978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:15 smithi044 ceph-mon[32404]: pgmap v7978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:15 smithi116 ceph-mon[37097]: pgmap v7978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14240.974470 secs 2024-07-26T12:38:17.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:17.246+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T12:38:17.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:17.246+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:38:17.248 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14240.974470 secs 2024-07-26T12:38:17.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:17.247+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006480 con 0x7fcf6c1057b0 2024-07-26T12:38:17.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:17.247+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:38:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14240.974470 secs 2024-07-26T12:38:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:17 smithi171 ceph-mon[37789]: pgmap v7979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:18.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:17 smithi044 ceph-mon[32404]: pgmap v7979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:17 smithi116 ceph-mon[37097]: pgmap v7979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:19 smithi171 ceph-mon[37789]: pgmap v7980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:19 smithi044 ceph-mon[32404]: pgmap v7980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:19 smithi116 ceph-mon[37097]: pgmap v7980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14245.974666 secs 2024-07-26T12:38:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14245.974666 secs 2024-07-26T12:38:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14245.974666 secs 2024-07-26T12:38:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:22 smithi044 ceph-mon[32404]: pgmap v7981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:22 smithi116 ceph-mon[37097]: pgmap v7981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:22 smithi171 ceph-mon[37789]: pgmap v7981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:23 smithi171 ceph-mon[37789]: pgmap v7982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:23 smithi044 ceph-mon[32404]: pgmap v7982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:23 smithi116 ceph-mon[37097]: pgmap v7982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:25 smithi171 ceph-mon[37789]: pgmap v7983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:25 smithi044 ceph-mon[32404]: pgmap v7983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:25 smithi116 ceph-mon[37097]: pgmap v7983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14250.974810 secs 2024-07-26T12:38:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14250.974810 secs 2024-07-26T12:38:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14250.974810 secs 2024-07-26T12:38:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:27 smithi171 ceph-mon[37789]: pgmap v7984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:27 smithi044 ceph-mon[32404]: pgmap v7984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:27 smithi116 ceph-mon[37097]: pgmap v7984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:29 smithi171 ceph-mon[37789]: pgmap v7985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:30.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:30.262+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:38:30.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:30.262+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:30.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:30.262+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:38:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:29 smithi044 ceph-mon[32404]: pgmap v7985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:29 smithi116 ceph-mon[37097]: pgmap v7985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14255.974962 secs 2024-07-26T12:38:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14255.974962 secs 2024-07-26T12:38:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14255.974962 secs 2024-07-26T12:38:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:32 smithi044 ceph-mon[32404]: pgmap v7986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:32 smithi116 ceph-mon[37097]: pgmap v7986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:32 smithi171 ceph-mon[37789]: pgmap v7986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:33 smithi171 ceph-mon[37789]: pgmap v7987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:33 smithi044 ceph-mon[32404]: pgmap v7987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:33 smithi116 ceph-mon[37097]: pgmap v7987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:35 smithi171 ceph-mon[37789]: pgmap v7988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:35 smithi044 ceph-mon[32404]: pgmap v7988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:35 smithi116 ceph-mon[37097]: pgmap v7988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14260.975101 secs 2024-07-26T12:38:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14260.975101 secs 2024-07-26T12:38:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14260.975101 secs 2024-07-26T12:38:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:37 smithi171 ceph-mon[37789]: pgmap v7989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:37 smithi044 ceph-mon[32404]: pgmap v7989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:37 smithi116 ceph-mon[37097]: pgmap v7989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:39 smithi171 ceph-mon[37789]: pgmap v7990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:39 smithi044 ceph-mon[32404]: pgmap v7990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:39 smithi116 ceph-mon[37097]: pgmap v7990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:38:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14265.975267 secs 2024-07-26T12:38:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14265.975267 secs 2024-07-26T12:38:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14265.975267 secs 2024-07-26T12:38:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:42 smithi044 ceph-mon[32404]: pgmap v7991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:42 smithi116 ceph-mon[37097]: pgmap v7991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:42 smithi171 ceph-mon[37789]: pgmap v7991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:43 smithi171 ceph-mon[37789]: pgmap v7992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:43 smithi044 ceph-mon[32404]: pgmap v7992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:43 smithi116 ceph-mon[37097]: pgmap v7992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:45.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:45.267+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:38:45.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:45.267+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:45.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:38:45.267+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:38:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:45 smithi171 ceph-mon[37789]: pgmap v7993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:45 smithi044 ceph-mon[32404]: pgmap v7993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:45 smithi116 ceph-mon[37097]: pgmap v7993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14270.975424 secs 2024-07-26T12:38:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:47.098 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14270.975424 secs 2024-07-26T12:38:47.098 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14270.975424 secs 2024-07-26T12:38:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:47.972 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:47 smithi044 ceph-mon[32404]: pgmap v7994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:38:48.062 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:47 smithi116 ceph-mon[37097]: pgmap v7994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:38:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:47 smithi171 ceph-mon[37789]: pgmap v7994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:38:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:50 smithi044 ceph-mon[32404]: pgmap v7995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:51.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:50 smithi116 ceph-mon[37097]: pgmap v7995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:50 smithi171 ceph-mon[37789]: pgmap v7995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14275.975579 secs 2024-07-26T12:38:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14275.975579 secs 2024-07-26T12:38:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14275.975579 secs 2024-07-26T12:38:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: pgmap v7996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:38:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:53.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:52 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: pgmap v7996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:53.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:52 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: pgmap v7996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:38:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:53.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:53.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:53.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:52 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:53 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:38:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:53 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:38:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:53 smithi044 ceph-mon[32404]: pgmap v7997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:54.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:53 smithi044 ceph-mon[32404]: pgmap v7998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:53 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:53 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:53 smithi116 ceph-mon[37097]: pgmap v7997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:53 smithi116 ceph-mon[37097]: pgmap v7998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:53 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:53 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:53 smithi171 ceph-mon[37789]: pgmap v7997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:53 smithi171 ceph-mon[37789]: pgmap v7998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:38:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:54 smithi171 ceph-mon[37789]: pgmap v7999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:38:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:54 smithi044 ceph-mon[32404]: pgmap v7999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:38:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:54 smithi116 ceph-mon[37097]: pgmap v7999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:38:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14280.975747 secs 2024-07-26T12:38:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14280.975747 secs 2024-07-26T12:38:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14280.975747 secs 2024-07-26T12:38:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:57 smithi044 ceph-mon[32404]: pgmap v8000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:57 smithi116 ceph-mon[37097]: pgmap v8000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:57 smithi171 ceph-mon[37789]: pgmap v8000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:38:59 smithi044 ceph-mon[32404]: pgmap v8001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:39:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:38:59 smithi116 ceph-mon[37097]: pgmap v8001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:39:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:38:59 smithi171 ceph-mon[37789]: pgmap v8001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:39:00.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:00.268+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:39:00.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:00.268+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:00.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:00.268+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:39:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:01 smithi044 ceph-mon[32404]: pgmap v8002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:39:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14285.975904 secs 2024-07-26T12:39:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:01 smithi116 ceph-mon[37097]: pgmap v8002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:39:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14285.975904 secs 2024-07-26T12:39:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:01 smithi171 ceph-mon[37789]: pgmap v8002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:39:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14285.975904 secs 2024-07-26T12:39:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:03 smithi044 ceph-mon[32404]: pgmap v8003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:03 smithi116 ceph-mon[37097]: pgmap v8003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:03 smithi171 ceph-mon[37789]: pgmap v8003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:05.889 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:05 smithi044 ceph-mon[32404]: pgmap v8004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:05 smithi116 ceph-mon[37097]: pgmap v8004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:05 smithi171 ceph-mon[37789]: pgmap v8004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14290.976011 secs 2024-07-26T12:39:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14290.976011 secs 2024-07-26T12:39:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14290.976011 secs 2024-07-26T12:39:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:07 smithi044 ceph-mon[32404]: pgmap v8005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:07 smithi116 ceph-mon[37097]: pgmap v8005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:07 smithi171 ceph-mon[37789]: pgmap v8005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:09 smithi044 ceph-mon[32404]: pgmap v8006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:09 smithi116 ceph-mon[37097]: pgmap v8006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:09 smithi171 ceph-mon[37789]: pgmap v8006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:11 smithi044 ceph-mon[32404]: pgmap v8007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14295.976126 secs 2024-07-26T12:39:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:11 smithi116 ceph-mon[37097]: pgmap v8007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14295.976126 secs 2024-07-26T12:39:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:11 smithi171 ceph-mon[37789]: pgmap v8007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14295.976126 secs 2024-07-26T12:39:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:13 smithi044 ceph-mon[32404]: pgmap v8008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:13 smithi116 ceph-mon[37097]: pgmap v8008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:13 smithi171 ceph-mon[37789]: pgmap v8008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:15.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:15.274+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:39:15.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:15.274+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:39:15.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:15.274+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:15 smithi044 ceph-mon[32404]: pgmap v8009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:15 smithi116 ceph-mon[37097]: pgmap v8009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:15 smithi171 ceph-mon[37789]: pgmap v8009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:17.005 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14300.976280 secs 2024-07-26T12:39:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14300.976280 secs 2024-07-26T12:39:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14300.976280 secs 2024-07-26T12:39:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:17 smithi044 ceph-mon[32404]: pgmap v8010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:17 smithi116 ceph-mon[37097]: pgmap v8010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:17 smithi171 ceph-mon[37789]: pgmap v8010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:19 smithi044 ceph-mon[32404]: pgmap v8011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:19 smithi116 ceph-mon[37097]: pgmap v8011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:19 smithi171 ceph-mon[37789]: pgmap v8011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:21 smithi044 ceph-mon[32404]: pgmap v8012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14305.976423 secs 2024-07-26T12:39:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:21 smithi116 ceph-mon[37097]: pgmap v8012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14305.976423 secs 2024-07-26T12:39:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:21 smithi171 ceph-mon[37789]: pgmap v8012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14305.976423 secs 2024-07-26T12:39:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:23 smithi044 ceph-mon[32404]: pgmap v8013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:23 smithi116 ceph-mon[37097]: pgmap v8013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:23 smithi171 ceph-mon[37789]: pgmap v8013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:25 smithi044 ceph-mon[32404]: pgmap v8014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:25 smithi116 ceph-mon[37097]: pgmap v8014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:25 smithi171 ceph-mon[37789]: pgmap v8014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14310.976583 secs 2024-07-26T12:39:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14310.976583 secs 2024-07-26T12:39:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14310.976583 secs 2024-07-26T12:39:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:27 smithi044 ceph-mon[32404]: pgmap v8015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:27 smithi116 ceph-mon[37097]: pgmap v8015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:27 smithi171 ceph-mon[37789]: pgmap v8015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:29 smithi044 ceph-mon[32404]: pgmap v8016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:29 smithi116 ceph-mon[37097]: pgmap v8016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:29 smithi171 ceph-mon[37789]: pgmap v8016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:30.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:30.275+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:39:30.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:30.275+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:39:30.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:30.275+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:31 smithi044 ceph-mon[32404]: pgmap v8017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14315.976742 secs 2024-07-26T12:39:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:31 smithi116 ceph-mon[37097]: pgmap v8017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14315.976742 secs 2024-07-26T12:39:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:31 smithi171 ceph-mon[37789]: pgmap v8017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14315.976742 secs 2024-07-26T12:39:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:33 smithi044 ceph-mon[32404]: pgmap v8018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:33 smithi116 ceph-mon[37097]: pgmap v8018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:33 smithi171 ceph-mon[37789]: pgmap v8018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:36.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:35 smithi044 ceph-mon[32404]: pgmap v8019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:35 smithi116 ceph-mon[37097]: pgmap v8019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:35 smithi171 ceph-mon[37789]: pgmap v8019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:36.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14320.976905 secs 2024-07-26T12:39:36.918 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:36 smithi044 ceph-mon[32404]: pgmap v8020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14320.976905 secs 2024-07-26T12:39:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:36 smithi116 ceph-mon[37097]: pgmap v8020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14320.976905 secs 2024-07-26T12:39:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:36 smithi171 ceph-mon[37789]: pgmap v8020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:39 smithi044 ceph-mon[32404]: pgmap v8021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:39 smithi116 ceph-mon[37097]: pgmap v8021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:39 smithi171 ceph-mon[37789]: pgmap v8021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:41 smithi044 ceph-mon[32404]: pgmap v8022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14325.977075 secs 2024-07-26T12:39:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:41 smithi116 ceph-mon[37097]: pgmap v8022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14325.977075 secs 2024-07-26T12:39:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:41 smithi171 ceph-mon[37789]: pgmap v8022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14325.977075 secs 2024-07-26T12:39:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:43 smithi044 ceph-mon[32404]: pgmap v8023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:43 smithi116 ceph-mon[37097]: pgmap v8023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:43 smithi171 ceph-mon[37789]: pgmap v8023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:45.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:45.280+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:39:45.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:45.280+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:45.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:39:45.280+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:39:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:45 smithi044 ceph-mon[32404]: pgmap v8024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:45 smithi116 ceph-mon[37097]: pgmap v8024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:45 smithi171 ceph-mon[37789]: pgmap v8024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14330.977171 secs 2024-07-26T12:39:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14330.977171 secs 2024-07-26T12:39:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14330.977171 secs 2024-07-26T12:39:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:47 smithi044 ceph-mon[32404]: pgmap v8025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:47 smithi116 ceph-mon[37097]: pgmap v8025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:47 smithi171 ceph-mon[37789]: pgmap v8025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:39:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:49 smithi044 ceph-mon[32404]: pgmap v8026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:49 smithi116 ceph-mon[37097]: pgmap v8026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:49 smithi171 ceph-mon[37789]: pgmap v8026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:39:51.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:51 smithi044 ceph-mon[32404]: pgmap v8027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:51.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14335.977333 secs 2024-07-26T12:39:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:51 smithi116 ceph-mon[37097]: pgmap v8027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14335.977333 secs 2024-07-26T12:39:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:51 smithi171 ceph-mon[37789]: pgmap v8027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14335.977333 secs 2024-07-26T12:39:54.016 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: pgmap v8028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:54.016 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:54.016 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.017 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:53 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: pgmap v8028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:54.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:54.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.021 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.021 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.021 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:53 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: pgmap v8028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:54.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:53 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:54 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:39:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:54 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:39:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:54 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:39:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:54 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:39:55.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:54 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:39:55.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:54 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:39:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:55 smithi171 ceph-mon[37789]: pgmap v8029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:55 smithi044 ceph-mon[32404]: pgmap v8029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:55 smithi116 ceph-mon[37097]: pgmap v8029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:39:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14340.977461 secs 2024-07-26T12:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:57.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:56 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14340.977461 secs 2024-07-26T12:39:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:57.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:56 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14340.977461 secs 2024-07-26T12:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:57.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:56 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:57 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:39:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:57 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:39:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:57 smithi044 ceph-mon[32404]: pgmap v8030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:57 smithi044 ceph-mon[32404]: pgmap v8031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:39:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:57 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:39:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:57 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:39:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:57 smithi116 ceph-mon[37097]: pgmap v8030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:57 smithi116 ceph-mon[37097]: pgmap v8031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:39:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:57 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:39:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:57 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:39:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:57 smithi171 ceph-mon[37789]: pgmap v8030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:39:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:57 smithi171 ceph-mon[37789]: pgmap v8031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:39:59 smithi044 ceph-mon[32404]: pgmap v8032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:39:59 smithi116 ceph-mon[37097]: pgmap v8032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:39:59 smithi171 ceph-mon[37789]: pgmap v8032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:00.281+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:40:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:00.281+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:40:00.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:00.281+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:40:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:01 smithi044 ceph-mon[32404]: pgmap v8033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14345.977596 secs 2024-07-26T12:40:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:01 smithi116 ceph-mon[37097]: pgmap v8033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14345.977596 secs 2024-07-26T12:40:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:01 smithi171 ceph-mon[37789]: pgmap v8033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14345.977596 secs 2024-07-26T12:40:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:03 smithi044 ceph-mon[32404]: pgmap v8034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:03 smithi116 ceph-mon[37097]: pgmap v8034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:03 smithi171 ceph-mon[37789]: pgmap v8034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:40:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:05 smithi044 ceph-mon[32404]: pgmap v8035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:05 smithi116 ceph-mon[37097]: pgmap v8035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:06.059 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:05 smithi171 ceph-mon[37789]: pgmap v8035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14350.977736 secs 2024-07-26T12:40:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14350.977736 secs 2024-07-26T12:40:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14350.977736 secs 2024-07-26T12:40:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:07 smithi044 ceph-mon[32404]: pgmap v8036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:07 smithi116 ceph-mon[37097]: pgmap v8036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:07 smithi171 ceph-mon[37789]: pgmap v8036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:08 smithi044 ceph-mon[32404]: pgmap v8037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:08 smithi116 ceph-mon[37097]: pgmap v8037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:08 smithi171 ceph-mon[37789]: pgmap v8037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:11 smithi171 ceph-mon[37789]: pgmap v8038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14355.977829 secs 2024-07-26T12:40:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:11 smithi044 ceph-mon[32404]: pgmap v8038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14355.977829 secs 2024-07-26T12:40:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:11 smithi116 ceph-mon[37097]: pgmap v8038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14355.977829 secs 2024-07-26T12:40:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:13 smithi171 ceph-mon[37789]: pgmap v8039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:13 smithi044 ceph-mon[32404]: pgmap v8039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:13 smithi116 ceph-mon[37097]: pgmap v8039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:15.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:15.287+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:40:15.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:15.287+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:15.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:15.287+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:40:15.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:15 smithi171 ceph-mon[37789]: pgmap v8040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:15 smithi044 ceph-mon[32404]: pgmap v8040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:15 smithi116 ceph-mon[37097]: pgmap v8040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14360.977990 secs 2024-07-26T12:40:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14360.977990 secs 2024-07-26T12:40:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14360.977990 secs 2024-07-26T12:40:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:17 smithi171 ceph-mon[37789]: pgmap v8041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:17 smithi044 ceph-mon[32404]: pgmap v8041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:17 smithi116 ceph-mon[37097]: pgmap v8041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:19 smithi171 ceph-mon[37789]: pgmap v8042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:19 smithi044 ceph-mon[32404]: pgmap v8042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:19 smithi116 ceph-mon[37097]: pgmap v8042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:21 smithi171 ceph-mon[37789]: pgmap v8043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14365.978152 secs 2024-07-26T12:40:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:21 smithi044 ceph-mon[32404]: pgmap v8043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14365.978152 secs 2024-07-26T12:40:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:21 smithi116 ceph-mon[37097]: pgmap v8043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14365.978152 secs 2024-07-26T12:40:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:23 smithi044 ceph-mon[32404]: pgmap v8044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:23 smithi116 ceph-mon[37097]: pgmap v8044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:23 smithi171 ceph-mon[37789]: pgmap v8044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:25 smithi044 ceph-mon[32404]: pgmap v8045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:25 smithi116 ceph-mon[37097]: pgmap v8045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:25 smithi171 ceph-mon[37789]: pgmap v8045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14370.978304 secs 2024-07-26T12:40:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14370.978304 secs 2024-07-26T12:40:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14370.978304 secs 2024-07-26T12:40:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:27 smithi044 ceph-mon[32404]: pgmap v8046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:27 smithi116 ceph-mon[37097]: pgmap v8046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:27 smithi171 ceph-mon[37789]: pgmap v8046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:29 smithi044 ceph-mon[32404]: pgmap v8047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:29 smithi116 ceph-mon[37097]: pgmap v8047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:29 smithi171 ceph-mon[37789]: pgmap v8047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:30.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:30.288+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:40:30.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:30.288+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:30.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:30.288+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:40:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:31 smithi044 ceph-mon[32404]: pgmap v8048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14375.978455 secs 2024-07-26T12:40:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:31 smithi116 ceph-mon[37097]: pgmap v8048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14375.978455 secs 2024-07-26T12:40:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:31 smithi171 ceph-mon[37789]: pgmap v8048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14375.978455 secs 2024-07-26T12:40:32.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:32.271+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:40:32.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:32.271+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:40:32.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:32.271+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:40:32.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:32.272+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:40:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:33 smithi044 ceph-mon[32404]: pgmap v8049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:33 smithi116 ceph-mon[37097]: pgmap v8049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:33 smithi171 ceph-mon[37789]: pgmap v8049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:36.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:35 smithi044 ceph-mon[32404]: pgmap v8050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:35 smithi116 ceph-mon[37097]: pgmap v8050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:35 smithi171 ceph-mon[37789]: pgmap v8050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14380.978565 secs 2024-07-26T12:40:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14380.978565 secs 2024-07-26T12:40:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14380.978565 secs 2024-07-26T12:40:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:37 smithi044 ceph-mon[32404]: pgmap v8051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:37 smithi116 ceph-mon[37097]: pgmap v8051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:37 smithi171 ceph-mon[37789]: pgmap v8051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:39 smithi044 ceph-mon[32404]: pgmap v8052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:39 smithi116 ceph-mon[37097]: pgmap v8052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:39 smithi171 ceph-mon[37789]: pgmap v8052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:41 smithi044 ceph-mon[32404]: pgmap v8053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14385.978729 secs 2024-07-26T12:40:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:41 smithi116 ceph-mon[37097]: pgmap v8053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14385.978729 secs 2024-07-26T12:40:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:41 smithi171 ceph-mon[37789]: pgmap v8053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14385.978729 secs 2024-07-26T12:40:44.014 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:43 smithi116 ceph-mon[37097]: pgmap v8054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:43 smithi044 ceph-mon[32404]: pgmap v8054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:43 smithi171 ceph-mon[37789]: pgmap v8054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:45.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:45.293+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:40:45.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:45.293+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:40:45.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:40:45.293+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:45 smithi044 ceph-mon[32404]: pgmap v8055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:45 smithi116 ceph-mon[37097]: pgmap v8055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:45 smithi171 ceph-mon[37789]: pgmap v8055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14390.978869 secs 2024-07-26T12:40:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14390.978869 secs 2024-07-26T12:40:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14390.978869 secs 2024-07-26T12:40:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:47 smithi044 ceph-mon[32404]: pgmap v8056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:47 smithi116 ceph-mon[37097]: pgmap v8056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:47 smithi171 ceph-mon[37789]: pgmap v8056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:49 smithi044 ceph-mon[32404]: pgmap v8057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:49 smithi116 ceph-mon[37097]: pgmap v8057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:49 smithi171 ceph-mon[37789]: pgmap v8057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:40:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:51 smithi044 ceph-mon[32404]: pgmap v8058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14395.979014 secs 2024-07-26T12:40:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:51 smithi116 ceph-mon[37097]: pgmap v8058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14395.979014 secs 2024-07-26T12:40:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:51 smithi171 ceph-mon[37789]: pgmap v8058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14395.979014 secs 2024-07-26T12:40:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:53 smithi116 ceph-mon[37097]: pgmap v8059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:53 smithi044 ceph-mon[32404]: pgmap v8059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:53 smithi171 ceph-mon[37789]: pgmap v8059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:40:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:55 smithi044 ceph-mon[32404]: pgmap v8060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:55 smithi116 ceph-mon[37097]: pgmap v8060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:55 smithi171 ceph-mon[37789]: pgmap v8060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:40:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14400.979171 secs 2024-07-26T12:40:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14400.979171 secs 2024-07-26T12:40:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14400.979171 secs 2024-07-26T12:40:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:57 smithi044 ceph-mon[32404]: pgmap v8061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:57 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:57 smithi116 ceph-mon[37097]: pgmap v8061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:57 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:57 smithi171 ceph-mon[37789]: pgmap v8061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:40:58.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:57 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:59 smithi044 ceph-mon[32404]: pgmap v8062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:40:59 smithi044 ceph-mon[32404]: pgmap v8063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:41:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:59 smithi116 ceph-mon[37097]: pgmap v8062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:40:59 smithi116 ceph-mon[37097]: pgmap v8063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:41:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:59 smithi171 ceph-mon[37789]: pgmap v8062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:40:59 smithi171 ceph-mon[37789]: pgmap v8063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:41:00.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:00.295+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:41:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:00.295+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:00.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:00.295+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:41:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:01 smithi044 ceph-mon[32404]: pgmap v8064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:41:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14405.979334 secs 2024-07-26T12:41:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:01 smithi116 ceph-mon[37097]: pgmap v8064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:41:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14405.979334 secs 2024-07-26T12:41:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:01 smithi171 ceph-mon[37789]: pgmap v8064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:41:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14405.979334 secs 2024-07-26T12:41:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:03 smithi044 ceph-mon[32404]: pgmap v8065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:41:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:03 smithi116 ceph-mon[37097]: pgmap v8065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:41:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:03 smithi171 ceph-mon[37789]: pgmap v8065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:41:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:05 smithi044 ceph-mon[32404]: pgmap v8066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:41:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:05 smithi116 ceph-mon[37097]: pgmap v8066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:41:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:05 smithi171 ceph-mon[37789]: pgmap v8066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:41:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14410.979477 secs 2024-07-26T12:41:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14410.979477 secs 2024-07-26T12:41:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14410.979477 secs 2024-07-26T12:41:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:07 smithi116 ceph-mon[37097]: pgmap v8067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:41:08.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:07 smithi171 ceph-mon[37789]: pgmap v8067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:41:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:07 smithi044 ceph-mon[32404]: pgmap v8067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:41:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:08 smithi171 ceph-mon[37789]: pgmap v8068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:41:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:08 smithi044 ceph-mon[32404]: pgmap v8068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:41:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:08 smithi116 ceph-mon[37097]: pgmap v8068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:41:11.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:10 smithi044 ceph-mon[32404]: pgmap v8069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:41:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:10 smithi116 ceph-mon[37097]: pgmap v8069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:41:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:10 smithi171 ceph-mon[37789]: pgmap v8069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:41:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14415.979680 secs 2024-07-26T12:41:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14415.979680 secs 2024-07-26T12:41:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14415.979680 secs 2024-07-26T12:41:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:12 smithi044 ceph-mon[32404]: pgmap v8070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:12 smithi116 ceph-mon[37097]: pgmap v8070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:12 smithi171 ceph-mon[37789]: pgmap v8070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T12:41:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:41:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf5808d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:41:14.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf5808d000 comp rx=0 tx=0).stop 2024-07-26T12:41:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T12:41:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:41:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:14.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.733+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:14.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.734+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:14.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:14.735+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58074040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:41:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:14 smithi044 ceph-mon[32404]: pgmap v8071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:14 smithi116 ceph-mon[37097]: pgmap v8071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:15.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:15.296+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:41:15.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:15.296+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:15.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:15.297+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:41:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:14 smithi171 ceph-mon[37789]: pgmap v8071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14420.979825 secs 2024-07-26T12:41:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:16 smithi044 ceph-mon[32404]: pgmap v8072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:41:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14420.979825 secs 2024-07-26T12:41:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:16 smithi116 ceph-mon[37097]: pgmap v8072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:41:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:17.279+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:41:17.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:17.279+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:41:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:17.279+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:41:17.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:17.280+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:41:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14420.979825 secs 2024-07-26T12:41:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:16 smithi171 ceph-mon[37789]: pgmap v8072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:41:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:18 smithi044 ceph-mon[32404]: pgmap v8073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:18 smithi116 ceph-mon[37097]: pgmap v8073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:18 smithi171 ceph-mon[37789]: pgmap v8073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:41:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:20 smithi044 ceph-mon[32404]: pgmap v8074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:20 smithi116 ceph-mon[37097]: pgmap v8074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:20 smithi171 ceph-mon[37789]: pgmap v8074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14425.979978 secs 2024-07-26T12:41:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14425.979978 secs 2024-07-26T12:41:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14425.979978 secs 2024-07-26T12:41:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:22 smithi044 ceph-mon[32404]: pgmap v8075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:22 smithi116 ceph-mon[37097]: pgmap v8075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:22 smithi171 ceph-mon[37789]: pgmap v8075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:24 smithi044 ceph-mon[32404]: pgmap v8076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:24 smithi116 ceph-mon[37097]: pgmap v8076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:24 smithi171 ceph-mon[37789]: pgmap v8076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14430.980121 secs 2024-07-26T12:41:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:26 smithi044 ceph-mon[32404]: pgmap v8077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14430.980121 secs 2024-07-26T12:41:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:26 smithi116 ceph-mon[37097]: pgmap v8077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14430.980121 secs 2024-07-26T12:41:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:26 smithi171 ceph-mon[37789]: pgmap v8077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:28 smithi044 ceph-mon[32404]: pgmap v8078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:28 smithi116 ceph-mon[37097]: pgmap v8078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:28 smithi171 ceph-mon[37789]: pgmap v8078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:30.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:30.298+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:41:30.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:30.299+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:41:30.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:30.299+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:30 smithi044 ceph-mon[32404]: pgmap v8079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:30 smithi116 ceph-mon[37097]: pgmap v8079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:30 smithi171 ceph-mon[37789]: pgmap v8079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14435.980282 secs 2024-07-26T12:41:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14435.980282 secs 2024-07-26T12:41:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14435.980282 secs 2024-07-26T12:41:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:32 smithi044 ceph-mon[32404]: pgmap v8080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:32 smithi116 ceph-mon[37097]: pgmap v8080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:32 smithi171 ceph-mon[37789]: pgmap v8080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:34 smithi044 ceph-mon[32404]: pgmap v8081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:34 smithi116 ceph-mon[37097]: pgmap v8081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:34 smithi171 ceph-mon[37789]: pgmap v8081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14440.980453 secs 2024-07-26T12:41:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:36 smithi044 ceph-mon[32404]: pgmap v8082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14440.980453 secs 2024-07-26T12:41:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:36 smithi116 ceph-mon[37097]: pgmap v8082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14440.980453 secs 2024-07-26T12:41:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:36 smithi171 ceph-mon[37789]: pgmap v8082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:38 smithi044 ceph-mon[32404]: pgmap v8083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:38 smithi116 ceph-mon[37097]: pgmap v8083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:38 smithi171 ceph-mon[37789]: pgmap v8083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:40 smithi044 ceph-mon[32404]: pgmap v8084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:40 smithi116 ceph-mon[37097]: pgmap v8084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:40 smithi171 ceph-mon[37789]: pgmap v8084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14445.980600 secs 2024-07-26T12:41:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14445.980600 secs 2024-07-26T12:41:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14445.980600 secs 2024-07-26T12:41:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:42 smithi044 ceph-mon[32404]: pgmap v8085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:42 smithi116 ceph-mon[37097]: pgmap v8085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:42 smithi171 ceph-mon[37789]: pgmap v8085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:44 smithi044 ceph-mon[32404]: pgmap v8086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:44 smithi116 ceph-mon[37097]: pgmap v8086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:45.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:45.304+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:41:45.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:45.304+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:45.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:41:45.304+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:41:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:44 smithi171 ceph-mon[37789]: pgmap v8086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14450.980756 secs 2024-07-26T12:41:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:46 smithi044 ceph-mon[32404]: pgmap v8087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14450.980756 secs 2024-07-26T12:41:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:46 smithi116 ceph-mon[37097]: pgmap v8087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:47.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14450.980756 secs 2024-07-26T12:41:47.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:46 smithi171 ceph-mon[37789]: pgmap v8087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:49 smithi044 ceph-mon[32404]: pgmap v8088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:49 smithi116 ceph-mon[37097]: pgmap v8088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:49 smithi171 ceph-mon[37789]: pgmap v8088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:51 smithi044 ceph-mon[32404]: pgmap v8089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:51 smithi116 ceph-mon[37097]: pgmap v8089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:51 smithi171 ceph-mon[37789]: pgmap v8089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:41:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14455.980864 secs 2024-07-26T12:41:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14455.980864 secs 2024-07-26T12:41:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14455.980864 secs 2024-07-26T12:41:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:53 smithi171 ceph-mon[37789]: pgmap v8090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:53 smithi044 ceph-mon[32404]: pgmap v8090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:53 smithi116 ceph-mon[37097]: pgmap v8090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:55 smithi171 ceph-mon[37789]: pgmap v8091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:55 smithi044 ceph-mon[32404]: pgmap v8091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:55 smithi116 ceph-mon[37097]: pgmap v8091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:41:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14460.981008 secs 2024-07-26T12:41:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:57 smithi171 ceph-mon[37789]: pgmap v8092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14460.981008 secs 2024-07-26T12:41:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:57 smithi044 ceph-mon[32404]: pgmap v8092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14460.981008 secs 2024-07-26T12:41:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:57 smithi116 ceph-mon[37097]: pgmap v8092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:59 smithi044 ceph-mon[32404]: pgmap v8093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:41:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:59 smithi116 ceph-mon[37097]: pgmap v8093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:41:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:59 smithi171 ceph-mon[37789]: pgmap v8093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:41:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:00.306+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:42:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:00.306+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:00.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:00.306+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:42:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:01 smithi044 ceph-mon[32404]: pgmap v8094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:01 smithi044 ceph-mon[32404]: pgmap v8095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14465.981173 secs 2024-07-26T12:42:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:01 smithi116 ceph-mon[37097]: pgmap v8094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:01 smithi116 ceph-mon[37097]: pgmap v8095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14465.981173 secs 2024-07-26T12:42:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:01 smithi171 ceph-mon[37789]: pgmap v8094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:01 smithi171 ceph-mon[37789]: pgmap v8095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14465.981173 secs 2024-07-26T12:42:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:03 smithi044 ceph-mon[32404]: pgmap v8096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:03 smithi116 ceph-mon[37097]: pgmap v8096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:03 smithi171 ceph-mon[37789]: pgmap v8096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:05 smithi044 ceph-mon[32404]: pgmap v8097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:05 smithi116 ceph-mon[37097]: pgmap v8097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:05 smithi171 ceph-mon[37789]: pgmap v8097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14470.981297 secs 2024-07-26T12:42:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14470.981297 secs 2024-07-26T12:42:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14470.981297 secs 2024-07-26T12:42:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:07 smithi044 ceph-mon[32404]: pgmap v8098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:07 smithi116 ceph-mon[37097]: pgmap v8098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:07 smithi171 ceph-mon[37789]: pgmap v8098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:09 smithi044 ceph-mon[32404]: pgmap v8099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:09 smithi116 ceph-mon[37097]: pgmap v8099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:09 smithi171 ceph-mon[37789]: pgmap v8099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T12:42:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:11 smithi044 ceph-mon[32404]: pgmap v8100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14475.981459 secs 2024-07-26T12:42:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:11 smithi116 ceph-mon[37097]: pgmap v8100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14475.981459 secs 2024-07-26T12:42:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:11 smithi171 ceph-mon[37789]: pgmap v8100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:12.183 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14475.981459 secs 2024-07-26T12:42:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:13 smithi044 ceph-mon[32404]: pgmap v8101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:13 smithi116 ceph-mon[37097]: pgmap v8101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:13 smithi171 ceph-mon[37789]: pgmap v8101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:15.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:15.311+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:42:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:15.311+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:15.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:15.311+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:42:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:15 smithi044 ceph-mon[32404]: pgmap v8102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:15 smithi116 ceph-mon[37097]: pgmap v8102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:15 smithi171 ceph-mon[37789]: pgmap v8102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14480.981662 secs 2024-07-26T12:42:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14480.981662 secs 2024-07-26T12:42:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14480.981662 secs 2024-07-26T12:42:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:17 smithi171 ceph-mon[37789]: pgmap v8103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:17 smithi044 ceph-mon[32404]: pgmap v8103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:17 smithi116 ceph-mon[37097]: pgmap v8103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:18 smithi171 ceph-mon[37789]: pgmap v8104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:18 smithi044 ceph-mon[32404]: pgmap v8104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:18 smithi116 ceph-mon[37097]: pgmap v8104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:20 smithi044 ceph-mon[32404]: pgmap v8105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:20 smithi116 ceph-mon[37097]: pgmap v8105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:20 smithi171 ceph-mon[37789]: pgmap v8105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14485.981799 secs 2024-07-26T12:42:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14485.981799 secs 2024-07-26T12:42:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14485.981799 secs 2024-07-26T12:42:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:22 smithi044 ceph-mon[32404]: pgmap v8106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:22 smithi116 ceph-mon[37097]: pgmap v8106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:22 smithi171 ceph-mon[37789]: pgmap v8106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:24 smithi044 ceph-mon[32404]: pgmap v8107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:24 smithi116 ceph-mon[37097]: pgmap v8107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:24 smithi171 ceph-mon[37789]: pgmap v8107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14490.981952 secs 2024-07-26T12:42:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:26 smithi044 ceph-mon[32404]: pgmap v8108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14490.981952 secs 2024-07-26T12:42:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:26 smithi116 ceph-mon[37097]: pgmap v8108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14490.981952 secs 2024-07-26T12:42:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:26 smithi171 ceph-mon[37789]: pgmap v8108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:28 smithi044 ceph-mon[32404]: pgmap v8109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:28 smithi116 ceph-mon[37097]: pgmap v8109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:28 smithi171 ceph-mon[37789]: pgmap v8109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:30.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:30.312+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:42:30.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:30.312+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:30.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:30.312+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:42:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:30 smithi044 ceph-mon[32404]: pgmap v8110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:30 smithi116 ceph-mon[37097]: pgmap v8110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:30 smithi171 ceph-mon[37789]: pgmap v8110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14495.982094 secs 2024-07-26T12:42:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14495.982094 secs 2024-07-26T12:42:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14495.982094 secs 2024-07-26T12:42:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:32 smithi044 ceph-mon[32404]: pgmap v8111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:32 smithi116 ceph-mon[37097]: pgmap v8111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:32 smithi171 ceph-mon[37789]: pgmap v8111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:34 smithi044 ceph-mon[32404]: pgmap v8112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:34 smithi116 ceph-mon[37097]: pgmap v8112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:34 smithi171 ceph-mon[37789]: pgmap v8112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14500.982251 secs 2024-07-26T12:42:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:36 smithi044 ceph-mon[32404]: pgmap v8113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14500.982251 secs 2024-07-26T12:42:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:36 smithi116 ceph-mon[37097]: pgmap v8113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14500.982251 secs 2024-07-26T12:42:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:36 smithi171 ceph-mon[37789]: pgmap v8113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:38 smithi044 ceph-mon[32404]: pgmap v8114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:38 smithi116 ceph-mon[37097]: pgmap v8114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:38 smithi171 ceph-mon[37789]: pgmap v8114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:40 smithi044 ceph-mon[32404]: pgmap v8115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:40 smithi116 ceph-mon[37097]: pgmap v8115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:40 smithi171 ceph-mon[37789]: pgmap v8115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14505.982413 secs 2024-07-26T12:42:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14505.982413 secs 2024-07-26T12:42:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14505.982413 secs 2024-07-26T12:42:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:42 smithi044 ceph-mon[32404]: pgmap v8116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:42 smithi116 ceph-mon[37097]: pgmap v8116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:42 smithi171 ceph-mon[37789]: pgmap v8116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:44 smithi044 ceph-mon[32404]: pgmap v8117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:44 smithi116 ceph-mon[37097]: pgmap v8117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:45.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:45.317+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:42:45.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:45.317+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:45.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:42:45.317+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:42:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:44 smithi171 ceph-mon[37789]: pgmap v8117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14510.982573 secs 2024-07-26T12:42:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:47 smithi044 ceph-mon[32404]: pgmap v8118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14510.982573 secs 2024-07-26T12:42:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:47 smithi116 ceph-mon[37097]: pgmap v8118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14510.982573 secs 2024-07-26T12:42:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:47 smithi171 ceph-mon[37789]: pgmap v8118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:49 smithi044 ceph-mon[32404]: pgmap v8119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:49 smithi116 ceph-mon[37097]: pgmap v8119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:49 smithi171 ceph-mon[37789]: pgmap v8119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:51 smithi044 ceph-mon[32404]: pgmap v8120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:51 smithi116 ceph-mon[37097]: pgmap v8120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:51 smithi171 ceph-mon[37789]: pgmap v8120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:42:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14515.982721 secs 2024-07-26T12:42:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14515.982721 secs 2024-07-26T12:42:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14515.982721 secs 2024-07-26T12:42:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:53 smithi171 ceph-mon[37789]: pgmap v8121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:53 smithi044 ceph-mon[32404]: pgmap v8121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:53 smithi116 ceph-mon[37097]: pgmap v8121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:55 smithi171 ceph-mon[37789]: pgmap v8122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:55 smithi044 ceph-mon[32404]: pgmap v8122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:55 smithi116 ceph-mon[37097]: pgmap v8122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14520.982870 secs 2024-07-26T12:42:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:57 smithi171 ceph-mon[37789]: pgmap v8123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14520.982870 secs 2024-07-26T12:42:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:57 smithi044 ceph-mon[32404]: pgmap v8123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14520.982870 secs 2024-07-26T12:42:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:57 smithi116 ceph-mon[37097]: pgmap v8123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:42:59 smithi171 ceph-mon[37789]: pgmap v8124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:42:59 smithi044 ceph-mon[32404]: pgmap v8124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:42:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:42:59 smithi116 ceph-mon[37097]: pgmap v8124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:00.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:00.318+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:43:00.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:00.318+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:00.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:00.318+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:43:01.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:01 smithi044 ceph-mon[32404]: pgmap v8125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:01.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:01.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.316 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.317 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:01 smithi171 ceph-mon[37789]: pgmap v8125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:01 smithi116 ceph-mon[37097]: pgmap v8125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14525.982951 secs 2024-07-26T12:43:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14525.982951 secs 2024-07-26T12:43:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14525.982951 secs 2024-07-26T12:43:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:03 smithi171 ceph-mon[37789]: pgmap v8126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:03 smithi044 ceph-mon[32404]: pgmap v8126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:03.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:03 smithi116 ceph-mon[37097]: pgmap v8126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:05.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:05.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:05 smithi044 ceph-mon[32404]: pgmap v8127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:05 smithi044 ceph-mon[32404]: pgmap v8128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:05 smithi044 ceph-mon[32404]: pgmap v8129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:06.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:05 smithi116 ceph-mon[37097]: pgmap v8127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:05 smithi116 ceph-mon[37097]: pgmap v8128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:05 smithi116 ceph-mon[37097]: pgmap v8129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:05 smithi171 ceph-mon[37789]: pgmap v8127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:05 smithi171 ceph-mon[37789]: pgmap v8128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:05 smithi171 ceph-mon[37789]: pgmap v8129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14530.983089 secs 2024-07-26T12:43:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14530.983089 secs 2024-07-26T12:43:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14530.983089 secs 2024-07-26T12:43:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:07 smithi044 ceph-mon[32404]: pgmap v8130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:07 smithi116 ceph-mon[37097]: pgmap v8130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:07 smithi171 ceph-mon[37789]: pgmap v8130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:43:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:09 smithi171 ceph-mon[37789]: pgmap v8131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:43:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:09 smithi044 ceph-mon[32404]: pgmap v8131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:43:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:09 smithi116 ceph-mon[37097]: pgmap v8131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:43:12.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14535.983220 secs 2024-07-26T12:43:12.075 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:11 smithi116 ceph-mon[37097]: pgmap v8132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:43:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14535.983220 secs 2024-07-26T12:43:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:11 smithi171 ceph-mon[37789]: pgmap v8132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:43:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14535.983220 secs 2024-07-26T12:43:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:11 smithi044 ceph-mon[32404]: pgmap v8132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:43:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:13 smithi171 ceph-mon[37789]: pgmap v8133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:43:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:13 smithi044 ceph-mon[32404]: pgmap v8133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:43:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:13 smithi116 ceph-mon[37097]: pgmap v8133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:43:15.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:14 smithi044 ceph-mon[32404]: pgmap v8134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:43:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:14 smithi116 ceph-mon[37097]: pgmap v8134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:43:15.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:15.323+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:43:15.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:15.323+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:15.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:15.323+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:43:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:14 smithi171 ceph-mon[37789]: pgmap v8134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:43:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14540.983378 secs 2024-07-26T12:43:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14540.983378 secs 2024-07-26T12:43:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14540.983378 secs 2024-07-26T12:43:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:17 smithi044 ceph-mon[32404]: pgmap v8135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:17 smithi116 ceph-mon[37097]: pgmap v8135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:17 smithi171 ceph-mon[37789]: pgmap v8135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:19 smithi171 ceph-mon[37789]: pgmap v8136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:20.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:19 smithi044 ceph-mon[32404]: pgmap v8136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:19 smithi116 ceph-mon[37097]: pgmap v8136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14545.983538 secs 2024-07-26T12:43:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:21 smithi171 ceph-mon[37789]: pgmap v8137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14545.983538 secs 2024-07-26T12:43:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:21 smithi044 ceph-mon[32404]: pgmap v8137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14545.983538 secs 2024-07-26T12:43:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:21 smithi116 ceph-mon[37097]: pgmap v8137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:23 smithi171 ceph-mon[37789]: pgmap v8138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:23 smithi044 ceph-mon[32404]: pgmap v8138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:23 smithi116 ceph-mon[37097]: pgmap v8138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:24 smithi171 ceph-mon[37789]: pgmap v8139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:24 smithi044 ceph-mon[32404]: pgmap v8139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:24 smithi116 ceph-mon[37097]: pgmap v8139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14550.983779 secs 2024-07-26T12:43:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14550.983779 secs 2024-07-26T12:43:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14550.983779 secs 2024-07-26T12:43:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:27 smithi044 ceph-mon[32404]: pgmap v8140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:27 smithi116 ceph-mon[37097]: pgmap v8140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:27 smithi171 ceph-mon[37789]: pgmap v8140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:29 smithi171 ceph-mon[37789]: pgmap v8141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:29 smithi044 ceph-mon[32404]: pgmap v8141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:29 smithi116 ceph-mon[37097]: pgmap v8141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:30.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:30.325+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:43:30.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:30.325+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:30.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:30.325+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:43:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14555.983916 secs 2024-07-26T12:43:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:31 smithi171 ceph-mon[37789]: pgmap v8142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14555.983916 secs 2024-07-26T12:43:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:31 smithi044 ceph-mon[32404]: pgmap v8142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14555.983916 secs 2024-07-26T12:43:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:31 smithi116 ceph-mon[37097]: pgmap v8142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:32 smithi171 ceph-mon[37789]: pgmap v8143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:32 smithi044 ceph-mon[32404]: pgmap v8143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:32 smithi116 ceph-mon[37097]: pgmap v8143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:35 smithi171 ceph-mon[37789]: pgmap v8144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:35 smithi044 ceph-mon[32404]: pgmap v8144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:35 smithi116 ceph-mon[37097]: pgmap v8144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:37.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14560.984059 secs 2024-07-26T12:43:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14560.984059 secs 2024-07-26T12:43:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14560.984059 secs 2024-07-26T12:43:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:37 smithi171 ceph-mon[37789]: pgmap v8145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:37 smithi044 ceph-mon[32404]: pgmap v8145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:37 smithi116 ceph-mon[37097]: pgmap v8145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:38 smithi171 ceph-mon[37789]: pgmap v8146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:38 smithi044 ceph-mon[32404]: pgmap v8146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:38 smithi116 ceph-mon[37097]: pgmap v8146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14565.984223 secs 2024-07-26T12:43:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:41 smithi171 ceph-mon[37789]: pgmap v8147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14565.984223 secs 2024-07-26T12:43:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:41 smithi044 ceph-mon[32404]: pgmap v8147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14565.984223 secs 2024-07-26T12:43:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:41 smithi116 ceph-mon[37097]: pgmap v8147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:42 smithi171 ceph-mon[37789]: pgmap v8148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:43.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:42 smithi116 ceph-mon[37097]: pgmap v8148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:42 smithi044 ceph-mon[32404]: pgmap v8148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:44 smithi044 ceph-mon[32404]: pgmap v8149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:44 smithi116 ceph-mon[37097]: pgmap v8149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:45.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:45.331+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:43:45.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:45.331+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:43:45.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:43:45.331+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:44 smithi171 ceph-mon[37789]: pgmap v8149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14570.984382 secs 2024-07-26T12:43:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14570.984382 secs 2024-07-26T12:43:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14570.984382 secs 2024-07-26T12:43:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:47 smithi044 ceph-mon[32404]: pgmap v8150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:47 smithi116 ceph-mon[37097]: pgmap v8150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:47 smithi171 ceph-mon[37789]: pgmap v8150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:48 smithi044 ceph-mon[32404]: pgmap v8151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:48 smithi116 ceph-mon[37097]: pgmap v8151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:48 smithi171 ceph-mon[37789]: pgmap v8151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:51.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:50 smithi171 ceph-mon[37789]: pgmap v8152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:50 smithi044 ceph-mon[32404]: pgmap v8152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:50 smithi116 ceph-mon[37097]: pgmap v8152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14575.984533 secs 2024-07-26T12:43:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14575.984533 secs 2024-07-26T12:43:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14575.984533 secs 2024-07-26T12:43:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:52 smithi044 ceph-mon[32404]: pgmap v8153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:52 smithi116 ceph-mon[37097]: pgmap v8153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:52 smithi171 ceph-mon[37789]: pgmap v8153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:54 smithi044 ceph-mon[32404]: pgmap v8154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:54 smithi116 ceph-mon[37097]: pgmap v8154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:54 smithi171 ceph-mon[37789]: pgmap v8154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:43:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14580.984696 secs 2024-07-26T12:43:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14580.984696 secs 2024-07-26T12:43:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14580.984696 secs 2024-07-26T12:43:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:57 smithi044 ceph-mon[32404]: pgmap v8155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:57 smithi116 ceph-mon[37097]: pgmap v8155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:57 smithi171 ceph-mon[37789]: pgmap v8155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:43:58 smithi044 ceph-mon[32404]: pgmap v8156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:43:58 smithi116 ceph-mon[37097]: pgmap v8156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:43:58 smithi171 ceph-mon[37789]: pgmap v8156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:00.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:00.331+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:44:00.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:00.331+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:00.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:00.331+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:44:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:00 smithi044 ceph-mon[32404]: pgmap v8157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:00 smithi116 ceph-mon[37097]: pgmap v8157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:00 smithi171 ceph-mon[37789]: pgmap v8157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14585.984864 secs 2024-07-26T12:44:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14585.984864 secs 2024-07-26T12:44:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14585.984864 secs 2024-07-26T12:44:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:02 smithi044 ceph-mon[32404]: pgmap v8158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:02 smithi116 ceph-mon[37097]: pgmap v8158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:02 smithi171 ceph-mon[37789]: pgmap v8158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:04 smithi044 ceph-mon[32404]: pgmap v8159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:04 smithi116 ceph-mon[37097]: pgmap v8159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:04 smithi171 ceph-mon[37789]: pgmap v8159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14590.984951 secs 2024-07-26T12:44:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:06 smithi044 ceph-mon[32404]: pgmap v8160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:44:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:06 smithi044 ceph-mon[32404]: pgmap v8161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:44:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14590.984951 secs 2024-07-26T12:44:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:06 smithi116 ceph-mon[37097]: pgmap v8160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:44:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:06 smithi116 ceph-mon[37097]: pgmap v8161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:44:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14590.984951 secs 2024-07-26T12:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:06 smithi171 ceph-mon[37789]: pgmap v8160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:06 smithi171 ceph-mon[37789]: pgmap v8161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:44:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:09 smithi044 ceph-mon[32404]: pgmap v8162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:09 smithi116 ceph-mon[37097]: pgmap v8162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:09 smithi171 ceph-mon[37789]: pgmap v8162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:11 smithi044 ceph-mon[32404]: pgmap v8163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:11 smithi116 ceph-mon[37097]: pgmap v8163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:11 smithi171 ceph-mon[37789]: pgmap v8163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14595.985094 secs 2024-07-26T12:44:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14595.985094 secs 2024-07-26T12:44:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14595.985094 secs 2024-07-26T12:44:13.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:13 smithi116 ceph-mon[37097]: pgmap v8164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:13 smithi044 ceph-mon[32404]: pgmap v8164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:13 smithi171 ceph-mon[37789]: pgmap v8164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:15.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:15.332+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:44:15.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:15.332+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:44:15.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:15.332+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:15 smithi044 ceph-mon[32404]: pgmap v8165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:15 smithi116 ceph-mon[37097]: pgmap v8165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:15 smithi171 ceph-mon[37789]: pgmap v8165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:44:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:17.312+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:44:17.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:17.312+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:44:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:17.312+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:44:17.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:17.313+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:44:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14600.985225 secs 2024-07-26T12:44:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:17 smithi044 ceph-mon[32404]: pgmap v8166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14600.985225 secs 2024-07-26T12:44:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:17 smithi116 ceph-mon[37097]: pgmap v8166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14600.985225 secs 2024-07-26T12:44:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:17 smithi171 ceph-mon[37789]: pgmap v8166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:19 smithi044 ceph-mon[32404]: pgmap v8167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:19 smithi116 ceph-mon[37097]: pgmap v8167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:19 smithi171 ceph-mon[37789]: pgmap v8167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:21 smithi044 ceph-mon[32404]: pgmap v8168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:21 smithi116 ceph-mon[37097]: pgmap v8168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:21 smithi171 ceph-mon[37789]: pgmap v8168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14605.985380 secs 2024-07-26T12:44:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14605.985380 secs 2024-07-26T12:44:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14605.985380 secs 2024-07-26T12:44:23.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:23 smithi044 ceph-mon[32404]: pgmap v8169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:23 smithi116 ceph-mon[37097]: pgmap v8169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:23 smithi171 ceph-mon[37789]: pgmap v8169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:25 smithi044 ceph-mon[32404]: pgmap v8170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:25 smithi116 ceph-mon[37097]: pgmap v8170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:25 smithi171 ceph-mon[37789]: pgmap v8170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14610.985551 secs 2024-07-26T12:44:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:27 smithi044 ceph-mon[32404]: pgmap v8171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14610.985551 secs 2024-07-26T12:44:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:27 smithi116 ceph-mon[37097]: pgmap v8171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14610.985551 secs 2024-07-26T12:44:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:27 smithi171 ceph-mon[37789]: pgmap v8171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:29 smithi044 ceph-mon[32404]: pgmap v8172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:29 smithi116 ceph-mon[37097]: pgmap v8172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:29 smithi171 ceph-mon[37789]: pgmap v8172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:30.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:30.333+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:44:30.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:30.333+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:30.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:30.333+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:44:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:31 smithi044 ceph-mon[32404]: pgmap v8173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:31 smithi116 ceph-mon[37097]: pgmap v8173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:31 smithi171 ceph-mon[37789]: pgmap v8173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14615.985698 secs 2024-07-26T12:44:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14615.985698 secs 2024-07-26T12:44:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14615.985698 secs 2024-07-26T12:44:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:33 smithi044 ceph-mon[32404]: pgmap v8174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:33 smithi116 ceph-mon[37097]: pgmap v8174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:33 smithi171 ceph-mon[37789]: pgmap v8174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:35.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:35 smithi044 ceph-mon[32404]: pgmap v8175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:35.664 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:35 smithi171 ceph-mon[37789]: pgmap v8175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:35.664 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:35 smithi116 ceph-mon[37097]: pgmap v8175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14620.985841 secs 2024-07-26T12:44:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14620.985841 secs 2024-07-26T12:44:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14620.985841 secs 2024-07-26T12:44:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:37 smithi044 ceph-mon[32404]: pgmap v8176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:37 smithi116 ceph-mon[37097]: pgmap v8176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:37 smithi171 ceph-mon[37789]: pgmap v8176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:39.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:39 smithi044 ceph-mon[32404]: pgmap v8177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:39 smithi116 ceph-mon[37097]: pgmap v8177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:39 smithi171 ceph-mon[37789]: pgmap v8177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:41 smithi044 ceph-mon[32404]: pgmap v8178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14625.985934 secs 2024-07-26T12:44:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:41 smithi116 ceph-mon[37097]: pgmap v8178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14625.985934 secs 2024-07-26T12:44:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:41 smithi171 ceph-mon[37789]: pgmap v8178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14625.985934 secs 2024-07-26T12:44:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:43 smithi044 ceph-mon[32404]: pgmap v8179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:43 smithi116 ceph-mon[37097]: pgmap v8179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:43 smithi171 ceph-mon[37789]: pgmap v8179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:45.339+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:44:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:45.339+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:45.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:44:45.339+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:44:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:45 smithi044 ceph-mon[32404]: pgmap v8180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:45 smithi116 ceph-mon[37097]: pgmap v8180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:45 smithi171 ceph-mon[37789]: pgmap v8180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14630.986083 secs 2024-07-26T12:44:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14630.986083 secs 2024-07-26T12:44:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14630.986083 secs 2024-07-26T12:44:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:47 smithi044 ceph-mon[32404]: pgmap v8181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:47 smithi116 ceph-mon[37097]: pgmap v8181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:47 smithi171 ceph-mon[37789]: pgmap v8181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:49 smithi044 ceph-mon[32404]: pgmap v8182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:49 smithi116 ceph-mon[37097]: pgmap v8182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:49 smithi171 ceph-mon[37789]: pgmap v8182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:51 smithi116 ceph-mon[37097]: pgmap v8183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:51.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14635.986230 secs 2024-07-26T12:44:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:51 smithi171 ceph-mon[37789]: pgmap v8183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14635.986230 secs 2024-07-26T12:44:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:51 smithi044 ceph-mon[32404]: pgmap v8183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14635.986230 secs 2024-07-26T12:44:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:53 smithi171 ceph-mon[37789]: pgmap v8184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:53 smithi044 ceph-mon[32404]: pgmap v8184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:53 smithi116 ceph-mon[37097]: pgmap v8184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:55 smithi171 ceph-mon[37789]: pgmap v8185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:44:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:55 smithi044 ceph-mon[32404]: pgmap v8185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:44:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:55 smithi116 ceph-mon[37097]: pgmap v8185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:44:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14640.986377 secs 2024-07-26T12:44:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14640.986377 secs 2024-07-26T12:44:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14640.986377 secs 2024-07-26T12:44:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:57 smithi171 ceph-mon[37789]: pgmap v8186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:44:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:57 smithi044 ceph-mon[32404]: pgmap v8186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:44:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:57 smithi116 ceph-mon[37097]: pgmap v8186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:44:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:44:59 smithi171 ceph-mon[37789]: pgmap v8187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:44:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:44:59 smithi044 ceph-mon[32404]: pgmap v8187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:44:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:44:59 smithi116 ceph-mon[37097]: pgmap v8187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:45:00.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:00.339+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:45:00.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:00.339+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:45:00.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:00.339+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:01 smithi171 ceph-mon[37789]: pgmap v8188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:45:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14645.986493 secs 2024-07-26T12:45:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:01 smithi044 ceph-mon[32404]: pgmap v8188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:45:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14645.986493 secs 2024-07-26T12:45:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:01 smithi116 ceph-mon[37097]: pgmap v8188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:45:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14645.986493 secs 2024-07-26T12:45:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:03 smithi171 ceph-mon[37789]: pgmap v8189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:45:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:03 smithi044 ceph-mon[32404]: pgmap v8189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:45:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:03 smithi116 ceph-mon[37097]: pgmap v8189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:45:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:05 smithi171 ceph-mon[37789]: pgmap v8190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:45:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:05 smithi044 ceph-mon[32404]: pgmap v8190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:45:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:05 smithi116 ceph-mon[37097]: pgmap v8190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:45:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14650.986683 secs 2024-07-26T12:45:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14650.986683 secs 2024-07-26T12:45:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14650.986683 secs 2024-07-26T12:45:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:07 smithi171 ceph-mon[37789]: pgmap v8191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:07 smithi044 ceph-mon[32404]: pgmap v8191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:07 smithi116 ceph-mon[37097]: pgmap v8191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:08 smithi044 ceph-mon[32404]: pgmap v8192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:45:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:08 smithi044 ceph-mon[32404]: pgmap v8193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:08 smithi116 ceph-mon[37097]: pgmap v8192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:45:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:08 smithi116 ceph-mon[37097]: pgmap v8193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:08 smithi171 ceph-mon[37789]: pgmap v8192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:45:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:08 smithi171 ceph-mon[37789]: pgmap v8193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:10 smithi044 ceph-mon[32404]: pgmap v8194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:10 smithi116 ceph-mon[37097]: pgmap v8194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:10 smithi171 ceph-mon[37789]: pgmap v8194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14655.986776 secs 2024-07-26T12:45:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14655.986776 secs 2024-07-26T12:45:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14655.986776 secs 2024-07-26T12:45:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:12 smithi044 ceph-mon[32404]: pgmap v8195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:12 smithi116 ceph-mon[37097]: pgmap v8195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:12 smithi171 ceph-mon[37789]: pgmap v8195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:14 smithi044 ceph-mon[32404]: pgmap v8196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:14 smithi116 ceph-mon[37097]: pgmap v8196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:14 smithi171 ceph-mon[37789]: pgmap v8196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:15.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:15.344+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:45:15.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:15.344+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:15.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:15.344+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:45:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:16 smithi044 ceph-mon[32404]: pgmap v8197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14660.986940 secs 2024-07-26T12:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:16 smithi116 ceph-mon[37097]: pgmap v8197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14660.986940 secs 2024-07-26T12:45:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:16 smithi171 ceph-mon[37789]: pgmap v8197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:45:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14660.986940 secs 2024-07-26T12:45:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:18 smithi044 ceph-mon[32404]: pgmap v8198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:18 smithi116 ceph-mon[37097]: pgmap v8198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:18 smithi171 ceph-mon[37789]: pgmap v8198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:20 smithi044 ceph-mon[32404]: pgmap v8199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:20 smithi116 ceph-mon[37097]: pgmap v8199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:20 smithi171 ceph-mon[37789]: pgmap v8199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:21.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14665.987104 secs 2024-07-26T12:45:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14665.987104 secs 2024-07-26T12:45:21.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14665.987104 secs 2024-07-26T12:45:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:22 smithi044 ceph-mon[32404]: pgmap v8200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:22 smithi116 ceph-mon[37097]: pgmap v8200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:22 smithi171 ceph-mon[37789]: pgmap v8200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:24 smithi044 ceph-mon[32404]: pgmap v8201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:24 smithi116 ceph-mon[37097]: pgmap v8201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:24 smithi171 ceph-mon[37789]: pgmap v8201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:26 smithi044 ceph-mon[32404]: pgmap v8202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14670.987265 secs 2024-07-26T12:45:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:26 smithi116 ceph-mon[37097]: pgmap v8202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14670.987265 secs 2024-07-26T12:45:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:26 smithi171 ceph-mon[37789]: pgmap v8202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14670.987265 secs 2024-07-26T12:45:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:28 smithi044 ceph-mon[32404]: pgmap v8203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:28 smithi116 ceph-mon[37097]: pgmap v8203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:28 smithi171 ceph-mon[37789]: pgmap v8203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:30.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:30.345+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:45:30.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:30.345+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:30.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:30.345+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:45:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:30 smithi044 ceph-mon[32404]: pgmap v8204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:30 smithi116 ceph-mon[37097]: pgmap v8204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:30 smithi171 ceph-mon[37789]: pgmap v8204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14675.987426 secs 2024-07-26T12:45:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14675.987426 secs 2024-07-26T12:45:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14675.987426 secs 2024-07-26T12:45:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:32 smithi044 ceph-mon[32404]: pgmap v8205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:32 smithi116 ceph-mon[37097]: pgmap v8205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:32 smithi171 ceph-mon[37789]: pgmap v8205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:34 smithi044 ceph-mon[32404]: pgmap v8206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:34 smithi116 ceph-mon[37097]: pgmap v8206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:34 smithi171 ceph-mon[37789]: pgmap v8206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:36 smithi044 ceph-mon[32404]: pgmap v8207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14680.987584 secs 2024-07-26T12:45:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:36 smithi116 ceph-mon[37097]: pgmap v8207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14680.987584 secs 2024-07-26T12:45:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:36 smithi171 ceph-mon[37789]: pgmap v8207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14680.987584 secs 2024-07-26T12:45:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:38 smithi044 ceph-mon[32404]: pgmap v8208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:38 smithi116 ceph-mon[37097]: pgmap v8208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:38 smithi171 ceph-mon[37789]: pgmap v8208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:40 smithi044 ceph-mon[32404]: pgmap v8209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:40 smithi116 ceph-mon[37097]: pgmap v8209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:40 smithi171 ceph-mon[37789]: pgmap v8209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14685.987726 secs 2024-07-26T12:45:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14685.987726 secs 2024-07-26T12:45:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14685.987726 secs 2024-07-26T12:45:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:42 smithi044 ceph-mon[32404]: pgmap v8210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:45:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:42 smithi116 ceph-mon[37097]: pgmap v8210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:45:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:42 smithi171 ceph-mon[37789]: pgmap v8210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:45:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:44 smithi044 ceph-mon[32404]: pgmap v8211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:44 smithi116 ceph-mon[37097]: pgmap v8211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:44 smithi171 ceph-mon[37789]: pgmap v8211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:45.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:45.350+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:45:45.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:45.350+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:45.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:45:45.350+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:45:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:46 smithi044 ceph-mon[32404]: pgmap v8212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14690.987864 secs 2024-07-26T12:45:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:46 smithi116 ceph-mon[37097]: pgmap v8212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14690.987864 secs 2024-07-26T12:45:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:46 smithi171 ceph-mon[37789]: pgmap v8212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14690.987864 secs 2024-07-26T12:45:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:48 smithi044 ceph-mon[32404]: pgmap v8213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:48 smithi116 ceph-mon[37097]: pgmap v8213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:48 smithi171 ceph-mon[37789]: pgmap v8213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:50 smithi044 ceph-mon[32404]: pgmap v8214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:50 smithi116 ceph-mon[37097]: pgmap v8214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:50 smithi171 ceph-mon[37789]: pgmap v8214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:45:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14695.988026 secs 2024-07-26T12:45:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14695.988026 secs 2024-07-26T12:45:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14695.988026 secs 2024-07-26T12:45:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:52 smithi171 ceph-mon[37789]: pgmap v8215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:52 smithi044 ceph-mon[32404]: pgmap v8215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:52 smithi116 ceph-mon[37097]: pgmap v8215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:54 smithi171 ceph-mon[37789]: pgmap v8216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:54 smithi044 ceph-mon[32404]: pgmap v8216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:54 smithi116 ceph-mon[37097]: pgmap v8216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:56 smithi171 ceph-mon[37789]: pgmap v8217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14700.988180 secs 2024-07-26T12:45:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:56 smithi044 ceph-mon[32404]: pgmap v8217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14700.988180 secs 2024-07-26T12:45:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:56 smithi116 ceph-mon[37097]: pgmap v8217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14700.988180 secs 2024-07-26T12:45:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:45:58 smithi171 ceph-mon[37789]: pgmap v8218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:45:58 smithi044 ceph-mon[32404]: pgmap v8218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:45:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:45:58 smithi116 ceph-mon[37097]: pgmap v8218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:00.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:00.350+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:46:00.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:00.350+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:00.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:00.350+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:46:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:00 smithi044 ceph-mon[32404]: pgmap v8219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:00 smithi171 ceph-mon[37789]: pgmap v8219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:00 smithi116 ceph-mon[37097]: pgmap v8219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14705.988324 secs 2024-07-26T12:46:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14705.988324 secs 2024-07-26T12:46:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14705.988324 secs 2024-07-26T12:46:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:02 smithi171 ceph-mon[37789]: pgmap v8220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:02 smithi044 ceph-mon[32404]: pgmap v8220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:02 smithi116 ceph-mon[37097]: pgmap v8220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:04 smithi171 ceph-mon[37789]: pgmap v8221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:46:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:04 smithi044 ceph-mon[32404]: pgmap v8221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:46:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:04 smithi116 ceph-mon[37097]: pgmap v8221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:46:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:06 smithi171 ceph-mon[37789]: pgmap v8222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:46:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14710.988469 secs 2024-07-26T12:46:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:06 smithi044 ceph-mon[32404]: pgmap v8222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:46:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14710.988469 secs 2024-07-26T12:46:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:06 smithi116 ceph-mon[37097]: pgmap v8222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:46:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14710.988469 secs 2024-07-26T12:46:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:08 smithi044 ceph-mon[32404]: pgmap v8223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:46:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:08 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:46:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:08 smithi116 ceph-mon[37097]: pgmap v8223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:46:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:09.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:08 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:46:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:08 smithi171 ceph-mon[37789]: pgmap v8223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:46:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:09.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:08 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:46:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:09 smithi044 ceph-mon[32404]: pgmap v8224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:09 smithi044 ceph-mon[32404]: pgmap v8225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:09 smithi116 ceph-mon[37097]: pgmap v8224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:09 smithi116 ceph-mon[37097]: pgmap v8225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:09 smithi171 ceph-mon[37789]: pgmap v8224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:09 smithi171 ceph-mon[37789]: pgmap v8225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:11 smithi044 ceph-mon[32404]: pgmap v8226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14715.988601 secs 2024-07-26T12:46:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:11 smithi116 ceph-mon[37097]: pgmap v8226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14715.988601 secs 2024-07-26T12:46:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:11 smithi171 ceph-mon[37789]: pgmap v8226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14715.988601 secs 2024-07-26T12:46:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:13 smithi044 ceph-mon[32404]: pgmap v8227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:46:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:13 smithi116 ceph-mon[37097]: pgmap v8227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:46:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:13 smithi171 ceph-mon[37789]: pgmap v8227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:46:15.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:15.356+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:46:15.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:15.356+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:15.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:15.356+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:46:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:15 smithi044 ceph-mon[32404]: pgmap v8228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:46:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:15 smithi116 ceph-mon[37097]: pgmap v8228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:46:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:15 smithi171 ceph-mon[37789]: pgmap v8228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:46:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14720.988717 secs 2024-07-26T12:46:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14720.988717 secs 2024-07-26T12:46:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14720.988717 secs 2024-07-26T12:46:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:17 smithi044 ceph-mon[32404]: pgmap v8229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:46:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:17 smithi116 ceph-mon[37097]: pgmap v8229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:46:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:17 smithi171 ceph-mon[37789]: pgmap v8229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:46:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:19 smithi044 ceph-mon[32404]: pgmap v8230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:19 smithi116 ceph-mon[37097]: pgmap v8230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:19 smithi171 ceph-mon[37789]: pgmap v8230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:21 smithi044 ceph-mon[32404]: pgmap v8231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14725.988827 secs 2024-07-26T12:46:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:21 smithi116 ceph-mon[37097]: pgmap v8231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14725.988827 secs 2024-07-26T12:46:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:21 smithi171 ceph-mon[37789]: pgmap v8231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14725.988827 secs 2024-07-26T12:46:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:23 smithi044 ceph-mon[32404]: pgmap v8232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:23 smithi116 ceph-mon[37097]: pgmap v8232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:23 smithi171 ceph-mon[37789]: pgmap v8232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:25 smithi044 ceph-mon[32404]: pgmap v8233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:25 smithi116 ceph-mon[37097]: pgmap v8233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:25 smithi171 ceph-mon[37789]: pgmap v8233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14730.988960 secs 2024-07-26T12:46:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14730.988960 secs 2024-07-26T12:46:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14730.988960 secs 2024-07-26T12:46:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:27 smithi044 ceph-mon[32404]: pgmap v8234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:27 smithi116 ceph-mon[37097]: pgmap v8234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:27 smithi171 ceph-mon[37789]: pgmap v8234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:29 smithi044 ceph-mon[32404]: pgmap v8235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:29 smithi116 ceph-mon[37097]: pgmap v8235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:29 smithi171 ceph-mon[37789]: pgmap v8235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:30.358+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:46:30.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:30.358+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:30.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:30.358+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:46:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:31 smithi044 ceph-mon[32404]: pgmap v8236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14735.989089 secs 2024-07-26T12:46:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:31 smithi116 ceph-mon[37097]: pgmap v8236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14735.989089 secs 2024-07-26T12:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:31 smithi171 ceph-mon[37789]: pgmap v8236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14735.989089 secs 2024-07-26T12:46:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:33 smithi044 ceph-mon[32404]: pgmap v8237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:33 smithi116 ceph-mon[37097]: pgmap v8237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:33 smithi171 ceph-mon[37789]: pgmap v8237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:35 smithi044 ceph-mon[32404]: pgmap v8238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:35 smithi116 ceph-mon[37097]: pgmap v8238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:35 smithi171 ceph-mon[37789]: pgmap v8238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:36.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14740.989223 secs 2024-07-26T12:46:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14740.989223 secs 2024-07-26T12:46:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14740.989223 secs 2024-07-26T12:46:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:37 smithi044 ceph-mon[32404]: pgmap v8239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:37 smithi116 ceph-mon[37097]: pgmap v8239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:37 smithi171 ceph-mon[37789]: pgmap v8239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:39 smithi044 ceph-mon[32404]: pgmap v8240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:39 smithi116 ceph-mon[37097]: pgmap v8240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:39 smithi171 ceph-mon[37789]: pgmap v8240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:41 smithi044 ceph-mon[32404]: pgmap v8241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14745.989383 secs 2024-07-26T12:46:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:41 smithi116 ceph-mon[37097]: pgmap v8241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14745.989383 secs 2024-07-26T12:46:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:41 smithi171 ceph-mon[37789]: pgmap v8241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14745.989383 secs 2024-07-26T12:46:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:43 smithi044 ceph-mon[32404]: pgmap v8242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:43 smithi116 ceph-mon[37097]: pgmap v8242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:43 smithi171 ceph-mon[37789]: pgmap v8242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:45.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:45.363+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:46:45.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:45.363+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:45.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:46:45.363+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:46:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:45 smithi044 ceph-mon[32404]: pgmap v8243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:45 smithi116 ceph-mon[37097]: pgmap v8243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:45 smithi171 ceph-mon[37789]: pgmap v8243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:47.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14750.989538 secs 2024-07-26T12:46:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14750.989538 secs 2024-07-26T12:46:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14750.989538 secs 2024-07-26T12:46:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:47 smithi044 ceph-mon[32404]: pgmap v8244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:47 smithi116 ceph-mon[37097]: pgmap v8244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:47 smithi171 ceph-mon[37789]: pgmap v8244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:49 smithi044 ceph-mon[32404]: pgmap v8245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:49 smithi116 ceph-mon[37097]: pgmap v8245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:49 smithi171 ceph-mon[37789]: pgmap v8245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:51 smithi044 ceph-mon[32404]: pgmap v8246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14755.989699 secs 2024-07-26T12:46:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:51 smithi116 ceph-mon[37097]: pgmap v8246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14755.989699 secs 2024-07-26T12:46:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:51 smithi171 ceph-mon[37789]: pgmap v8246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14755.989699 secs 2024-07-26T12:46:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:53 smithi044 ceph-mon[32404]: pgmap v8247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:53 smithi116 ceph-mon[37097]: pgmap v8247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:53 smithi171 ceph-mon[37789]: pgmap v8247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:55 smithi044 ceph-mon[32404]: pgmap v8248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:55 smithi116 ceph-mon[37097]: pgmap v8248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:55 smithi171 ceph-mon[37789]: pgmap v8248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14760.989861 secs 2024-07-26T12:46:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14760.989861 secs 2024-07-26T12:46:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14760.989861 secs 2024-07-26T12:46:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:57 smithi044 ceph-mon[32404]: pgmap v8249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:57 smithi116 ceph-mon[37097]: pgmap v8249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:46:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:57 smithi171 ceph-mon[37789]: pgmap v8249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:46:59 smithi044 ceph-mon[32404]: pgmap v8250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:47:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:46:59 smithi116 ceph-mon[37097]: pgmap v8250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:47:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:46:59 smithi171 ceph-mon[37789]: pgmap v8250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:47:00.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:00.364+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:47:00.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:00.364+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:47:00.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:00.364+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:01 smithi044 ceph-mon[32404]: pgmap v8251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:47:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14765.989967 secs 2024-07-26T12:47:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:01 smithi116 ceph-mon[37097]: pgmap v8251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:47:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14765.989967 secs 2024-07-26T12:47:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:01 smithi171 ceph-mon[37789]: pgmap v8251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:47:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14765.989967 secs 2024-07-26T12:47:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:03 smithi044 ceph-mon[32404]: pgmap v8252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:47:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:03 smithi116 ceph-mon[37097]: pgmap v8252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:47:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:03 smithi171 ceph-mon[37789]: pgmap v8252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:47:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:05 smithi044 ceph-mon[32404]: pgmap v8253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:47:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:05 smithi116 ceph-mon[37097]: pgmap v8253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:47:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:05 smithi171 ceph-mon[37789]: pgmap v8253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:47:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14770.990104 secs 2024-07-26T12:47:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14770.990104 secs 2024-07-26T12:47:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14770.990104 secs 2024-07-26T12:47:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:07 smithi044 ceph-mon[32404]: pgmap v8254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:07 smithi116 ceph-mon[37097]: pgmap v8254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:07 smithi171 ceph-mon[37789]: pgmap v8254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:47:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:09 smithi044 ceph-mon[32404]: pgmap v8255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:47:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:09 smithi116 ceph-mon[37097]: pgmap v8255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:47:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:09 smithi171 ceph-mon[37789]: pgmap v8255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:47:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:47:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:47:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:11 smithi044 ceph-mon[32404]: pgmap v8256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:47:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:11 smithi044 ceph-mon[32404]: pgmap v8257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14775.990845 secs 2024-07-26T12:47:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:11 smithi116 ceph-mon[37097]: pgmap v8256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:47:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:11 smithi116 ceph-mon[37097]: pgmap v8257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14775.990845 secs 2024-07-26T12:47:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:11 smithi171 ceph-mon[37789]: pgmap v8256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:47:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:11 smithi171 ceph-mon[37789]: pgmap v8257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14775.990845 secs 2024-07-26T12:47:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:13 smithi044 ceph-mon[32404]: pgmap v8258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:13 smithi116 ceph-mon[37097]: pgmap v8258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:13 smithi171 ceph-mon[37789]: pgmap v8258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:15.368+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:47:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:15.368+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:47:15.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:15.368+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:15 smithi044 ceph-mon[32404]: pgmap v8259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:15 smithi116 ceph-mon[37097]: pgmap v8259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:15 smithi171 ceph-mon[37789]: pgmap v8259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14780.990992 secs 2024-07-26T12:47:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14780.990992 secs 2024-07-26T12:47:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14780.990992 secs 2024-07-26T12:47:17.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:17.344+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:47:17.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:17.344+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:47:17.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:17.345+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:47:17.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:17.346+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:47:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:17 smithi044 ceph-mon[32404]: pgmap v8260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:47:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:17 smithi116 ceph-mon[37097]: pgmap v8260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:47:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:17 smithi171 ceph-mon[37789]: pgmap v8260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:47:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:19 smithi044 ceph-mon[32404]: pgmap v8261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:19 smithi116 ceph-mon[37097]: pgmap v8261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:19 smithi171 ceph-mon[37789]: pgmap v8261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:21 smithi116 ceph-mon[37097]: pgmap v8262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14785.991129 secs 2024-07-26T12:47:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:21 smithi171 ceph-mon[37789]: pgmap v8262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:47:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14785.991129 secs 2024-07-26T12:47:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:21 smithi044 ceph-mon[32404]: pgmap v8262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:47:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14785.991129 secs 2024-07-26T12:47:24.066 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:23 smithi044 ceph-mon[32404]: pgmap v8263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:23 smithi171 ceph-mon[37789]: pgmap v8263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:23 smithi116 ceph-mon[37097]: pgmap v8263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:25 smithi171 ceph-mon[37789]: pgmap v8264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:25 smithi044 ceph-mon[32404]: pgmap v8264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:25 smithi116 ceph-mon[37097]: pgmap v8264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14790.991289 secs 2024-07-26T12:47:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14790.991289 secs 2024-07-26T12:47:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14790.991289 secs 2024-07-26T12:47:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:27 smithi171 ceph-mon[37789]: pgmap v8265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:27 smithi044 ceph-mon[32404]: pgmap v8265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:27 smithi116 ceph-mon[37097]: pgmap v8265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:29 smithi171 ceph-mon[37789]: pgmap v8266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:30.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:29 smithi044 ceph-mon[32404]: pgmap v8266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:29 smithi116 ceph-mon[37097]: pgmap v8266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:30.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:30.370+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:47:30.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:30.370+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:30.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:30.370+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:47:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:31 smithi171 ceph-mon[37789]: pgmap v8267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14795.991425 secs 2024-07-26T12:47:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:31 smithi044 ceph-mon[32404]: pgmap v8267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14795.991425 secs 2024-07-26T12:47:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:31 smithi116 ceph-mon[37097]: pgmap v8267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14795.991425 secs 2024-07-26T12:47:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:33 smithi171 ceph-mon[37789]: pgmap v8268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:34.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:33 smithi044 ceph-mon[32404]: pgmap v8268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:33 smithi116 ceph-mon[37097]: pgmap v8268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:35 smithi171 ceph-mon[37789]: pgmap v8269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:35 smithi044 ceph-mon[32404]: pgmap v8269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:35 smithi116 ceph-mon[37097]: pgmap v8269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14800.991574 secs 2024-07-26T12:47:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14800.991574 secs 2024-07-26T12:47:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14800.991574 secs 2024-07-26T12:47:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:37 smithi171 ceph-mon[37789]: pgmap v8270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:37 smithi044 ceph-mon[32404]: pgmap v8270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:37 smithi116 ceph-mon[37097]: pgmap v8270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:39 smithi044 ceph-mon[32404]: pgmap v8271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:39 smithi116 ceph-mon[37097]: pgmap v8271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:39 smithi171 ceph-mon[37789]: pgmap v8271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:41 smithi044 ceph-mon[32404]: pgmap v8272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14805.991727 secs 2024-07-26T12:47:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:41 smithi116 ceph-mon[37097]: pgmap v8272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14805.991727 secs 2024-07-26T12:47:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:41 smithi171 ceph-mon[37789]: pgmap v8272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14805.991727 secs 2024-07-26T12:47:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:42 smithi044 ceph-mon[32404]: pgmap v8273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:42 smithi116 ceph-mon[37097]: pgmap v8273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:42 smithi171 ceph-mon[37789]: pgmap v8273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:44 smithi044 ceph-mon[32404]: pgmap v8274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:45.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:44 smithi116 ceph-mon[37097]: pgmap v8274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:44 smithi171 ceph-mon[37789]: pgmap v8274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:45.371+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:47:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:45.371+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:47:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:47:45.371+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14810.991887 secs 2024-07-26T12:47:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:46 smithi044 ceph-mon[32404]: pgmap v8275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14810.991887 secs 2024-07-26T12:47:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:46 smithi116 ceph-mon[37097]: pgmap v8275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14810.991887 secs 2024-07-26T12:47:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:46 smithi171 ceph-mon[37789]: pgmap v8275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:48 smithi044 ceph-mon[32404]: pgmap v8276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:48 smithi116 ceph-mon[37097]: pgmap v8276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:48 smithi171 ceph-mon[37789]: pgmap v8276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:50 smithi044 ceph-mon[32404]: pgmap v8277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:50 smithi116 ceph-mon[37097]: pgmap v8277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:50 smithi171 ceph-mon[37789]: pgmap v8277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14815.992723 secs 2024-07-26T12:47:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14815.992723 secs 2024-07-26T12:47:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14815.992723 secs 2024-07-26T12:47:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:52 smithi044 ceph-mon[32404]: pgmap v8278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:52 smithi116 ceph-mon[37097]: pgmap v8278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:52 smithi171 ceph-mon[37789]: pgmap v8278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:54 smithi044 ceph-mon[32404]: pgmap v8279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:54 smithi116 ceph-mon[37097]: pgmap v8279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:54 smithi171 ceph-mon[37789]: pgmap v8279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:47:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14820.992866 secs 2024-07-26T12:47:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:56 smithi044 ceph-mon[32404]: pgmap v8280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14820.992866 secs 2024-07-26T12:47:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:56 smithi116 ceph-mon[37097]: pgmap v8280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14820.992866 secs 2024-07-26T12:47:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:56 smithi171 ceph-mon[37789]: pgmap v8280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:47:58 smithi044 ceph-mon[32404]: pgmap v8281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:47:58 smithi116 ceph-mon[37097]: pgmap v8281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:47:58 smithi171 ceph-mon[37789]: pgmap v8281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:00.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:00.372+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:48:00.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:00.372+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:00.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:00.372+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:48:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:00 smithi044 ceph-mon[32404]: pgmap v8282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:00 smithi116 ceph-mon[37097]: pgmap v8282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:00 smithi171 ceph-mon[37789]: pgmap v8282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14825.993003 secs 2024-07-26T12:48:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14825.993003 secs 2024-07-26T12:48:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14825.993003 secs 2024-07-26T12:48:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:02 smithi044 ceph-mon[32404]: pgmap v8283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:02 smithi116 ceph-mon[37097]: pgmap v8283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:02 smithi171 ceph-mon[37789]: pgmap v8283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:04 smithi044 ceph-mon[32404]: pgmap v8284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:04 smithi116 ceph-mon[37097]: pgmap v8284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:04 smithi171 ceph-mon[37789]: pgmap v8284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14830.993138 secs 2024-07-26T12:48:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:06 smithi044 ceph-mon[32404]: pgmap v8285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14830.993138 secs 2024-07-26T12:48:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:06 smithi116 ceph-mon[37097]: pgmap v8285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14830.993138 secs 2024-07-26T12:48:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:06 smithi171 ceph-mon[37789]: pgmap v8285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:08 smithi044 ceph-mon[32404]: pgmap v8286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:08 smithi116 ceph-mon[37097]: pgmap v8286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:08 smithi171 ceph-mon[37789]: pgmap v8286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:10 smithi044 ceph-mon[32404]: pgmap v8287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:10 smithi116 ceph-mon[37097]: pgmap v8287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:11.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:10 smithi171 ceph-mon[37789]: pgmap v8287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14835.993257 secs 2024-07-26T12:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:12 smithi044 ceph-mon[32404]: pgmap v8288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:48:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:12 smithi044 ceph-mon[32404]: pgmap v8289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:48:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14835.993257 secs 2024-07-26T12:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:12 smithi116 ceph-mon[37097]: pgmap v8288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:48:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:12 smithi116 ceph-mon[37097]: pgmap v8289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:48:12.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:48:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14835.993257 secs 2024-07-26T12:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:12 smithi171 ceph-mon[37789]: pgmap v8288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:12 smithi171 ceph-mon[37789]: pgmap v8289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:48:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:48:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:14 smithi044 ceph-mon[32404]: pgmap v8290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:48:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:14 smithi116 ceph-mon[37097]: pgmap v8290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:48:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:14 smithi171 ceph-mon[37789]: pgmap v8290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:48:15.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:15.377+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:48:15.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:15.377+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:15.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:15.377+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:48:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:16 smithi044 ceph-mon[32404]: pgmap v8291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:48:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:16 smithi116 ceph-mon[37097]: pgmap v8291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:48:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:16 smithi171 ceph-mon[37789]: pgmap v8291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:48:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14840.993399 secs 2024-07-26T12:48:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14840.993399 secs 2024-07-26T12:48:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14840.993399 secs 2024-07-26T12:48:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:18 smithi044 ceph-mon[32404]: pgmap v8292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:48:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:18 smithi116 ceph-mon[37097]: pgmap v8292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:48:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:18 smithi171 ceph-mon[37789]: pgmap v8292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:48:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:20 smithi044 ceph-mon[32404]: pgmap v8293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:48:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:20 smithi116 ceph-mon[37097]: pgmap v8293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:48:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:20 smithi171 ceph-mon[37789]: pgmap v8293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14845.993534 secs 2024-07-26T12:48:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:22 smithi044 ceph-mon[32404]: pgmap v8294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14845.993534 secs 2024-07-26T12:48:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:22 smithi116 ceph-mon[37097]: pgmap v8294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14845.993534 secs 2024-07-26T12:48:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:22 smithi171 ceph-mon[37789]: pgmap v8294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:24 smithi044 ceph-mon[32404]: pgmap v8295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:24 smithi116 ceph-mon[37097]: pgmap v8295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:24 smithi171 ceph-mon[37789]: pgmap v8295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:26 smithi044 ceph-mon[32404]: pgmap v8296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:26 smithi116 ceph-mon[37097]: pgmap v8296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:26 smithi171 ceph-mon[37789]: pgmap v8296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:27.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14850.993674 secs 2024-07-26T12:48:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14850.993674 secs 2024-07-26T12:48:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14850.993674 secs 2024-07-26T12:48:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:28 smithi044 ceph-mon[32404]: pgmap v8297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:28 smithi116 ceph-mon[37097]: pgmap v8297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:28 smithi171 ceph-mon[37789]: pgmap v8297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:30.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:30.378+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:48:30.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:30.378+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:48:30.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:30.378+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:30 smithi044 ceph-mon[32404]: pgmap v8298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:30 smithi116 ceph-mon[37097]: pgmap v8298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:30 smithi171 ceph-mon[37789]: pgmap v8298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14855.994022 secs 2024-07-26T12:48:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14855.994022 secs 2024-07-26T12:48:31.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14855.994022 secs 2024-07-26T12:48:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:32 smithi171 ceph-mon[37789]: pgmap v8299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:32 smithi044 ceph-mon[32404]: pgmap v8299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:32 smithi116 ceph-mon[37097]: pgmap v8299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:34 smithi171 ceph-mon[37789]: pgmap v8300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:34 smithi044 ceph-mon[32404]: pgmap v8300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:34 smithi116 ceph-mon[37097]: pgmap v8300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:36 smithi171 ceph-mon[37789]: pgmap v8301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14860.994178 secs 2024-07-26T12:48:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:36 smithi044 ceph-mon[32404]: pgmap v8301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14860.994178 secs 2024-07-26T12:48:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:36 smithi116 ceph-mon[37097]: pgmap v8301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14860.994178 secs 2024-07-26T12:48:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:38 smithi171 ceph-mon[37789]: pgmap v8302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:38 smithi044 ceph-mon[32404]: pgmap v8302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:38 smithi116 ceph-mon[37097]: pgmap v8302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:40 smithi171 ceph-mon[37789]: pgmap v8303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:40 smithi044 ceph-mon[32404]: pgmap v8303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:40 smithi116 ceph-mon[37097]: pgmap v8303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14865.994313 secs 2024-07-26T12:48:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14865.994313 secs 2024-07-26T12:48:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14865.994313 secs 2024-07-26T12:48:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:42 smithi171 ceph-mon[37789]: pgmap v8304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:42 smithi044 ceph-mon[32404]: pgmap v8304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:42.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:42 smithi116 ceph-mon[37097]: pgmap v8304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:44 smithi171 ceph-mon[37789]: pgmap v8305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:44 smithi044 ceph-mon[32404]: pgmap v8305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:44 smithi116 ceph-mon[37097]: pgmap v8305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:45.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:45.384+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:48:45.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:45.384+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:48:45.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:45.384+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:46 smithi171 ceph-mon[37789]: pgmap v8306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14870.994467 secs 2024-07-26T12:48:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:46 smithi044 ceph-mon[32404]: pgmap v8306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14870.994467 secs 2024-07-26T12:48:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:46 smithi116 ceph-mon[37097]: pgmap v8306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14870.994467 secs 2024-07-26T12:48:47.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:47.360+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:48:47.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:47.360+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:48:47.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:47.361+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:48:47.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:48:47.361+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:48:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:48 smithi171 ceph-mon[37789]: pgmap v8307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:48 smithi044 ceph-mon[32404]: pgmap v8307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:48 smithi116 ceph-mon[37097]: pgmap v8307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:50.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:50 smithi044 ceph-mon[32404]: pgmap v8308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:50 smithi116 ceph-mon[37097]: pgmap v8308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:50 smithi171 ceph-mon[37789]: pgmap v8308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14875.994602 secs 2024-07-26T12:48:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14875.994602 secs 2024-07-26T12:48:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14875.994602 secs 2024-07-26T12:48:52.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:52 smithi171 ceph-mon[37789]: pgmap v8309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:52 smithi044 ceph-mon[32404]: pgmap v8309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:52 smithi116 ceph-mon[37097]: pgmap v8309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:54 smithi044 ceph-mon[32404]: pgmap v8310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:54 smithi116 ceph-mon[37097]: pgmap v8310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:54 smithi171 ceph-mon[37789]: pgmap v8310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:56 smithi044 ceph-mon[32404]: pgmap v8311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14880.994747 secs 2024-07-26T12:48:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:56 smithi116 ceph-mon[37097]: pgmap v8311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14880.994747 secs 2024-07-26T12:48:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:56 smithi171 ceph-mon[37789]: pgmap v8311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14880.994747 secs 2024-07-26T12:48:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:48:58 smithi044 ceph-mon[32404]: pgmap v8312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:48:58 smithi116 ceph-mon[37097]: pgmap v8312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:48:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:48:58 smithi171 ceph-mon[37789]: pgmap v8312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:00.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:00.385+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:49:00.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:00.385+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:00.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:00.385+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:49:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:00 smithi044 ceph-mon[32404]: pgmap v8313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:00 smithi116 ceph-mon[37097]: pgmap v8313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:00 smithi171 ceph-mon[37789]: pgmap v8313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14885.994842 secs 2024-07-26T12:49:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14885.994842 secs 2024-07-26T12:49:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14885.994842 secs 2024-07-26T12:49:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:02 smithi044 ceph-mon[32404]: pgmap v8314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:02 smithi116 ceph-mon[37097]: pgmap v8314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:02 smithi171 ceph-mon[37789]: pgmap v8314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:04 smithi044 ceph-mon[32404]: pgmap v8315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:04 smithi116 ceph-mon[37097]: pgmap v8315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:04 smithi171 ceph-mon[37789]: pgmap v8315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:06 smithi044 ceph-mon[32404]: pgmap v8316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14890.994981 secs 2024-07-26T12:49:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:06 smithi116 ceph-mon[37097]: pgmap v8316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14890.994981 secs 2024-07-26T12:49:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:06 smithi171 ceph-mon[37789]: pgmap v8316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14890.994981 secs 2024-07-26T12:49:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:08 smithi044 ceph-mon[32404]: pgmap v8317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:08 smithi116 ceph-mon[37097]: pgmap v8317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:08 smithi171 ceph-mon[37789]: pgmap v8317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:10 smithi044 ceph-mon[32404]: pgmap v8318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:10 smithi116 ceph-mon[37097]: pgmap v8318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:10 smithi171 ceph-mon[37789]: pgmap v8318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14895.995174 secs 2024-07-26T12:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14895.995174 secs 2024-07-26T12:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14895.995174 secs 2024-07-26T12:49:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: pgmap v8319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: pgmap v8319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: pgmap v8319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:13.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: pgmap v8320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: pgmap v8321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:49:13.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:49:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:49:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:49:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: pgmap v8320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: pgmap v8321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:49:13.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:49:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: pgmap v8320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: pgmap v8321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:49:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:49:15.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:15.390+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:49:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:15.390+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:49:15.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:15.390+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:15.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:15 smithi044 ceph-mon[32404]: pgmap v8322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:15 smithi116 ceph-mon[37097]: pgmap v8322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:15.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:15 smithi171 ceph-mon[37789]: pgmap v8322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14900.995277 secs 2024-07-26T12:49:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14900.995277 secs 2024-07-26T12:49:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14900.995277 secs 2024-07-26T12:49:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:17 smithi116 ceph-mon[37097]: pgmap v8323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:49:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:17 smithi171 ceph-mon[37789]: pgmap v8323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:49:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:17 smithi044 ceph-mon[32404]: pgmap v8323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:49:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:19 smithi171 ceph-mon[37789]: pgmap v8324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:19 smithi044 ceph-mon[32404]: pgmap v8324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:19 smithi116 ceph-mon[37097]: pgmap v8324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:21 smithi171 ceph-mon[37789]: pgmap v8325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14905.995433 secs 2024-07-26T12:49:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:21 smithi044 ceph-mon[32404]: pgmap v8325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14905.995433 secs 2024-07-26T12:49:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:21 smithi116 ceph-mon[37097]: pgmap v8325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:49:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14905.995433 secs 2024-07-26T12:49:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:23 smithi171 ceph-mon[37789]: pgmap v8326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:23 smithi044 ceph-mon[32404]: pgmap v8326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:23 smithi116 ceph-mon[37097]: pgmap v8326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:49:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:25 smithi171 ceph-mon[37789]: pgmap v8327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:25 smithi044 ceph-mon[32404]: pgmap v8327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:25 smithi116 ceph-mon[37097]: pgmap v8327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14910.995582 secs 2024-07-26T12:49:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14910.995582 secs 2024-07-26T12:49:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14910.995582 secs 2024-07-26T12:49:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:27 smithi171 ceph-mon[37789]: pgmap v8328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:27 smithi044 ceph-mon[32404]: pgmap v8328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:27 smithi116 ceph-mon[37097]: pgmap v8328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:29 smithi171 ceph-mon[37789]: pgmap v8329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:29 smithi044 ceph-mon[32404]: pgmap v8329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:29 smithi116 ceph-mon[37097]: pgmap v8329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:30.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:30.392+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:49:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:30.392+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:49:30.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:30.392+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:31 smithi171 ceph-mon[37789]: pgmap v8330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14915.995722 secs 2024-07-26T12:49:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:31 smithi044 ceph-mon[32404]: pgmap v8330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14915.995722 secs 2024-07-26T12:49:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:31 smithi116 ceph-mon[37097]: pgmap v8330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14915.995722 secs 2024-07-26T12:49:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:33 smithi044 ceph-mon[32404]: pgmap v8331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:33 smithi116 ceph-mon[37097]: pgmap v8331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:33 smithi171 ceph-mon[37789]: pgmap v8331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:35 smithi116 ceph-mon[37097]: pgmap v8332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:35 smithi171 ceph-mon[37789]: pgmap v8332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:35 smithi044 ceph-mon[32404]: pgmap v8332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14920.995886 secs 2024-07-26T12:49:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14920.995886 secs 2024-07-26T12:49:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14920.995886 secs 2024-07-26T12:49:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:37 smithi171 ceph-mon[37789]: pgmap v8333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:37 smithi044 ceph-mon[32404]: pgmap v8333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:37 smithi116 ceph-mon[37097]: pgmap v8333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:39.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:39 smithi171 ceph-mon[37789]: pgmap v8334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:39 smithi044 ceph-mon[32404]: pgmap v8334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:39 smithi116 ceph-mon[37097]: pgmap v8334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:41 smithi171 ceph-mon[37789]: pgmap v8335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14925.995987 secs 2024-07-26T12:49:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:41 smithi044 ceph-mon[32404]: pgmap v8335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14925.995987 secs 2024-07-26T12:49:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:41 smithi116 ceph-mon[37097]: pgmap v8335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14925.995987 secs 2024-07-26T12:49:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:43 smithi171 ceph-mon[37789]: pgmap v8336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:43 smithi044 ceph-mon[32404]: pgmap v8336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:43 smithi116 ceph-mon[37097]: pgmap v8336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:45.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:45.397+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:49:45.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:45.398+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:49:45.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:49:45.398+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:45 smithi044 ceph-mon[32404]: pgmap v8337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:45 smithi171 ceph-mon[37789]: pgmap v8337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:45 smithi116 ceph-mon[37097]: pgmap v8337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14930.996127 secs 2024-07-26T12:49:46.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14930.996127 secs 2024-07-26T12:49:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14930.996127 secs 2024-07-26T12:49:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:47 smithi171 ceph-mon[37789]: pgmap v8338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:47 smithi044 ceph-mon[32404]: pgmap v8338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:47 smithi116 ceph-mon[37097]: pgmap v8338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:49 smithi171 ceph-mon[37789]: pgmap v8339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:49 smithi044 ceph-mon[32404]: pgmap v8339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:49 smithi116 ceph-mon[37097]: pgmap v8339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:49:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:51 smithi171 ceph-mon[37789]: pgmap v8340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14935.996261 secs 2024-07-26T12:49:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:51 smithi044 ceph-mon[32404]: pgmap v8340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:52.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14935.996261 secs 2024-07-26T12:49:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:51 smithi116 ceph-mon[37097]: pgmap v8340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14935.996261 secs 2024-07-26T12:49:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:53 smithi044 ceph-mon[32404]: pgmap v8341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:53 smithi116 ceph-mon[37097]: pgmap v8341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:53 smithi171 ceph-mon[37789]: pgmap v8341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:49:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:55 smithi044 ceph-mon[32404]: pgmap v8342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:55 smithi116 ceph-mon[37097]: pgmap v8342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:55 smithi171 ceph-mon[37789]: pgmap v8342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:49:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14940.996405 secs 2024-07-26T12:49:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14940.996405 secs 2024-07-26T12:49:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14940.996405 secs 2024-07-26T12:49:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:57 smithi044 ceph-mon[32404]: pgmap v8343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:57 smithi116 ceph-mon[37097]: pgmap v8343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:49:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:57 smithi171 ceph-mon[37789]: pgmap v8343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:49:59 smithi044 ceph-mon[32404]: pgmap v8344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:49:59 smithi116 ceph-mon[37097]: pgmap v8344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:49:59 smithi171 ceph-mon[37789]: pgmap v8344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:00.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:00.399+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:50:00.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:00.400+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:50:00.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:00.400+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:00.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T12:50:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:01 smithi044 ceph-mon[32404]: pgmap v8345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14945.996509 secs 2024-07-26T12:50:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:01 smithi116 ceph-mon[37097]: pgmap v8345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14945.996509 secs 2024-07-26T12:50:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:01 smithi171 ceph-mon[37789]: pgmap v8345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14945.996509 secs 2024-07-26T12:50:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:03 smithi044 ceph-mon[32404]: pgmap v8346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:03 smithi116 ceph-mon[37097]: pgmap v8346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:03 smithi171 ceph-mon[37789]: pgmap v8346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:06.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:05 smithi044 ceph-mon[32404]: pgmap v8347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:05 smithi116 ceph-mon[37097]: pgmap v8347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:05 smithi171 ceph-mon[37789]: pgmap v8347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14950.996678 secs 2024-07-26T12:50:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14950.996678 secs 2024-07-26T12:50:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14950.996678 secs 2024-07-26T12:50:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:07 smithi044 ceph-mon[32404]: pgmap v8348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:07 smithi116 ceph-mon[37097]: pgmap v8348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:07 smithi171 ceph-mon[37789]: pgmap v8348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:09 smithi044 ceph-mon[32404]: pgmap v8349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:09 smithi116 ceph-mon[37097]: pgmap v8349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:09 smithi171 ceph-mon[37789]: pgmap v8349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:11 smithi044 ceph-mon[32404]: pgmap v8350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14955.996840 secs 2024-07-26T12:50:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:11 smithi116 ceph-mon[37097]: pgmap v8350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14955.996840 secs 2024-07-26T12:50:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:11 smithi171 ceph-mon[37789]: pgmap v8350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14955.996840 secs 2024-07-26T12:50:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: pgmap v8351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: pgmap v8351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: pgmap v8351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:14.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:50:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:50:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:50:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:50:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:50:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: pgmap v8352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:50:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: pgmap v8353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:50:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:50:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:50:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:50:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:50:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:50:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: pgmap v8352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:50:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: pgmap v8353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:50:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:50:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T12:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T12:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T12:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: pgmap v8352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:50:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: pgmap v8353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:50:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:15.406+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:15.406+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:50:15.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:15.406+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14960.997005 secs 2024-07-26T12:50:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:16 smithi171 ceph-mon[37789]: pgmap v8354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:50:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14960.997005 secs 2024-07-26T12:50:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:16 smithi044 ceph-mon[32404]: pgmap v8354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:50:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14960.997005 secs 2024-07-26T12:50:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:16 smithi116 ceph-mon[37097]: pgmap v8354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:50:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:17.377+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:50:17.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:17.377+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:50:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:17.377+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:50:17.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:17.377+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:50:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:18 smithi116 ceph-mon[37097]: pgmap v8355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:50:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:18 smithi171 ceph-mon[37789]: pgmap v8355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:50:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:18 smithi044 ceph-mon[32404]: pgmap v8355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:50:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:20 smithi171 ceph-mon[37789]: pgmap v8356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:50:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:20 smithi044 ceph-mon[32404]: pgmap v8356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:50:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:20 smithi116 ceph-mon[37097]: pgmap v8356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:50:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14965.997170 secs 2024-07-26T12:50:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14965.997170 secs 2024-07-26T12:50:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14965.997170 secs 2024-07-26T12:50:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:22 smithi171 ceph-mon[37789]: pgmap v8357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:50:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:22 smithi044 ceph-mon[32404]: pgmap v8357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:50:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:22 smithi116 ceph-mon[37097]: pgmap v8357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:50:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:24 smithi044 ceph-mon[32404]: pgmap v8358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:50:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:24 smithi116 ceph-mon[37097]: pgmap v8358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:50:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:24 smithi171 ceph-mon[37789]: pgmap v8358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:50:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14970.997318 secs 2024-07-26T12:50:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:26 smithi044 ceph-mon[32404]: pgmap v8359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14970.997318 secs 2024-07-26T12:50:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:26 smithi116 ceph-mon[37097]: pgmap v8359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14970.997318 secs 2024-07-26T12:50:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:26 smithi171 ceph-mon[37789]: pgmap v8359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:28 smithi044 ceph-mon[32404]: pgmap v8360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:28 smithi116 ceph-mon[37097]: pgmap v8360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:28 smithi171 ceph-mon[37789]: pgmap v8360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:30.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:30.407+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:50:30.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:30.407+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:30.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:30.407+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:50:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:30 smithi044 ceph-mon[32404]: pgmap v8361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:30 smithi116 ceph-mon[37097]: pgmap v8361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:30 smithi171 ceph-mon[37789]: pgmap v8361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:32.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14975.997470 secs 2024-07-26T12:50:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14975.997470 secs 2024-07-26T12:50:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14975.997470 secs 2024-07-26T12:50:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:32 smithi044 ceph-mon[32404]: pgmap v8362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:32 smithi116 ceph-mon[37097]: pgmap v8362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:32 smithi171 ceph-mon[37789]: pgmap v8362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:34 smithi044 ceph-mon[32404]: pgmap v8363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:34 smithi116 ceph-mon[37097]: pgmap v8363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:34 smithi171 ceph-mon[37789]: pgmap v8363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14980.997657 secs 2024-07-26T12:50:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:36 smithi044 ceph-mon[32404]: pgmap v8364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14980.997657 secs 2024-07-26T12:50:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:36 smithi116 ceph-mon[37097]: pgmap v8364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14980.997657 secs 2024-07-26T12:50:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:36 smithi171 ceph-mon[37789]: pgmap v8364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:37 smithi044 ceph-mon[32404]: pgmap v8365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:37 smithi116 ceph-mon[37097]: pgmap v8365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:37 smithi171 ceph-mon[37789]: pgmap v8365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:40 smithi171 ceph-mon[37789]: pgmap v8366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:40 smithi044 ceph-mon[32404]: pgmap v8366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:40 smithi116 ceph-mon[37097]: pgmap v8366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14985.997817 secs 2024-07-26T12:50:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14985.997817 secs 2024-07-26T12:50:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14985.997817 secs 2024-07-26T12:50:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:42 smithi044 ceph-mon[32404]: pgmap v8367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:42 smithi116 ceph-mon[37097]: pgmap v8367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:42 smithi171 ceph-mon[37789]: pgmap v8367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:44 smithi044 ceph-mon[32404]: pgmap v8368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:44 smithi116 ceph-mon[37097]: pgmap v8368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:44 smithi171 ceph-mon[37789]: pgmap v8368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:45.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:45.413+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:50:45.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:45.413+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:45.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:50:45.413+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:50:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14990.997980 secs 2024-07-26T12:50:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:46 smithi044 ceph-mon[32404]: pgmap v8369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14990.997980 secs 2024-07-26T12:50:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:46 smithi116 ceph-mon[37097]: pgmap v8369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14990.997980 secs 2024-07-26T12:50:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:46 smithi171 ceph-mon[37789]: pgmap v8369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:48 smithi044 ceph-mon[32404]: pgmap v8370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:49.270 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:48 smithi116 ceph-mon[37097]: pgmap v8370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:49.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:48 smithi171 ceph-mon[37789]: pgmap v8370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:50 smithi044 ceph-mon[32404]: pgmap v8371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:50 smithi116 ceph-mon[37097]: pgmap v8371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:50 smithi171 ceph-mon[37789]: pgmap v8371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:50:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 14995.998144 secs 2024-07-26T12:50:52.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 14995.998144 secs 2024-07-26T12:50:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 14995.998144 secs 2024-07-26T12:50:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:52 smithi044 ceph-mon[32404]: pgmap v8372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:52 smithi116 ceph-mon[37097]: pgmap v8372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:52 smithi171 ceph-mon[37789]: pgmap v8372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:55.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:54 smithi171 ceph-mon[37789]: pgmap v8373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:54 smithi044 ceph-mon[32404]: pgmap v8373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:54 smithi116 ceph-mon[37097]: pgmap v8373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15000.998818 secs 2024-07-26T12:50:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:56 smithi044 ceph-mon[32404]: pgmap v8374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15000.998818 secs 2024-07-26T12:50:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:56 smithi116 ceph-mon[37097]: pgmap v8374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15000.998818 secs 2024-07-26T12:50:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:56 smithi171 ceph-mon[37789]: pgmap v8374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:50:57 smithi044 ceph-mon[32404]: pgmap v8375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:50:57 smithi116 ceph-mon[37097]: pgmap v8375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:50:58.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:50:57 smithi171 ceph-mon[37789]: pgmap v8375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:00.414+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:51:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:00.414+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:00.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:00.414+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:51:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:00 smithi171 ceph-mon[37789]: pgmap v8376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:00 smithi044 ceph-mon[32404]: pgmap v8376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:00 smithi116 ceph-mon[37097]: pgmap v8376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15005.998976 secs 2024-07-26T12:51:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15005.998976 secs 2024-07-26T12:51:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15005.998976 secs 2024-07-26T12:51:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:02 smithi044 ceph-mon[32404]: pgmap v8377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:02 smithi116 ceph-mon[37097]: pgmap v8377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:02 smithi171 ceph-mon[37789]: pgmap v8377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:04 smithi044 ceph-mon[32404]: pgmap v8378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:04 smithi116 ceph-mon[37097]: pgmap v8378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:04 smithi171 ceph-mon[37789]: pgmap v8378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15010.999116 secs 2024-07-26T12:51:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:06 smithi044 ceph-mon[32404]: pgmap v8379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15010.999116 secs 2024-07-26T12:51:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:06 smithi116 ceph-mon[37097]: pgmap v8379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15010.999116 secs 2024-07-26T12:51:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:06 smithi171 ceph-mon[37789]: pgmap v8379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:08 smithi044 ceph-mon[32404]: pgmap v8380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:08 smithi116 ceph-mon[37097]: pgmap v8380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:08 smithi171 ceph-mon[37789]: pgmap v8380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:10 smithi044 ceph-mon[32404]: pgmap v8381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:10 smithi116 ceph-mon[37097]: pgmap v8381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:10 smithi171 ceph-mon[37789]: pgmap v8381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:51:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15015.999251 secs 2024-07-26T12:51:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15015.999251 secs 2024-07-26T12:51:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15015.999251 secs 2024-07-26T12:51:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:12 smithi044 ceph-mon[32404]: pgmap v8382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:12 smithi116 ceph-mon[37097]: pgmap v8382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:12 smithi171 ceph-mon[37789]: pgmap v8382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:14 smithi044 ceph-mon[32404]: pgmap v8383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:14 smithi116 ceph-mon[37097]: pgmap v8383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:14 smithi171 ceph-mon[37789]: pgmap v8383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:15.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:15.417+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:51:15.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:15.417+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:15.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:15.417+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:51:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:51:17.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:16 smithi116 ceph-mon[37097]: pgmap v8384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:16 smithi116 ceph-mon[37097]: pgmap v8385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:17.174 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15020.999390 secs 2024-07-26T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:16 smithi044 ceph-mon[32404]: pgmap v8384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:16 smithi044 ceph-mon[32404]: pgmap v8385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15020.999390 secs 2024-07-26T12:51:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:16 smithi171 ceph-mon[37789]: pgmap v8384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:16 smithi171 ceph-mon[37789]: pgmap v8385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15020.999390 secs 2024-07-26T12:51:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:17 smithi044 ceph-mon[32404]: pgmap v8386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:17 smithi116 ceph-mon[37097]: pgmap v8386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:17 smithi171 ceph-mon[37789]: pgmap v8386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:20 smithi044 ceph-mon[32404]: pgmap v8387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:20 smithi116 ceph-mon[37097]: pgmap v8387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:20 smithi171 ceph-mon[37789]: pgmap v8387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:22.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15025.999524 secs 2024-07-26T12:51:22.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:22 smithi044 ceph-mon[32404]: pgmap v8388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:51:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15025.999524 secs 2024-07-26T12:51:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:22 smithi116 ceph-mon[37097]: pgmap v8388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:51:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15025.999524 secs 2024-07-26T12:51:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:22 smithi171 ceph-mon[37789]: pgmap v8388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:51:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:24 smithi044 ceph-mon[32404]: pgmap v8389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:24 smithi116 ceph-mon[37097]: pgmap v8389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:24 smithi171 ceph-mon[37789]: pgmap v8389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:51:26.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:26 smithi171 ceph-mon[37789]: pgmap v8390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:26 smithi044 ceph-mon[32404]: pgmap v8390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:26 smithi116 ceph-mon[37097]: pgmap v8390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:51:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15030.999655 secs 2024-07-26T12:51:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15030.999655 secs 2024-07-26T12:51:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15030.999655 secs 2024-07-26T12:51:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:28 smithi044 ceph-mon[32404]: pgmap v8391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:28 smithi116 ceph-mon[37097]: pgmap v8391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:28 smithi171 ceph-mon[37789]: pgmap v8391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:30.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:30.417+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:51:30.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:30.417+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:30.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:30.417+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:51:30.420 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:30 smithi044 ceph-mon[32404]: pgmap v8392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:30 smithi116 ceph-mon[37097]: pgmap v8392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:30 smithi171 ceph-mon[37789]: pgmap v8392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15035.999753 secs 2024-07-26T12:51:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:32 smithi044 ceph-mon[32404]: pgmap v8393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15035.999753 secs 2024-07-26T12:51:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:32 smithi116 ceph-mon[37097]: pgmap v8393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15035.999753 secs 2024-07-26T12:51:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:32 smithi171 ceph-mon[37789]: pgmap v8393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:34 smithi044 ceph-mon[32404]: pgmap v8394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:34 smithi116 ceph-mon[37097]: pgmap v8394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:34 smithi171 ceph-mon[37789]: pgmap v8394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:36 smithi044 ceph-mon[32404]: pgmap v8395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:36 smithi116 ceph-mon[37097]: pgmap v8395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:36 smithi171 ceph-mon[37789]: pgmap v8395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15040.999887 secs 2024-07-26T12:51:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15040.999887 secs 2024-07-26T12:51:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15040.999887 secs 2024-07-26T12:51:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:38 smithi044 ceph-mon[32404]: pgmap v8396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:38 smithi116 ceph-mon[37097]: pgmap v8396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:38 smithi171 ceph-mon[37789]: pgmap v8396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:40 smithi044 ceph-mon[32404]: pgmap v8397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:40 smithi116 ceph-mon[37097]: pgmap v8397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:40 smithi171 ceph-mon[37789]: pgmap v8397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15046.000024 secs 2024-07-26T12:51:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:42 smithi044 ceph-mon[32404]: pgmap v8398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15046.000024 secs 2024-07-26T12:51:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:42 smithi116 ceph-mon[37097]: pgmap v8398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15046.000024 secs 2024-07-26T12:51:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:42 smithi171 ceph-mon[37789]: pgmap v8398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:44 smithi044 ceph-mon[32404]: pgmap v8399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:44 smithi116 ceph-mon[37097]: pgmap v8399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:44 smithi171 ceph-mon[37789]: pgmap v8399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:45.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:45.423+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:51:45.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:45.423+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:45.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:51:45.423+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:51:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:46 smithi044 ceph-mon[32404]: pgmap v8400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:46 smithi116 ceph-mon[37097]: pgmap v8400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:46 smithi171 ceph-mon[37789]: pgmap v8400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15051.000152 secs 2024-07-26T12:51:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15051.000152 secs 2024-07-26T12:51:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15051.000152 secs 2024-07-26T12:51:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:48 smithi044 ceph-mon[32404]: pgmap v8401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:48 smithi116 ceph-mon[37097]: pgmap v8401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:48 smithi171 ceph-mon[37789]: pgmap v8401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:50.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:50 smithi044 ceph-mon[32404]: pgmap v8402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:50 smithi116 ceph-mon[37097]: pgmap v8402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:50 smithi171 ceph-mon[37789]: pgmap v8402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15056.000289 secs 2024-07-26T12:51:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:52 smithi044 ceph-mon[32404]: pgmap v8403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15056.000289 secs 2024-07-26T12:51:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:52 smithi116 ceph-mon[37097]: pgmap v8403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15056.000289 secs 2024-07-26T12:51:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:52 smithi171 ceph-mon[37789]: pgmap v8403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:54.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:54 smithi044 ceph-mon[32404]: pgmap v8404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:54 smithi116 ceph-mon[37097]: pgmap v8404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:54 smithi171 ceph-mon[37789]: pgmap v8404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:56 smithi044 ceph-mon[32404]: pgmap v8405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:56 smithi116 ceph-mon[37097]: pgmap v8405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:56 smithi171 ceph-mon[37789]: pgmap v8405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15061.000419 secs 2024-07-26T12:51:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15061.000419 secs 2024-07-26T12:51:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15061.000419 secs 2024-07-26T12:51:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:51:58 smithi044 ceph-mon[32404]: pgmap v8406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:51:58 smithi116 ceph-mon[37097]: pgmap v8406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:51:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:51:58 smithi171 ceph-mon[37789]: pgmap v8406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:00.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:00.424+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:52:00.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:00.424+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:00.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:00.424+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:52:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:00 smithi044 ceph-mon[32404]: pgmap v8407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:00 smithi116 ceph-mon[37097]: pgmap v8407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:00 smithi171 ceph-mon[37789]: pgmap v8407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15066.000564 secs 2024-07-26T12:52:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:02 smithi116 ceph-mon[37097]: pgmap v8408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15066.000564 secs 2024-07-26T12:52:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:02 smithi171 ceph-mon[37789]: pgmap v8408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15066.000564 secs 2024-07-26T12:52:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:02 smithi044 ceph-mon[32404]: pgmap v8408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:04 smithi171 ceph-mon[37789]: pgmap v8409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:04 smithi044 ceph-mon[32404]: pgmap v8409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:04 smithi116 ceph-mon[37097]: pgmap v8409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:06 smithi171 ceph-mon[37789]: pgmap v8410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:06.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:06 smithi044 ceph-mon[32404]: pgmap v8410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:06 smithi116 ceph-mon[37097]: pgmap v8410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15071.000699 secs 2024-07-26T12:52:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15071.000699 secs 2024-07-26T12:52:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15071.000699 secs 2024-07-26T12:52:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:08 smithi171 ceph-mon[37789]: pgmap v8411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:08 smithi044 ceph-mon[32404]: pgmap v8411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:08 smithi116 ceph-mon[37097]: pgmap v8411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:10 smithi171 ceph-mon[37789]: pgmap v8412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:10 smithi044 ceph-mon[32404]: pgmap v8412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:10 smithi116 ceph-mon[37097]: pgmap v8412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15076.000849 secs 2024-07-26T12:52:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15076.000849 secs 2024-07-26T12:52:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15076.000849 secs 2024-07-26T12:52:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:12 smithi171 ceph-mon[37789]: pgmap v8413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:12 smithi044 ceph-mon[32404]: pgmap v8413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:12 smithi116 ceph-mon[37097]: pgmap v8413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:14 smithi171 ceph-mon[37789]: pgmap v8414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:14 smithi044 ceph-mon[32404]: pgmap v8414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:14 smithi116 ceph-mon[37097]: pgmap v8414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:15.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:15.429+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:52:15.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:15.429+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:52:15.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:15.429+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:15.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:16 smithi171 ceph-mon[37789]: pgmap v8415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15081.000969 secs 2024-07-26T12:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:16 smithi044 ceph-mon[32404]: pgmap v8415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15081.000969 secs 2024-07-26T12:52:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:16 smithi116 ceph-mon[37097]: pgmap v8415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15081.000969 secs 2024-07-26T12:52:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:17.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:17.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:17.674 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:18 smithi044 ceph-mon[32404]: pgmap v8416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:18 smithi044 ceph-mon[32404]: pgmap v8417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:18 smithi116 ceph-mon[37097]: pgmap v8416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:18 smithi116 ceph-mon[37097]: pgmap v8417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:18 smithi171 ceph-mon[37789]: pgmap v8416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:18.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:18 smithi171 ceph-mon[37789]: pgmap v8417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:20 smithi044 ceph-mon[32404]: pgmap v8418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:20 smithi116 ceph-mon[37097]: pgmap v8418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:20 smithi171 ceph-mon[37789]: pgmap v8418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15086.001148 secs 2024-07-26T12:52:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15086.001148 secs 2024-07-26T12:52:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15086.001148 secs 2024-07-26T12:52:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:22 smithi044 ceph-mon[32404]: pgmap v8419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:22 smithi116 ceph-mon[37097]: pgmap v8419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:22 smithi171 ceph-mon[37789]: pgmap v8419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:24 smithi044 ceph-mon[32404]: pgmap v8420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:24 smithi116 ceph-mon[37097]: pgmap v8420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:24 smithi171 ceph-mon[37789]: pgmap v8420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:26.638 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:26 smithi171 ceph-mon[37789]: pgmap v8421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:26.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15091.001328 secs 2024-07-26T12:52:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:26 smithi044 ceph-mon[32404]: pgmap v8421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15091.001328 secs 2024-07-26T12:52:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:26 smithi116 ceph-mon[37097]: pgmap v8421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15091.001328 secs 2024-07-26T12:52:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:28 smithi044 ceph-mon[32404]: pgmap v8422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:28 smithi116 ceph-mon[37097]: pgmap v8422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:28 smithi171 ceph-mon[37789]: pgmap v8422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:52:30.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:30.429+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:52:30.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:30.429+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:52:30.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:30.429+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:30 smithi044 ceph-mon[32404]: pgmap v8423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:30 smithi116 ceph-mon[37097]: pgmap v8423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:30 smithi171 ceph-mon[37789]: pgmap v8423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15096.001493 secs 2024-07-26T12:52:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15096.001493 secs 2024-07-26T12:52:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15096.001493 secs 2024-07-26T12:52:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:32 smithi044 ceph-mon[32404]: pgmap v8424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:32 smithi116 ceph-mon[37097]: pgmap v8424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:32 smithi171 ceph-mon[37789]: pgmap v8424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:34 smithi044 ceph-mon[32404]: pgmap v8425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:52:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:34 smithi116 ceph-mon[37097]: pgmap v8425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:52:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:34 smithi171 ceph-mon[37789]: pgmap v8425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:52:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:36 smithi044 ceph-mon[32404]: pgmap v8426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15101.001665 secs 2024-07-26T12:52:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:36 smithi116 ceph-mon[37097]: pgmap v8426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15101.001665 secs 2024-07-26T12:52:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:36 smithi171 ceph-mon[37789]: pgmap v8426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15101.001665 secs 2024-07-26T12:52:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:38 smithi044 ceph-mon[32404]: pgmap v8427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:38.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:38 smithi116 ceph-mon[37097]: pgmap v8427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:38 smithi171 ceph-mon[37789]: pgmap v8427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:40 smithi044 ceph-mon[32404]: pgmap v8428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:40 smithi116 ceph-mon[37097]: pgmap v8428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:40 smithi171 ceph-mon[37789]: pgmap v8428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:52:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15106.001809 secs 2024-07-26T12:52:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15106.001809 secs 2024-07-26T12:52:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15106.001809 secs 2024-07-26T12:52:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:42 smithi044 ceph-mon[32404]: pgmap v8429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:42 smithi116 ceph-mon[37097]: pgmap v8429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:42 smithi171 ceph-mon[37789]: pgmap v8429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:44 smithi044 ceph-mon[32404]: pgmap v8430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:44.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:44 smithi116 ceph-mon[37097]: pgmap v8430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:44 smithi171 ceph-mon[37789]: pgmap v8430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:45.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:45.435+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:52:45.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:45.435+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:52:45.437 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:52:45.435+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:46 smithi044 ceph-mon[32404]: pgmap v8431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15111.001987 secs 2024-07-26T12:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:46 smithi116 ceph-mon[37097]: pgmap v8431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15111.001987 secs 2024-07-26T12:52:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:46 smithi171 ceph-mon[37789]: pgmap v8431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15111.001987 secs 2024-07-26T12:52:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:48 smithi044 ceph-mon[32404]: pgmap v8432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:48 smithi116 ceph-mon[37097]: pgmap v8432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:48 smithi171 ceph-mon[37789]: pgmap v8432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:50 smithi044 ceph-mon[32404]: pgmap v8433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:50 smithi116 ceph-mon[37097]: pgmap v8433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:50 smithi171 ceph-mon[37789]: pgmap v8433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15116.002155 secs 2024-07-26T12:52:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15116.002155 secs 2024-07-26T12:52:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15116.002155 secs 2024-07-26T12:52:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:52 smithi044 ceph-mon[32404]: pgmap v8434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:52 smithi116 ceph-mon[37097]: pgmap v8434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:52 smithi171 ceph-mon[37789]: pgmap v8434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:54 smithi044 ceph-mon[32404]: pgmap v8435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:52:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:54 smithi116 ceph-mon[37097]: pgmap v8435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:52:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:54 smithi171 ceph-mon[37789]: pgmap v8435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:52:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:56 smithi044 ceph-mon[32404]: pgmap v8436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15121.002319 secs 2024-07-26T12:52:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:56 smithi116 ceph-mon[37097]: pgmap v8436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15121.002319 secs 2024-07-26T12:52:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:56 smithi171 ceph-mon[37789]: pgmap v8436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15121.002319 secs 2024-07-26T12:52:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:52:58 smithi044 ceph-mon[32404]: pgmap v8437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:52:58 smithi116 ceph-mon[37097]: pgmap v8437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:52:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:52:58 smithi171 ceph-mon[37789]: pgmap v8437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:53:00.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:00.436+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:53:00.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:00.437+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:53:00.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:00.437+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:00 smithi044 ceph-mon[32404]: pgmap v8438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:00 smithi116 ceph-mon[37097]: pgmap v8438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:00 smithi171 ceph-mon[37789]: pgmap v8438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15126.002491 secs 2024-07-26T12:53:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15126.002491 secs 2024-07-26T12:53:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15126.002491 secs 2024-07-26T12:53:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:02 smithi116 ceph-mon[37097]: pgmap v8439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:02 smithi171 ceph-mon[37789]: pgmap v8439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:02 smithi044 ceph-mon[32404]: pgmap v8439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:04 smithi171 ceph-mon[37789]: pgmap v8440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:04 smithi044 ceph-mon[32404]: pgmap v8440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:04 smithi116 ceph-mon[37097]: pgmap v8440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:06 smithi171 ceph-mon[37789]: pgmap v8441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15131.002673 secs 2024-07-26T12:53:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:06 smithi044 ceph-mon[32404]: pgmap v8441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15131.002673 secs 2024-07-26T12:53:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:06 smithi116 ceph-mon[37097]: pgmap v8441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15131.002673 secs 2024-07-26T12:53:08.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:08 smithi044 ceph-mon[32404]: pgmap v8442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:08 smithi171 ceph-mon[37789]: pgmap v8442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:08 smithi116 ceph-mon[37097]: pgmap v8442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:10 smithi171 ceph-mon[37789]: pgmap v8443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:10 smithi044 ceph-mon[32404]: pgmap v8443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:10 smithi116 ceph-mon[37097]: pgmap v8443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15136.002852 secs 2024-07-26T12:53:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15136.002852 secs 2024-07-26T12:53:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15136.002852 secs 2024-07-26T12:53:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:12 smithi171 ceph-mon[37789]: pgmap v8444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:12 smithi044 ceph-mon[32404]: pgmap v8444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:12 smithi116 ceph-mon[37097]: pgmap v8444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:14 smithi171 ceph-mon[37789]: pgmap v8445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:14 smithi044 ceph-mon[32404]: pgmap v8445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:14 smithi116 ceph-mon[37097]: pgmap v8445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:15.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:15.442+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:53:15.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:15.442+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:53:15.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:15.442+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:16 smithi171 ceph-mon[37789]: pgmap v8446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15141.003013 secs 2024-07-26T12:53:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:16 smithi044 ceph-mon[32404]: pgmap v8446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15141.003013 secs 2024-07-26T12:53:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:16 smithi116 ceph-mon[37097]: pgmap v8446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15141.003013 secs 2024-07-26T12:53:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:17.409+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:53:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:17.409+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:53:17.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:17.410+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:53:17.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:17.410+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:53:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:17 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:17 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:17 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:18 smithi044 ceph-mon[32404]: pgmap v8447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:18.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:18 smithi116 ceph-mon[37097]: pgmap v8447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:18 smithi171 ceph-mon[37789]: pgmap v8447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:20 smithi171 ceph-mon[37789]: pgmap v8448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:20 smithi044 ceph-mon[32404]: pgmap v8448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:20 smithi116 ceph-mon[37097]: pgmap v8448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:20.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15146.003114 secs 2024-07-26T12:53:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15146.003114 secs 2024-07-26T12:53:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15146.003114 secs 2024-07-26T12:53:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:22 smithi044 ceph-mon[32404]: pgmap v8449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:23.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:22 smithi044 ceph-mon[32404]: pgmap v8450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:22 smithi044 ceph-mon[32404]: pgmap v8451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T12:53:23.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:22 smithi116 ceph-mon[37097]: pgmap v8449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:22 smithi116 ceph-mon[37097]: pgmap v8450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:22 smithi116 ceph-mon[37097]: pgmap v8451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T12:53:23.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:22 smithi171 ceph-mon[37789]: pgmap v8449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:22 smithi171 ceph-mon[37789]: pgmap v8450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:22 smithi171 ceph-mon[37789]: pgmap v8451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T12:53:23.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:24 smithi044 ceph-mon[32404]: pgmap v8452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:53:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:24 smithi116 ceph-mon[37097]: pgmap v8452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:53:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:24 smithi171 ceph-mon[37789]: pgmap v8452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:53:27.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15151.003260 secs 2024-07-26T12:53:27.140 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:26 smithi171 ceph-mon[37789]: pgmap v8453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:53:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15151.003260 secs 2024-07-26T12:53:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:26 smithi044 ceph-mon[32404]: pgmap v8453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:53:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15151.003260 secs 2024-07-26T12:53:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:26 smithi116 ceph-mon[37097]: pgmap v8453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:53:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:28 smithi044 ceph-mon[32404]: pgmap v8454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:53:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:28 smithi116 ceph-mon[37097]: pgmap v8454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:53:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:28 smithi171 ceph-mon[37789]: pgmap v8454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:53:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:30.442+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:53:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:30.442+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:53:30.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:30.442+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:30 smithi044 ceph-mon[32404]: pgmap v8455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:30 smithi116 ceph-mon[37097]: pgmap v8455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:30 smithi171 ceph-mon[37789]: pgmap v8455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15156.003440 secs 2024-07-26T12:53:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15156.003440 secs 2024-07-26T12:53:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15156.003440 secs 2024-07-26T12:53:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:32 smithi044 ceph-mon[32404]: pgmap v8456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:53:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:32 smithi116 ceph-mon[37097]: pgmap v8456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:53:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:32 smithi171 ceph-mon[37789]: pgmap v8456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:53:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:34 smithi044 ceph-mon[32404]: pgmap v8457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:34 smithi116 ceph-mon[37097]: pgmap v8457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:34 smithi171 ceph-mon[37789]: pgmap v8457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15161.003590 secs 2024-07-26T12:53:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:36 smithi044 ceph-mon[32404]: pgmap v8458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15161.003590 secs 2024-07-26T12:53:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:36 smithi116 ceph-mon[37097]: pgmap v8458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15161.003590 secs 2024-07-26T12:53:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:36 smithi171 ceph-mon[37789]: pgmap v8458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:37 smithi044 ceph-mon[32404]: pgmap v8459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:37 smithi116 ceph-mon[37097]: pgmap v8459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:37 smithi171 ceph-mon[37789]: pgmap v8459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:40 smithi044 ceph-mon[32404]: pgmap v8460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:40 smithi116 ceph-mon[37097]: pgmap v8460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:40 smithi171 ceph-mon[37789]: pgmap v8460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15166.003739 secs 2024-07-26T12:53:42.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15166.003739 secs 2024-07-26T12:53:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15166.003739 secs 2024-07-26T12:53:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:42 smithi044 ceph-mon[32404]: pgmap v8461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:42 smithi116 ceph-mon[37097]: pgmap v8461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:42 smithi171 ceph-mon[37789]: pgmap v8461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:44 smithi044 ceph-mon[32404]: pgmap v8462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:44 smithi116 ceph-mon[37097]: pgmap v8462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:44 smithi171 ceph-mon[37789]: pgmap v8462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:45.448+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:53:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:45.448+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:53:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:53:45.448+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:45 smithi044 ceph-mon[32404]: pgmap v8463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:45 smithi116 ceph-mon[37097]: pgmap v8463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:45 smithi171 ceph-mon[37789]: pgmap v8463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15171.003896 secs 2024-07-26T12:53:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15171.003896 secs 2024-07-26T12:53:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15171.003896 secs 2024-07-26T12:53:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:47 smithi044 ceph-mon[32404]: pgmap v8464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:47 smithi116 ceph-mon[37097]: pgmap v8464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:47 smithi171 ceph-mon[37789]: pgmap v8464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:51.157 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:50 smithi044 ceph-mon[32404]: pgmap v8465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:50 smithi116 ceph-mon[37097]: pgmap v8465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:50 smithi171 ceph-mon[37789]: pgmap v8465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:53:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15176.004062 secs 2024-07-26T12:53:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15176.004062 secs 2024-07-26T12:53:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15176.004062 secs 2024-07-26T12:53:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:52 smithi044 ceph-mon[32404]: pgmap v8466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:52 smithi116 ceph-mon[37097]: pgmap v8466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:52 smithi171 ceph-mon[37789]: pgmap v8466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:54 smithi044 ceph-mon[32404]: pgmap v8467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:54 smithi116 ceph-mon[37097]: pgmap v8467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:54 smithi171 ceph-mon[37789]: pgmap v8467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15181.004218 secs 2024-07-26T12:53:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:56 smithi044 ceph-mon[32404]: pgmap v8468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15181.004218 secs 2024-07-26T12:53:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:56 smithi116 ceph-mon[37097]: pgmap v8468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15181.004218 secs 2024-07-26T12:53:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:56 smithi171 ceph-mon[37789]: pgmap v8468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:58 smithi044 ceph-mon[32404]: pgmap v8469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:58 smithi116 ceph-mon[37097]: pgmap v8469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:53:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:58 smithi171 ceph-mon[37789]: pgmap v8469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:53:59 smithi044 ceph-mon[32404]: pgmap v8470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:53:59 smithi116 ceph-mon[37097]: pgmap v8470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:53:59 smithi171 ceph-mon[37789]: pgmap v8470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:00.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:00.449+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:54:00.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:00.449+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:54:00.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:00.449+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15186.004403 secs 2024-07-26T12:54:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15186.004403 secs 2024-07-26T12:54:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15186.004403 secs 2024-07-26T12:54:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:02 smithi044 ceph-mon[32404]: pgmap v8471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:02 smithi116 ceph-mon[37097]: pgmap v8471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:02 smithi171 ceph-mon[37789]: pgmap v8471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:05.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:04 smithi044 ceph-mon[32404]: pgmap v8472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:04 smithi116 ceph-mon[37097]: pgmap v8472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:04 smithi171 ceph-mon[37789]: pgmap v8472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15191.004551 secs 2024-07-26T12:54:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:06 smithi044 ceph-mon[32404]: pgmap v8473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15191.004551 secs 2024-07-26T12:54:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:06 smithi116 ceph-mon[37097]: pgmap v8473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15191.004551 secs 2024-07-26T12:54:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:06 smithi171 ceph-mon[37789]: pgmap v8473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:08 smithi044 ceph-mon[32404]: pgmap v8474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:08 smithi116 ceph-mon[37097]: pgmap v8474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:08 smithi171 ceph-mon[37789]: pgmap v8474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:10 smithi044 ceph-mon[32404]: pgmap v8475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:10 smithi116 ceph-mon[37097]: pgmap v8475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:10 smithi171 ceph-mon[37789]: pgmap v8475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15196.004730 secs 2024-07-26T12:54:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15196.004730 secs 2024-07-26T12:54:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15196.004730 secs 2024-07-26T12:54:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:12 smithi044 ceph-mon[32404]: pgmap v8476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:12 smithi116 ceph-mon[37097]: pgmap v8476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:12 smithi171 ceph-mon[37789]: pgmap v8476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:13 smithi044 ceph-mon[32404]: pgmap v8477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:13 smithi116 ceph-mon[37097]: pgmap v8477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:13 smithi171 ceph-mon[37789]: pgmap v8477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:15.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:15.455+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:54:15.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:15.455+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:54:15.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:15.455+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15201.004909 secs 2024-07-26T12:54:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:16 smithi044 ceph-mon[32404]: pgmap v8478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15201.004909 secs 2024-07-26T12:54:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:16 smithi116 ceph-mon[37097]: pgmap v8478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15201.004909 secs 2024-07-26T12:54:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:16 smithi171 ceph-mon[37789]: pgmap v8478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:18 smithi044 ceph-mon[32404]: pgmap v8479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:18 smithi116 ceph-mon[37097]: pgmap v8479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:18 smithi171 ceph-mon[37789]: pgmap v8479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:20 smithi044 ceph-mon[32404]: pgmap v8480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:20 smithi116 ceph-mon[37097]: pgmap v8480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:20 smithi171 ceph-mon[37789]: pgmap v8480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:54:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15206.005073 secs 2024-07-26T12:54:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15206.005073 secs 2024-07-26T12:54:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15206.005073 secs 2024-07-26T12:54:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:22 smithi044 ceph-mon[32404]: pgmap v8481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:23.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:22 smithi116 ceph-mon[37097]: pgmap v8481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:22 smithi171 ceph-mon[37789]: pgmap v8481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:24.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:24 smithi044 ceph-mon[32404]: pgmap v8482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:24 smithi044 ceph-mon[32404]: pgmap v8483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:24 smithi116 ceph-mon[37097]: pgmap v8482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:24 smithi116 ceph-mon[37097]: pgmap v8483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:24 smithi171 ceph-mon[37789]: pgmap v8482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:24 smithi171 ceph-mon[37789]: pgmap v8483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:25 smithi044 ceph-mon[32404]: pgmap v8484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:25 smithi116 ceph-mon[37097]: pgmap v8484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:25 smithi171 ceph-mon[37789]: pgmap v8484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:27.237 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15211.005234 secs 2024-07-26T12:54:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15211.005234 secs 2024-07-26T12:54:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15211.005234 secs 2024-07-26T12:54:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:27 smithi044 ceph-mon[32404]: pgmap v8485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:27 smithi116 ceph-mon[37097]: pgmap v8485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:27 smithi171 ceph-mon[37789]: pgmap v8485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:30.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:30.456+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:54:30.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:30.456+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:54:30.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:30.456+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:30.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:30 smithi044 ceph-mon[32404]: pgmap v8486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:30 smithi116 ceph-mon[37097]: pgmap v8486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:30 smithi171 ceph-mon[37789]: pgmap v8486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:54:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15216.005403 secs 2024-07-26T12:54:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:32 smithi044 ceph-mon[32404]: pgmap v8487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15216.005403 secs 2024-07-26T12:54:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:32 smithi116 ceph-mon[37097]: pgmap v8487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15216.005403 secs 2024-07-26T12:54:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:32 smithi171 ceph-mon[37789]: pgmap v8487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:34 smithi044 ceph-mon[32404]: pgmap v8488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:34 smithi116 ceph-mon[37097]: pgmap v8488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:34 smithi171 ceph-mon[37789]: pgmap v8488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:36 smithi044 ceph-mon[32404]: pgmap v8489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:36 smithi116 ceph-mon[37097]: pgmap v8489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:36 smithi171 ceph-mon[37789]: pgmap v8489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15221.005564 secs 2024-07-26T12:54:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15221.005564 secs 2024-07-26T12:54:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15221.005564 secs 2024-07-26T12:54:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:38 smithi044 ceph-mon[32404]: pgmap v8490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:38 smithi116 ceph-mon[37097]: pgmap v8490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:38 smithi171 ceph-mon[37789]: pgmap v8490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:40 smithi044 ceph-mon[32404]: pgmap v8491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:40 smithi116 ceph-mon[37097]: pgmap v8491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:40 smithi171 ceph-mon[37789]: pgmap v8491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15226.005709 secs 2024-07-26T12:54:42.528 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:42 smithi044 ceph-mon[32404]: pgmap v8492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:42.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15226.005709 secs 2024-07-26T12:54:42.529 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:42 smithi116 ceph-mon[37097]: pgmap v8492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15226.005709 secs 2024-07-26T12:54:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:42 smithi171 ceph-mon[37789]: pgmap v8492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:44 smithi044 ceph-mon[32404]: pgmap v8493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:54:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:44 smithi116 ceph-mon[37097]: pgmap v8493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:54:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:44 smithi171 ceph-mon[37789]: pgmap v8493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:54:45.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:45.461+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:54:45.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:45.461+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:54:45.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:54:45.461+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:46 smithi044 ceph-mon[32404]: pgmap v8494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:46 smithi116 ceph-mon[37097]: pgmap v8494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:46 smithi171 ceph-mon[37789]: pgmap v8494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15231.005859 secs 2024-07-26T12:54:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15231.005859 secs 2024-07-26T12:54:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15231.005859 secs 2024-07-26T12:54:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:48 smithi044 ceph-mon[32404]: pgmap v8495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:48 smithi116 ceph-mon[37097]: pgmap v8495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:48 smithi171 ceph-mon[37789]: pgmap v8495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:50 smithi044 ceph-mon[32404]: pgmap v8496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:50 smithi116 ceph-mon[37097]: pgmap v8496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:50 smithi171 ceph-mon[37789]: pgmap v8496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:54:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15236.006014 secs 2024-07-26T12:54:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:52 smithi044 ceph-mon[32404]: pgmap v8497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15236.006014 secs 2024-07-26T12:54:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:52 smithi116 ceph-mon[37097]: pgmap v8497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15236.006014 secs 2024-07-26T12:54:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:52 smithi171 ceph-mon[37789]: pgmap v8497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:54 smithi171 ceph-mon[37789]: pgmap v8498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:54 smithi044 ceph-mon[32404]: pgmap v8498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:54 smithi116 ceph-mon[37097]: pgmap v8498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:56 smithi044 ceph-mon[32404]: pgmap v8499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15241.006167 secs 2024-07-26T12:54:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:56 smithi116 ceph-mon[37097]: pgmap v8499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15241.006167 secs 2024-07-26T12:54:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:56 smithi171 ceph-mon[37789]: pgmap v8499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15241.006167 secs 2024-07-26T12:54:58.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:54:58 smithi171 ceph-mon[37789]: pgmap v8500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:54:58 smithi044 ceph-mon[32404]: pgmap v8500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:54:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:54:58 smithi116 ceph-mon[37097]: pgmap v8500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:00.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:00.463+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:55:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:00.463+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:00.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:00.463+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:55:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:00 smithi044 ceph-mon[32404]: pgmap v8501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:00 smithi116 ceph-mon[37097]: pgmap v8501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:00 smithi171 ceph-mon[37789]: pgmap v8501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15246.006309 secs 2024-07-26T12:55:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15246.006309 secs 2024-07-26T12:55:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15246.006309 secs 2024-07-26T12:55:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:02 smithi044 ceph-mon[32404]: pgmap v8502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:02 smithi116 ceph-mon[37097]: pgmap v8502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:02 smithi171 ceph-mon[37789]: pgmap v8502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:04 smithi044 ceph-mon[32404]: pgmap v8503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:04 smithi116 ceph-mon[37097]: pgmap v8503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:04 smithi171 ceph-mon[37789]: pgmap v8503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:06 smithi044 ceph-mon[32404]: pgmap v8504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15251.006470 secs 2024-07-26T12:55:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:06 smithi116 ceph-mon[37097]: pgmap v8504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15251.006470 secs 2024-07-26T12:55:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:06 smithi171 ceph-mon[37789]: pgmap v8504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15251.006470 secs 2024-07-26T12:55:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:08 smithi044 ceph-mon[32404]: pgmap v8505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:08 smithi116 ceph-mon[37097]: pgmap v8505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:08 smithi171 ceph-mon[37789]: pgmap v8505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:10 smithi044 ceph-mon[32404]: pgmap v8506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:10 smithi116 ceph-mon[37097]: pgmap v8506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:10 smithi171 ceph-mon[37789]: pgmap v8506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15256.006680 secs 2024-07-26T12:55:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15256.006680 secs 2024-07-26T12:55:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15256.006680 secs 2024-07-26T12:55:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:12 smithi044 ceph-mon[32404]: pgmap v8507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:12 smithi116 ceph-mon[37097]: pgmap v8507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:12 smithi171 ceph-mon[37789]: pgmap v8507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:14 smithi044 ceph-mon[32404]: pgmap v8508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:14 smithi116 ceph-mon[37097]: pgmap v8508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:14 smithi171 ceph-mon[37789]: pgmap v8508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:15.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:15.468+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:55:15.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:15.468+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:15.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:15.468+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:55:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:16 smithi044 ceph-mon[32404]: pgmap v8509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15261.006853 secs 2024-07-26T12:55:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:16 smithi116 ceph-mon[37097]: pgmap v8509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15261.006853 secs 2024-07-26T12:55:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:16 smithi171 ceph-mon[37789]: pgmap v8509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15261.006853 secs 2024-07-26T12:55:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:18 smithi044 ceph-mon[32404]: pgmap v8510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:18 smithi116 ceph-mon[37097]: pgmap v8510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:18 smithi171 ceph-mon[37789]: pgmap v8510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:20 smithi044 ceph-mon[32404]: pgmap v8511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:20 smithi116 ceph-mon[37097]: pgmap v8511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:20 smithi171 ceph-mon[37789]: pgmap v8511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15266.007033 secs 2024-07-26T12:55:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15266.007033 secs 2024-07-26T12:55:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15266.007033 secs 2024-07-26T12:55:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:22 smithi044 ceph-mon[32404]: pgmap v8512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:22 smithi116 ceph-mon[37097]: pgmap v8512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:22 smithi171 ceph-mon[37789]: pgmap v8512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:24 smithi044 ceph-mon[32404]: pgmap v8513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:24 smithi116 ceph-mon[37097]: pgmap v8513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:24 smithi171 ceph-mon[37789]: pgmap v8513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:25 smithi044 ceph-mon[32404]: pgmap v8514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:55:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:25 smithi044 ceph-mon[32404]: pgmap v8515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:25 smithi116 ceph-mon[37097]: pgmap v8514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:55:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:25 smithi116 ceph-mon[37097]: pgmap v8515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:25 smithi171 ceph-mon[37789]: pgmap v8514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:55:25.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:25 smithi171 ceph-mon[37789]: pgmap v8515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15271.007165 secs 2024-07-26T12:55:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15271.007165 secs 2024-07-26T12:55:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15271.007165 secs 2024-07-26T12:55:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:27 smithi044 ceph-mon[32404]: pgmap v8516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:27 smithi116 ceph-mon[37097]: pgmap v8516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:27 smithi171 ceph-mon[37789]: pgmap v8516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:29 smithi044 ceph-mon[32404]: pgmap v8517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:29 smithi116 ceph-mon[37097]: pgmap v8517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:29 smithi171 ceph-mon[37789]: pgmap v8517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:30.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:30.469+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:55:30.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:30.469+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:30.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:30.469+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:55:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:31 smithi044 ceph-mon[32404]: pgmap v8518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15276.007300 secs 2024-07-26T12:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:31 smithi116 ceph-mon[37097]: pgmap v8518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15276.007300 secs 2024-07-26T12:55:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:31 smithi171 ceph-mon[37789]: pgmap v8518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:55:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15276.007300 secs 2024-07-26T12:55:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:33 smithi044 ceph-mon[32404]: pgmap v8519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:33 smithi116 ceph-mon[37097]: pgmap v8519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:33 smithi171 ceph-mon[37789]: pgmap v8519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:55:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:35 smithi044 ceph-mon[32404]: pgmap v8520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:35 smithi116 ceph-mon[37097]: pgmap v8520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:35 smithi171 ceph-mon[37789]: pgmap v8520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15281.007392 secs 2024-07-26T12:55:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15281.007392 secs 2024-07-26T12:55:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15281.007392 secs 2024-07-26T12:55:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:37 smithi044 ceph-mon[32404]: pgmap v8521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:37 smithi116 ceph-mon[37097]: pgmap v8521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:37 smithi171 ceph-mon[37789]: pgmap v8521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:39 smithi116 ceph-mon[37097]: pgmap v8522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:39 smithi171 ceph-mon[37789]: pgmap v8522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:39 smithi044 ceph-mon[32404]: pgmap v8522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:41 smithi171 ceph-mon[37789]: pgmap v8523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15286.007542 secs 2024-07-26T12:55:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:41 smithi044 ceph-mon[32404]: pgmap v8523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15286.007542 secs 2024-07-26T12:55:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:41 smithi116 ceph-mon[37097]: pgmap v8523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15286.007542 secs 2024-07-26T12:55:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:43 smithi171 ceph-mon[37789]: pgmap v8524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:43 smithi044 ceph-mon[32404]: pgmap v8524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:43 smithi116 ceph-mon[37097]: pgmap v8524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:45.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:45.474+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:55:45.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:45.474+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:45.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:55:45.474+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:55:45.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:45 smithi171 ceph-mon[37789]: pgmap v8525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:45 smithi044 ceph-mon[32404]: pgmap v8525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:45 smithi116 ceph-mon[37097]: pgmap v8525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15291.007699 secs 2024-07-26T12:55:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15291.007699 secs 2024-07-26T12:55:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15291.007699 secs 2024-07-26T12:55:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:47 smithi171 ceph-mon[37789]: pgmap v8526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:47 smithi044 ceph-mon[32404]: pgmap v8526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:47 smithi116 ceph-mon[37097]: pgmap v8526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:49.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:49 smithi171 ceph-mon[37789]: pgmap v8527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:49 smithi044 ceph-mon[32404]: pgmap v8527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:49 smithi116 ceph-mon[37097]: pgmap v8527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:51.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:51 smithi116 ceph-mon[37097]: pgmap v8528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:51.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15296.007835 secs 2024-07-26T12:55:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:51 smithi171 ceph-mon[37789]: pgmap v8528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15296.007835 secs 2024-07-26T12:55:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:51 smithi044 ceph-mon[32404]: pgmap v8528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15296.007835 secs 2024-07-26T12:55:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:53 smithi171 ceph-mon[37789]: pgmap v8529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:53 smithi044 ceph-mon[32404]: pgmap v8529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:53 smithi116 ceph-mon[37097]: pgmap v8529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:55:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:55 smithi171 ceph-mon[37789]: pgmap v8530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:55 smithi044 ceph-mon[32404]: pgmap v8530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:55 smithi116 ceph-mon[37097]: pgmap v8530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:55:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15301.007999 secs 2024-07-26T12:55:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15301.007999 secs 2024-07-26T12:55:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15301.007999 secs 2024-07-26T12:55:58.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:57 smithi044 ceph-mon[32404]: pgmap v8531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:57 smithi116 ceph-mon[37097]: pgmap v8531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:57 smithi171 ceph-mon[37789]: pgmap v8531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:59.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:55:59 smithi171 ceph-mon[37789]: pgmap v8532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:55:59 smithi044 ceph-mon[32404]: pgmap v8532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:55:59 smithi116 ceph-mon[37097]: pgmap v8532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:00.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:00.475+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:56:00.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:00.475+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:00.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:00.475+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:56:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:01 smithi044 ceph-mon[32404]: pgmap v8533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15306.008099 secs 2024-07-26T12:56:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:01 smithi116 ceph-mon[37097]: pgmap v8533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15306.008099 secs 2024-07-26T12:56:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:01 smithi171 ceph-mon[37789]: pgmap v8533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15306.008099 secs 2024-07-26T12:56:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:03 smithi044 ceph-mon[32404]: pgmap v8534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:03 smithi116 ceph-mon[37097]: pgmap v8534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:03 smithi171 ceph-mon[37789]: pgmap v8534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:05 smithi044 ceph-mon[32404]: pgmap v8535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:05 smithi116 ceph-mon[37097]: pgmap v8535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:05 smithi171 ceph-mon[37789]: pgmap v8535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15311.008247 secs 2024-07-26T12:56:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15311.008247 secs 2024-07-26T12:56:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15311.008247 secs 2024-07-26T12:56:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:07 smithi044 ceph-mon[32404]: pgmap v8536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:07 smithi116 ceph-mon[37097]: pgmap v8536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:07 smithi171 ceph-mon[37789]: pgmap v8536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:09 smithi044 ceph-mon[32404]: pgmap v8537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:09 smithi116 ceph-mon[37097]: pgmap v8537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:10.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:09 smithi171 ceph-mon[37789]: pgmap v8537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:11 smithi044 ceph-mon[32404]: pgmap v8538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15316.008416 secs 2024-07-26T12:56:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:11 smithi116 ceph-mon[37097]: pgmap v8538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:12.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15316.008416 secs 2024-07-26T12:56:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:11 smithi171 ceph-mon[37789]: pgmap v8538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15316.008416 secs 2024-07-26T12:56:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:13 smithi044 ceph-mon[32404]: pgmap v8539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:13 smithi116 ceph-mon[37097]: pgmap v8539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:13 smithi171 ceph-mon[37789]: pgmap v8539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:14.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T12:56:14.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:56:14.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58074040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:56:14.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003970 tx=0x7fcf58074040 comp rx=0 tx=0).stop 2024-07-26T12:56:14.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T12:56:14.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:56:14.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:56:14.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.736+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:56:14.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.737+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:56:14.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:14.739+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58069040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:56:15.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:15.476+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:56:15.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:15.476+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:56:15.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:15.476+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:15 smithi044 ceph-mon[32404]: pgmap v8540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:15 smithi116 ceph-mon[37097]: pgmap v8540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:15 smithi171 ceph-mon[37789]: pgmap v8540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15321.008567 secs 2024-07-26T12:56:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15321.008567 secs 2024-07-26T12:56:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15321.008567 secs 2024-07-26T12:56:17.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:17.442+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:56:17.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:17.442+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:56:17.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:17.443+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:56:17.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:17.443+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:56:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:17 smithi044 ceph-mon[32404]: pgmap v8541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:17 smithi116 ceph-mon[37097]: pgmap v8541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:17 smithi171 ceph-mon[37789]: pgmap v8541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:19 smithi044 ceph-mon[32404]: pgmap v8542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:19 smithi116 ceph-mon[37097]: pgmap v8542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:19 smithi171 ceph-mon[37789]: pgmap v8542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:21 smithi044 ceph-mon[32404]: pgmap v8543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15326.008737 secs 2024-07-26T12:56:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:21 smithi116 ceph-mon[37097]: pgmap v8543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15326.008737 secs 2024-07-26T12:56:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:21 smithi171 ceph-mon[37789]: pgmap v8543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15326.008737 secs 2024-07-26T12:56:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:23 smithi044 ceph-mon[32404]: pgmap v8544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:23 smithi116 ceph-mon[37097]: pgmap v8544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:23 smithi171 ceph-mon[37789]: pgmap v8544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:25 smithi044 ceph-mon[32404]: pgmap v8545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:25 smithi116 ceph-mon[37097]: pgmap v8545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:25 smithi171 ceph-mon[37789]: pgmap v8545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15331.008827 secs 2024-07-26T12:56:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:26 smithi044 ceph-mon[32404]: pgmap v8546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:56:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:26 smithi044 ceph-mon[32404]: pgmap v8547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:27.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:56:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15331.008827 secs 2024-07-26T12:56:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:26 smithi116 ceph-mon[37097]: pgmap v8546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:56:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:26 smithi116 ceph-mon[37097]: pgmap v8547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:27.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:56:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15331.008827 secs 2024-07-26T12:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:26 smithi171 ceph-mon[37789]: pgmap v8546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:26 smithi171 ceph-mon[37789]: pgmap v8547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:56:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:28 smithi044 ceph-mon[32404]: pgmap v8548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:28 smithi116 ceph-mon[37097]: pgmap v8548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:28 smithi171 ceph-mon[37789]: pgmap v8548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:56:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:30.477+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:56:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:30.477+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:30.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:30.477+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:56:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:30 smithi116 ceph-mon[37097]: pgmap v8549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:30 smithi171 ceph-mon[37789]: pgmap v8549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:30 smithi044 ceph-mon[32404]: pgmap v8549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15336.008968 secs 2024-07-26T12:56:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15336.008968 secs 2024-07-26T12:56:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15336.008968 secs 2024-07-26T12:56:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:32 smithi171 ceph-mon[37789]: pgmap v8550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:56:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:32 smithi044 ceph-mon[32404]: pgmap v8550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:56:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:32 smithi116 ceph-mon[37097]: pgmap v8550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:56:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:34 smithi171 ceph-mon[37789]: pgmap v8551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:34 smithi044 ceph-mon[32404]: pgmap v8551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:34 smithi116 ceph-mon[37097]: pgmap v8551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15341.009124 secs 2024-07-26T12:56:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:36 smithi171 ceph-mon[37789]: pgmap v8552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15341.009124 secs 2024-07-26T12:56:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:36 smithi044 ceph-mon[32404]: pgmap v8552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15341.009124 secs 2024-07-26T12:56:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:36 smithi116 ceph-mon[37097]: pgmap v8552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:38 smithi171 ceph-mon[37789]: pgmap v8553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:38 smithi044 ceph-mon[32404]: pgmap v8553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:38 smithi116 ceph-mon[37097]: pgmap v8553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:40 smithi171 ceph-mon[37789]: pgmap v8554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:40 smithi044 ceph-mon[32404]: pgmap v8554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:40 smithi116 ceph-mon[37097]: pgmap v8554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15346.009283 secs 2024-07-26T12:56:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15346.009283 secs 2024-07-26T12:56:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15346.009283 secs 2024-07-26T12:56:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:42 smithi171 ceph-mon[37789]: pgmap v8555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:42 smithi044 ceph-mon[32404]: pgmap v8555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:42 smithi116 ceph-mon[37097]: pgmap v8555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:44 smithi171 ceph-mon[37789]: pgmap v8556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:44 smithi044 ceph-mon[32404]: pgmap v8556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:44 smithi116 ceph-mon[37097]: pgmap v8556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:45.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:45.482+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:56:45.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:45.482+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:45.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:56:45.482+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:56:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15351.009418 secs 2024-07-26T12:56:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:46 smithi044 ceph-mon[32404]: pgmap v8557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15351.009418 secs 2024-07-26T12:56:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:46 smithi171 ceph-mon[37789]: pgmap v8557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15351.009418 secs 2024-07-26T12:56:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:46 smithi116 ceph-mon[37097]: pgmap v8557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:48 smithi044 ceph-mon[32404]: pgmap v8558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:48 smithi116 ceph-mon[37097]: pgmap v8558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:48 smithi171 ceph-mon[37789]: pgmap v8558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:50 smithi044 ceph-mon[32404]: pgmap v8559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:50 smithi116 ceph-mon[37097]: pgmap v8559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:50 smithi171 ceph-mon[37789]: pgmap v8559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:56:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15356.009564 secs 2024-07-26T12:56:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15356.009564 secs 2024-07-26T12:56:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15356.009564 secs 2024-07-26T12:56:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:52 smithi044 ceph-mon[32404]: pgmap v8560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:52 smithi116 ceph-mon[37097]: pgmap v8560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:52 smithi171 ceph-mon[37789]: pgmap v8560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:54 smithi044 ceph-mon[32404]: pgmap v8561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:54 smithi116 ceph-mon[37097]: pgmap v8561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:54 smithi171 ceph-mon[37789]: pgmap v8561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15361.009709 secs 2024-07-26T12:56:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:56 smithi044 ceph-mon[32404]: pgmap v8562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15361.009709 secs 2024-07-26T12:56:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:56 smithi116 ceph-mon[37097]: pgmap v8562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15361.009709 secs 2024-07-26T12:56:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:56 smithi171 ceph-mon[37789]: pgmap v8562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:56:58 smithi044 ceph-mon[32404]: pgmap v8563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:56:58 smithi116 ceph-mon[37097]: pgmap v8563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:56:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:56:58 smithi171 ceph-mon[37789]: pgmap v8563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:00.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:00.484+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:57:00.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:00.484+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:00.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:00.484+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:57:01.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:00 smithi171 ceph-mon[37789]: pgmap v8564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:00 smithi044 ceph-mon[32404]: pgmap v8564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:00 smithi116 ceph-mon[37097]: pgmap v8564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15366.009867 secs 2024-07-26T12:57:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15366.009867 secs 2024-07-26T12:57:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15366.009867 secs 2024-07-26T12:57:02.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:02.451+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:57:02.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:02.451+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:57:02.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:02.452+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:57:02.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:02.452+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:57:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:02 smithi044 ceph-mon[32404]: pgmap v8565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:02 smithi116 ceph-mon[37097]: pgmap v8565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:02 smithi171 ceph-mon[37789]: pgmap v8565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:04 smithi044 ceph-mon[32404]: pgmap v8566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:04 smithi116 ceph-mon[37097]: pgmap v8566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:04 smithi171 ceph-mon[37789]: pgmap v8566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15371.010023 secs 2024-07-26T12:57:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:06 smithi044 ceph-mon[32404]: pgmap v8567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15371.010023 secs 2024-07-26T12:57:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:06 smithi116 ceph-mon[37097]: pgmap v8567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15371.010023 secs 2024-07-26T12:57:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:06 smithi171 ceph-mon[37789]: pgmap v8567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:08 smithi044 ceph-mon[32404]: pgmap v8568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:08 smithi116 ceph-mon[37097]: pgmap v8568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:09.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:08 smithi171 ceph-mon[37789]: pgmap v8568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:10 smithi044 ceph-mon[32404]: pgmap v8569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:10 smithi116 ceph-mon[37097]: pgmap v8569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:10 smithi171 ceph-mon[37789]: pgmap v8569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15376.010161 secs 2024-07-26T12:57:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15376.010161 secs 2024-07-26T12:57:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15376.010161 secs 2024-07-26T12:57:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:12 smithi044 ceph-mon[32404]: pgmap v8570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:12 smithi116 ceph-mon[37097]: pgmap v8570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:12 smithi171 ceph-mon[37789]: pgmap v8570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:15.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:14 smithi044 ceph-mon[32404]: pgmap v8571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:14 smithi116 ceph-mon[37097]: pgmap v8571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:14 smithi171 ceph-mon[37789]: pgmap v8571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:15.489+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:57:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:15.489+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:15.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:15.489+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:57:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15381.010292 secs 2024-07-26T12:57:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:16 smithi044 ceph-mon[32404]: pgmap v8572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15381.010292 secs 2024-07-26T12:57:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:16 smithi116 ceph-mon[37097]: pgmap v8572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15381.010292 secs 2024-07-26T12:57:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:16 smithi171 ceph-mon[37789]: pgmap v8572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:18 smithi044 ceph-mon[32404]: pgmap v8573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:18 smithi116 ceph-mon[37097]: pgmap v8573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:18 smithi171 ceph-mon[37789]: pgmap v8573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:20 smithi044 ceph-mon[32404]: pgmap v8574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:20 smithi116 ceph-mon[37097]: pgmap v8574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:20 smithi171 ceph-mon[37789]: pgmap v8574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:57:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15386.010456 secs 2024-07-26T12:57:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15386.010456 secs 2024-07-26T12:57:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15386.010456 secs 2024-07-26T12:57:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:22 smithi044 ceph-mon[32404]: pgmap v8575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:22 smithi116 ceph-mon[37097]: pgmap v8575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:22 smithi171 ceph-mon[37789]: pgmap v8575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:24.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:23 smithi044 ceph-mon[32404]: pgmap v8576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:23 smithi116 ceph-mon[37097]: pgmap v8576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:23 smithi171 ceph-mon[37789]: pgmap v8576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15391.010597 secs 2024-07-26T12:57:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:26 smithi171 ceph-mon[37789]: pgmap v8577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15391.010597 secs 2024-07-26T12:57:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:26 smithi044 ceph-mon[32404]: pgmap v8577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15391.010597 secs 2024-07-26T12:57:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:26 smithi116 ceph-mon[37097]: pgmap v8577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:57:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:57:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:57:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:28 smithi171 ceph-mon[37789]: pgmap v8578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:28 smithi171 ceph-mon[37789]: pgmap v8579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:28 smithi044 ceph-mon[32404]: pgmap v8578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:28 smithi044 ceph-mon[32404]: pgmap v8579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:28 smithi116 ceph-mon[37097]: pgmap v8578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:28 smithi116 ceph-mon[37097]: pgmap v8579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:30.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:30.490+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:57:30.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:30.490+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:57:30.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:30.490+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:30 smithi171 ceph-mon[37789]: pgmap v8580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:30 smithi044 ceph-mon[32404]: pgmap v8580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:30 smithi116 ceph-mon[37097]: pgmap v8580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15396.010760 secs 2024-07-26T12:57:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15396.010760 secs 2024-07-26T12:57:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15396.010760 secs 2024-07-26T12:57:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:32 smithi044 ceph-mon[32404]: pgmap v8581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:57:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:32 smithi116 ceph-mon[37097]: pgmap v8581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:57:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:32 smithi171 ceph-mon[37789]: pgmap v8581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:57:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:34 smithi044 ceph-mon[32404]: pgmap v8582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:34 smithi116 ceph-mon[37097]: pgmap v8582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:34 smithi171 ceph-mon[37789]: pgmap v8582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:36 smithi044 ceph-mon[32404]: pgmap v8583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15401.010901 secs 2024-07-26T12:57:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:36 smithi116 ceph-mon[37097]: pgmap v8583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15401.010901 secs 2024-07-26T12:57:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:36 smithi171 ceph-mon[37789]: pgmap v8583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:57:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15401.010901 secs 2024-07-26T12:57:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:38 smithi044 ceph-mon[32404]: pgmap v8584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:38 smithi116 ceph-mon[37097]: pgmap v8584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:38 smithi171 ceph-mon[37789]: pgmap v8584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:40 smithi044 ceph-mon[32404]: pgmap v8585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:40 smithi116 ceph-mon[37097]: pgmap v8585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:40 smithi171 ceph-mon[37789]: pgmap v8585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15406.011046 secs 2024-07-26T12:57:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15406.011046 secs 2024-07-26T12:57:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15406.011046 secs 2024-07-26T12:57:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:42 smithi044 ceph-mon[32404]: pgmap v8586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:42 smithi116 ceph-mon[37097]: pgmap v8586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:42 smithi171 ceph-mon[37789]: pgmap v8586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:44 smithi044 ceph-mon[32404]: pgmap v8587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:44 smithi116 ceph-mon[37097]: pgmap v8587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:44 smithi171 ceph-mon[37789]: pgmap v8587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:45.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:45.496+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:57:45.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:45.496+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:57:45.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:57:45.496+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:46 smithi044 ceph-mon[32404]: pgmap v8588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15411.011140 secs 2024-07-26T12:57:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:46 smithi116 ceph-mon[37097]: pgmap v8588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15411.011140 secs 2024-07-26T12:57:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:46 smithi171 ceph-mon[37789]: pgmap v8588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15411.011140 secs 2024-07-26T12:57:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:48 smithi044 ceph-mon[32404]: pgmap v8589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:48 smithi116 ceph-mon[37097]: pgmap v8589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:48 smithi171 ceph-mon[37789]: pgmap v8589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:50 smithi044 ceph-mon[32404]: pgmap v8590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:50 smithi116 ceph-mon[37097]: pgmap v8590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:50 smithi171 ceph-mon[37789]: pgmap v8590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15416.011302 secs 2024-07-26T12:57:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15416.011302 secs 2024-07-26T12:57:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15416.011302 secs 2024-07-26T12:57:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:52 smithi044 ceph-mon[32404]: pgmap v8591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:52 smithi116 ceph-mon[37097]: pgmap v8591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:52 smithi171 ceph-mon[37789]: pgmap v8591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:54 smithi044 ceph-mon[32404]: pgmap v8592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:54 smithi116 ceph-mon[37097]: pgmap v8592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:54 smithi171 ceph-mon[37789]: pgmap v8592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:56 smithi044 ceph-mon[32404]: pgmap v8593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15421.011446 secs 2024-07-26T12:57:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:56 smithi116 ceph-mon[37097]: pgmap v8593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15421.011446 secs 2024-07-26T12:57:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:56 smithi171 ceph-mon[37789]: pgmap v8593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15421.011446 secs 2024-07-26T12:57:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:57:58 smithi044 ceph-mon[32404]: pgmap v8594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:57:58 smithi116 ceph-mon[37097]: pgmap v8594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:57:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:57:58 smithi171 ceph-mon[37789]: pgmap v8594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:00.497+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:58:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:00.497+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:58:00.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:00.497+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:00 smithi044 ceph-mon[32404]: pgmap v8595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:00 smithi116 ceph-mon[37097]: pgmap v8595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:00 smithi171 ceph-mon[37789]: pgmap v8595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15426.011610 secs 2024-07-26T12:58:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15426.011610 secs 2024-07-26T12:58:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15426.011610 secs 2024-07-26T12:58:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:02 smithi044 ceph-mon[32404]: pgmap v8596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:02 smithi116 ceph-mon[37097]: pgmap v8596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:02 smithi171 ceph-mon[37789]: pgmap v8596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:04 smithi044 ceph-mon[32404]: pgmap v8597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:04 smithi116 ceph-mon[37097]: pgmap v8597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:04 smithi171 ceph-mon[37789]: pgmap v8597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:06 smithi044 ceph-mon[32404]: pgmap v8598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15431.011773 secs 2024-07-26T12:58:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:06 smithi116 ceph-mon[37097]: pgmap v8598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15431.011773 secs 2024-07-26T12:58:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:06 smithi171 ceph-mon[37789]: pgmap v8598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15431.011773 secs 2024-07-26T12:58:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:08 smithi044 ceph-mon[32404]: pgmap v8599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:08 smithi116 ceph-mon[37097]: pgmap v8599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:08 smithi171 ceph-mon[37789]: pgmap v8599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:10 smithi044 ceph-mon[32404]: pgmap v8600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:10 smithi116 ceph-mon[37097]: pgmap v8600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:10 smithi171 ceph-mon[37789]: pgmap v8600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15436.011937 secs 2024-07-26T12:58:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15436.011937 secs 2024-07-26T12:58:12.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15436.011937 secs 2024-07-26T12:58:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:12 smithi044 ceph-mon[32404]: pgmap v8601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:12 smithi116 ceph-mon[37097]: pgmap v8601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:12 smithi171 ceph-mon[37789]: pgmap v8601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:14 smithi044 ceph-mon[32404]: pgmap v8602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:14 smithi116 ceph-mon[37097]: pgmap v8602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:14 smithi171 ceph-mon[37789]: pgmap v8602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:15.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:15.503+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:58:15.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:15.503+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:58:15.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:15.503+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:16 smithi171 ceph-mon[37789]: pgmap v8603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:16 smithi044 ceph-mon[32404]: pgmap v8603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:16 smithi116 ceph-mon[37097]: pgmap v8603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15441.012086 secs 2024-07-26T12:58:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15441.012086 secs 2024-07-26T12:58:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15441.012086 secs 2024-07-26T12:58:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:18 smithi044 ceph-mon[32404]: pgmap v8604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:18 smithi116 ceph-mon[37097]: pgmap v8604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:18 smithi171 ceph-mon[37789]: pgmap v8604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:20 smithi044 ceph-mon[32404]: pgmap v8605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:20 smithi116 ceph-mon[37097]: pgmap v8605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:20 smithi171 ceph-mon[37789]: pgmap v8605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15446.012256 secs 2024-07-26T12:58:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:22 smithi044 ceph-mon[32404]: pgmap v8606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15446.012256 secs 2024-07-26T12:58:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:22 smithi116 ceph-mon[37097]: pgmap v8606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15446.012256 secs 2024-07-26T12:58:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:22 smithi171 ceph-mon[37789]: pgmap v8606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:24 smithi044 ceph-mon[32404]: pgmap v8607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:24 smithi116 ceph-mon[37097]: pgmap v8607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:24 smithi171 ceph-mon[37789]: pgmap v8607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:26 smithi044 ceph-mon[32404]: pgmap v8608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:26 smithi116 ceph-mon[37097]: pgmap v8608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:26 smithi171 ceph-mon[37789]: pgmap v8608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15451.012414 secs 2024-07-26T12:58:27.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15451.012414 secs 2024-07-26T12:58:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15451.012414 secs 2024-07-26T12:58:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:28 smithi044 ceph-mon[32404]: pgmap v8609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:28.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:28 smithi116 ceph-mon[37097]: pgmap v8609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:28 smithi171 ceph-mon[37789]: pgmap v8609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:28.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:29 smithi044 ceph-mon[32404]: pgmap v8610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:29 smithi044 ceph-mon[32404]: pgmap v8611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:58:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:29 smithi116 ceph-mon[37097]: pgmap v8610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:29 smithi116 ceph-mon[37097]: pgmap v8611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:29.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:58:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:29 smithi171 ceph-mon[37789]: pgmap v8610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:29 smithi171 ceph-mon[37789]: pgmap v8611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:58:30.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:30.504+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:58:30.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:30.504+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:30.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:30.504+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:58:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:31 smithi171 ceph-mon[37789]: pgmap v8612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:31 smithi044 ceph-mon[32404]: pgmap v8612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:31 smithi116 ceph-mon[37097]: pgmap v8612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15456.012560 secs 2024-07-26T12:58:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15456.012560 secs 2024-07-26T12:58:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15456.012560 secs 2024-07-26T12:58:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:33 smithi171 ceph-mon[37789]: pgmap v8613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:33 smithi044 ceph-mon[32404]: pgmap v8613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:33 smithi116 ceph-mon[37097]: pgmap v8613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:35 smithi171 ceph-mon[37789]: pgmap v8614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:35 smithi044 ceph-mon[32404]: pgmap v8614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:35 smithi116 ceph-mon[37097]: pgmap v8614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:58:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15461.012703 secs 2024-07-26T12:58:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:37 smithi171 ceph-mon[37789]: pgmap v8615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15461.012703 secs 2024-07-26T12:58:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:37 smithi044 ceph-mon[32404]: pgmap v8615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15461.012703 secs 2024-07-26T12:58:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:37 smithi116 ceph-mon[37097]: pgmap v8615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:39.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:39 smithi171 ceph-mon[37789]: pgmap v8616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:58:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:39 smithi044 ceph-mon[32404]: pgmap v8616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:58:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:39 smithi116 ceph-mon[37097]: pgmap v8616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:58:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:41 smithi171 ceph-mon[37789]: pgmap v8617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:41 smithi044 ceph-mon[32404]: pgmap v8617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:41.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:41 smithi116 ceph-mon[37097]: pgmap v8617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15466.012869 secs 2024-07-26T12:58:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15466.012869 secs 2024-07-26T12:58:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15466.012869 secs 2024-07-26T12:58:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:43 smithi044 ceph-mon[32404]: pgmap v8618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:58:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:43 smithi116 ceph-mon[37097]: pgmap v8618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:58:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:43 smithi171 ceph-mon[37789]: pgmap v8618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:58:45.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:45.510+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:58:45.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:45.510+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:58:45.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:58:45.510+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:45 smithi044 ceph-mon[32404]: pgmap v8619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:45 smithi116 ceph-mon[37097]: pgmap v8619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:45 smithi171 ceph-mon[37789]: pgmap v8619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:58:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15471.013029 secs 2024-07-26T12:58:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:47 smithi044 ceph-mon[32404]: pgmap v8620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:58:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15471.013029 secs 2024-07-26T12:58:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:47 smithi116 ceph-mon[37097]: pgmap v8620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:58:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15471.013029 secs 2024-07-26T12:58:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:47 smithi171 ceph-mon[37789]: pgmap v8620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:58:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:49 smithi044 ceph-mon[32404]: pgmap v8621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:58:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:49 smithi116 ceph-mon[37097]: pgmap v8621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:58:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:49 smithi171 ceph-mon[37789]: pgmap v8621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:58:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:51 smithi044 ceph-mon[32404]: pgmap v8622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:51 smithi116 ceph-mon[37097]: pgmap v8622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:51 smithi171 ceph-mon[37789]: pgmap v8622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:52.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15476.013178 secs 2024-07-26T12:58:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15476.013178 secs 2024-07-26T12:58:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15476.013178 secs 2024-07-26T12:58:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:53 smithi044 ceph-mon[32404]: pgmap v8623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:53 smithi116 ceph-mon[37097]: pgmap v8623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:53 smithi171 ceph-mon[37789]: pgmap v8623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:58:55.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:55 smithi044 ceph-mon[32404]: pgmap v8624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:55 smithi116 ceph-mon[37097]: pgmap v8624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:55 smithi171 ceph-mon[37789]: pgmap v8624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:58:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15481.013309 secs 2024-07-26T12:58:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15481.013309 secs 2024-07-26T12:58:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15481.013309 secs 2024-07-26T12:58:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:57 smithi044 ceph-mon[32404]: pgmap v8625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:57 smithi116 ceph-mon[37097]: pgmap v8625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:57 smithi171 ceph-mon[37789]: pgmap v8625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:58:59 smithi044 ceph-mon[32404]: pgmap v8626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:58:59 smithi116 ceph-mon[37097]: pgmap v8626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:58:59 smithi171 ceph-mon[37789]: pgmap v8626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:00.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:00.510+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:59:00.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:00.510+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:00.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:00.510+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:59:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:01 smithi044 ceph-mon[32404]: pgmap v8627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15486.013457 secs 2024-07-26T12:59:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:01 smithi116 ceph-mon[37097]: pgmap v8627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15486.013457 secs 2024-07-26T12:59:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:01 smithi171 ceph-mon[37789]: pgmap v8627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15486.013457 secs 2024-07-26T12:59:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:03 smithi044 ceph-mon[32404]: pgmap v8628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:03 smithi116 ceph-mon[37097]: pgmap v8628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:03.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:03 smithi171 ceph-mon[37789]: pgmap v8628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:05 smithi044 ceph-mon[32404]: pgmap v8629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:05 smithi116 ceph-mon[37097]: pgmap v8629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:05 smithi171 ceph-mon[37789]: pgmap v8629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15491.013594 secs 2024-07-26T12:59:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15491.013594 secs 2024-07-26T12:59:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15491.013594 secs 2024-07-26T12:59:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:07 smithi044 ceph-mon[32404]: pgmap v8630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:07 smithi116 ceph-mon[37097]: pgmap v8630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:07 smithi171 ceph-mon[37789]: pgmap v8630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:09 smithi116 ceph-mon[37097]: pgmap v8631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:09 smithi171 ceph-mon[37789]: pgmap v8631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:09 smithi044 ceph-mon[32404]: pgmap v8631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:11 smithi171 ceph-mon[37789]: pgmap v8632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15496.013744 secs 2024-07-26T12:59:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:11 smithi044 ceph-mon[32404]: pgmap v8632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15496.013744 secs 2024-07-26T12:59:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:11 smithi116 ceph-mon[37097]: pgmap v8632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15496.013744 secs 2024-07-26T12:59:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:13 smithi171 ceph-mon[37789]: pgmap v8633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:13 smithi044 ceph-mon[32404]: pgmap v8633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:13 smithi116 ceph-mon[37097]: pgmap v8633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:15.516+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:59:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:15.516+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:15.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:15.516+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:59:15.818 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:15 smithi044 ceph-mon[32404]: pgmap v8634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:15 smithi171 ceph-mon[37789]: pgmap v8634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:15 smithi116 ceph-mon[37097]: pgmap v8634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15501.013832 secs 2024-07-26T12:59:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15501.013832 secs 2024-07-26T12:59:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15501.013832 secs 2024-07-26T12:59:17.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:17.475+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:59:17.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:17.475+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:59:17.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:17.476+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T12:59:17.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:17.476+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T12:59:17.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:17 smithi044 ceph-mon[32404]: pgmap v8635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:17 smithi171 ceph-mon[37789]: pgmap v8635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:17 smithi116 ceph-mon[37097]: pgmap v8635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:19.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:19 smithi171 ceph-mon[37789]: pgmap v8636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:20.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:19 smithi044 ceph-mon[32404]: pgmap v8636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:19 smithi116 ceph-mon[37097]: pgmap v8636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:21 smithi171 ceph-mon[37789]: pgmap v8637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15506.013977 secs 2024-07-26T12:59:21.890 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:21 smithi044 ceph-mon[32404]: pgmap v8637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:21.890 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15506.013977 secs 2024-07-26T12:59:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:21 smithi116 ceph-mon[37097]: pgmap v8637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15506.013977 secs 2024-07-26T12:59:23.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:23 smithi116 ceph-mon[37097]: pgmap v8638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:23 smithi171 ceph-mon[37789]: pgmap v8638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:24.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:23 smithi044 ceph-mon[32404]: pgmap v8638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:59:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:25 smithi171 ceph-mon[37789]: pgmap v8639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:25 smithi044 ceph-mon[32404]: pgmap v8639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:25 smithi116 ceph-mon[37097]: pgmap v8639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:59:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15511.014128 secs 2024-07-26T12:59:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15511.014128 secs 2024-07-26T12:59:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15511.014128 secs 2024-07-26T12:59:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:27 smithi044 ceph-mon[32404]: pgmap v8640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:27 smithi116 ceph-mon[37097]: pgmap v8640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:27 smithi171 ceph-mon[37789]: pgmap v8640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: pgmap v8641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:30.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:59:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: pgmap v8641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:30.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:30.024 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: pgmap v8641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:59:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:30.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:59:30.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:30.518+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:59:30.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:30.518+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:59:30.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:30.518+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:30 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:59:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:30 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:59:31.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:30 smithi044 ceph-mon[32404]: pgmap v8642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:30 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:59:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:30 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:59:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:30 smithi116 ceph-mon[37097]: pgmap v8642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:30 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T12:59:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:30 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T12:59:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:30 smithi171 ceph-mon[37789]: pgmap v8642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:31.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15516.014256 secs 2024-07-26T12:59:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15516.014256 secs 2024-07-26T12:59:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15516.014256 secs 2024-07-26T12:59:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:32 smithi044 ceph-mon[32404]: pgmap v8643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:32 smithi116 ceph-mon[37097]: pgmap v8643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:32 smithi171 ceph-mon[37789]: pgmap v8643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:34 smithi044 ceph-mon[32404]: pgmap v8644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:59:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:34 smithi116 ceph-mon[37097]: pgmap v8644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:59:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:34 smithi171 ceph-mon[37789]: pgmap v8644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:59:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:36 smithi044 ceph-mon[32404]: pgmap v8645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15521.014419 secs 2024-07-26T12:59:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:36 smithi116 ceph-mon[37097]: pgmap v8645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15521.014419 secs 2024-07-26T12:59:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:36 smithi171 ceph-mon[37789]: pgmap v8645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15521.014419 secs 2024-07-26T12:59:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:38 smithi044 ceph-mon[32404]: pgmap v8646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:59:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:38 smithi116 ceph-mon[37097]: pgmap v8646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:59:39.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:38 smithi171 ceph-mon[37789]: pgmap v8646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:59:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:40 smithi044 ceph-mon[32404]: pgmap v8647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:40 smithi116 ceph-mon[37097]: pgmap v8647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:40 smithi171 ceph-mon[37789]: pgmap v8647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15526.014587 secs 2024-07-26T12:59:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15526.014587 secs 2024-07-26T12:59:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15526.014587 secs 2024-07-26T12:59:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:42 smithi044 ceph-mon[32404]: pgmap v8648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:42 smithi116 ceph-mon[37097]: pgmap v8648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:42 smithi171 ceph-mon[37789]: pgmap v8648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:44 smithi044 ceph-mon[32404]: pgmap v8649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:44 smithi116 ceph-mon[37097]: pgmap v8649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:44 smithi171 ceph-mon[37789]: pgmap v8649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:45.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:45.524+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T12:59:45.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:45.524+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T12:59:45.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T12:59:45.524+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:46 smithi044 ceph-mon[32404]: pgmap v8650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15531.014746 secs 2024-07-26T12:59:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:46 smithi116 ceph-mon[37097]: pgmap v8650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15531.014746 secs 2024-07-26T12:59:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:46 smithi171 ceph-mon[37789]: pgmap v8650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15531.014746 secs 2024-07-26T12:59:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:48 smithi044 ceph-mon[32404]: pgmap v8651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:59:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:48 smithi116 ceph-mon[37097]: pgmap v8651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:59:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:48 smithi171 ceph-mon[37789]: pgmap v8651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:59:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:50 smithi044 ceph-mon[32404]: pgmap v8652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:59:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:50 smithi116 ceph-mon[37097]: pgmap v8652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:59:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:50 smithi171 ceph-mon[37789]: pgmap v8652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:59:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15536.014889 secs 2024-07-26T12:59:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15536.014889 secs 2024-07-26T12:59:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15536.014889 secs 2024-07-26T12:59:52.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:52 smithi044 ceph-mon[32404]: pgmap v8653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:52 smithi116 ceph-mon[37097]: pgmap v8653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:52 smithi171 ceph-mon[37789]: pgmap v8653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:54 smithi044 ceph-mon[32404]: pgmap v8654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:59:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:54 smithi116 ceph-mon[37097]: pgmap v8654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:59:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:54 smithi171 ceph-mon[37789]: pgmap v8654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:59:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:56 smithi044 ceph-mon[32404]: pgmap v8655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15541.015048 secs 2024-07-26T12:59:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:56 smithi116 ceph-mon[37097]: pgmap v8655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15541.015048 secs 2024-07-26T12:59:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:56 smithi171 ceph-mon[37789]: pgmap v8655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15541.015048 secs 2024-07-26T12:59:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 12:59:58 smithi044 ceph-mon[32404]: pgmap v8656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:59:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 12:59:58 smithi116 ceph-mon[37097]: pgmap v8656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:59:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 12:59:58 smithi171 ceph-mon[37789]: pgmap v8656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:00:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:00.524+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:00:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:00.524+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:00:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:00.524+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:00 smithi044 ceph-mon[32404]: pgmap v8657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:00 smithi116 ceph-mon[37097]: pgmap v8657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:00 smithi171 ceph-mon[37789]: pgmap v8657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15546.015192 secs 2024-07-26T13:00:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15546.015192 secs 2024-07-26T13:00:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15546.015192 secs 2024-07-26T13:00:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:02 smithi044 ceph-mon[32404]: pgmap v8658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:02 smithi116 ceph-mon[37097]: pgmap v8658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:02 smithi171 ceph-mon[37789]: pgmap v8658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:04 smithi044 ceph-mon[32404]: pgmap v8659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:04 smithi116 ceph-mon[37097]: pgmap v8659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:04 smithi171 ceph-mon[37789]: pgmap v8659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:06 smithi044 ceph-mon[32404]: pgmap v8660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15551.015333 secs 2024-07-26T13:00:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:06 smithi116 ceph-mon[37097]: pgmap v8660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15551.015333 secs 2024-07-26T13:00:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:06 smithi171 ceph-mon[37789]: pgmap v8660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15551.015333 secs 2024-07-26T13:00:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:08 smithi044 ceph-mon[32404]: pgmap v8661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:08 smithi116 ceph-mon[37097]: pgmap v8661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:08 smithi171 ceph-mon[37789]: pgmap v8661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:10 smithi116 ceph-mon[37097]: pgmap v8662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:10 smithi171 ceph-mon[37789]: pgmap v8662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:10 smithi044 ceph-mon[32404]: pgmap v8662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15556.015481 secs 2024-07-26T13:00:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15556.015481 secs 2024-07-26T13:00:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15556.015481 secs 2024-07-26T13:00:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:12 smithi171 ceph-mon[37789]: pgmap v8663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:12 smithi044 ceph-mon[32404]: pgmap v8663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:12 smithi116 ceph-mon[37097]: pgmap v8663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:15.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:14 smithi044 ceph-mon[32404]: pgmap v8664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:14 smithi171 ceph-mon[37789]: pgmap v8664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:14 smithi116 ceph-mon[37097]: pgmap v8664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:15.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:15.530+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:00:15.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:15.530+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:00:15.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:15.530+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:16 smithi171 ceph-mon[37789]: pgmap v8665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15561.015675 secs 2024-07-26T13:00:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:16 smithi044 ceph-mon[32404]: pgmap v8665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15561.015675 secs 2024-07-26T13:00:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:16 smithi116 ceph-mon[37097]: pgmap v8665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15561.015675 secs 2024-07-26T13:00:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:18 smithi116 ceph-mon[37097]: pgmap v8666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:18 smithi171 ceph-mon[37789]: pgmap v8666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:18 smithi044 ceph-mon[32404]: pgmap v8666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:20 smithi171 ceph-mon[37789]: pgmap v8667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:20 smithi044 ceph-mon[32404]: pgmap v8667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:20 smithi116 ceph-mon[37097]: pgmap v8667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15566.015820 secs 2024-07-26T13:00:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15566.015820 secs 2024-07-26T13:00:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15566.015820 secs 2024-07-26T13:00:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:22 smithi171 ceph-mon[37789]: pgmap v8668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:22 smithi044 ceph-mon[32404]: pgmap v8668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:22 smithi116 ceph-mon[37097]: pgmap v8668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:24 smithi171 ceph-mon[37789]: pgmap v8669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:25.089 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:24 smithi116 ceph-mon[37097]: pgmap v8669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:24 smithi044 ceph-mon[32404]: pgmap v8669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:26 smithi171 ceph-mon[37789]: pgmap v8670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15571.015987 secs 2024-07-26T13:00:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:26 smithi044 ceph-mon[32404]: pgmap v8670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15571.015987 secs 2024-07-26T13:00:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:26 smithi116 ceph-mon[37097]: pgmap v8670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15571.015987 secs 2024-07-26T13:00:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:28 smithi044 ceph-mon[32404]: pgmap v8671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:28 smithi116 ceph-mon[37097]: pgmap v8671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:28 smithi171 ceph-mon[37789]: pgmap v8671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:30.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:30.531+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:00:30.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:30.531+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:00:30.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:30.531+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:31.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: pgmap v8672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.239 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.241 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.241 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.241 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:31.241 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:31.241 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:00:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: pgmap v8672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:31.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:31.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:00:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: pgmap v8672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:00:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:31.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:31.275 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:00:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15576.016119 secs 2024-07-26T13:00:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: pgmap v8673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:32.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:31 smithi044 ceph-mon[32404]: pgmap v8674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15576.016119 secs 2024-07-26T13:00:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: pgmap v8673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:32.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:31 smithi116 ceph-mon[37097]: pgmap v8674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15576.016119 secs 2024-07-26T13:00:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: pgmap v8673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:31 smithi171 ceph-mon[37789]: pgmap v8674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:33 smithi044 ceph-mon[32404]: pgmap v8675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:33 smithi116 ceph-mon[37097]: pgmap v8675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:33 smithi171 ceph-mon[37789]: pgmap v8675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:35 smithi044 ceph-mon[32404]: pgmap v8676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:35 smithi116 ceph-mon[37097]: pgmap v8676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:35 smithi171 ceph-mon[37789]: pgmap v8676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15581.016260 secs 2024-07-26T13:00:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15581.016260 secs 2024-07-26T13:00:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15581.016260 secs 2024-07-26T13:00:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:37 smithi044 ceph-mon[32404]: pgmap v8677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:37 smithi116 ceph-mon[37097]: pgmap v8677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:37 smithi171 ceph-mon[37789]: pgmap v8677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:39 smithi044 ceph-mon[32404]: pgmap v8678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:39 smithi116 ceph-mon[37097]: pgmap v8678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:39 smithi171 ceph-mon[37789]: pgmap v8678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15586.016434 secs 2024-07-26T13:00:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:41 smithi044 ceph-mon[32404]: pgmap v8679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15586.016434 secs 2024-07-26T13:00:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:41 smithi116 ceph-mon[37097]: pgmap v8679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15586.016434 secs 2024-07-26T13:00:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:41 smithi171 ceph-mon[37789]: pgmap v8679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:43 smithi044 ceph-mon[32404]: pgmap v8680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:43 smithi116 ceph-mon[37097]: pgmap v8680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:43 smithi171 ceph-mon[37789]: pgmap v8680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:45.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:45.536+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:00:45.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:45.536+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:00:45.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:00:45.536+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:45 smithi044 ceph-mon[32404]: pgmap v8681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:45 smithi116 ceph-mon[37097]: pgmap v8681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:45 smithi171 ceph-mon[37789]: pgmap v8681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15591.016593 secs 2024-07-26T13:00:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15591.016593 secs 2024-07-26T13:00:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15591.016593 secs 2024-07-26T13:00:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:47 smithi044 ceph-mon[32404]: pgmap v8682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:47 smithi116 ceph-mon[37097]: pgmap v8682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:47 smithi171 ceph-mon[37789]: pgmap v8682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:49 smithi044 ceph-mon[32404]: pgmap v8683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:49 smithi116 ceph-mon[37097]: pgmap v8683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:49 smithi171 ceph-mon[37789]: pgmap v8683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15596.016730 secs 2024-07-26T13:00:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:51 smithi044 ceph-mon[32404]: pgmap v8684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15596.016730 secs 2024-07-26T13:00:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:51 smithi116 ceph-mon[37097]: pgmap v8684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15596.016730 secs 2024-07-26T13:00:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:51 smithi171 ceph-mon[37789]: pgmap v8684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:53 smithi044 ceph-mon[32404]: pgmap v8685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:53 smithi116 ceph-mon[37097]: pgmap v8685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:53 smithi171 ceph-mon[37789]: pgmap v8685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:55 smithi044 ceph-mon[32404]: pgmap v8686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:55 smithi116 ceph-mon[37097]: pgmap v8686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:55 smithi171 ceph-mon[37789]: pgmap v8686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:00:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15601.016892 secs 2024-07-26T13:00:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15601.016892 secs 2024-07-26T13:00:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15601.016892 secs 2024-07-26T13:00:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:57 smithi044 ceph-mon[32404]: pgmap v8687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:57 smithi116 ceph-mon[37097]: pgmap v8687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:00:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:57 smithi171 ceph-mon[37789]: pgmap v8687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:00:59 smithi044 ceph-mon[32404]: pgmap v8688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:00:59 smithi116 ceph-mon[37097]: pgmap v8688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:00:59 smithi171 ceph-mon[37789]: pgmap v8688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:00.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:00.536+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:01:00.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:00.536+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:00.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:00.536+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:01:02.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15606.017057 secs 2024-07-26T13:01:02.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:01 smithi171 ceph-mon[37789]: pgmap v8689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15606.017057 secs 2024-07-26T13:01:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:01 smithi044 ceph-mon[32404]: pgmap v8689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15606.017057 secs 2024-07-26T13:01:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:01 smithi116 ceph-mon[37097]: pgmap v8689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:03 smithi044 ceph-mon[32404]: pgmap v8690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:03 smithi116 ceph-mon[37097]: pgmap v8690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:03 smithi171 ceph-mon[37789]: pgmap v8690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:05 smithi044 ceph-mon[32404]: pgmap v8691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:06 smithi116 ceph-mon[37097]: pgmap v8691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:05 smithi171 ceph-mon[37789]: pgmap v8691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15611.017204 secs 2024-07-26T13:01:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15611.017204 secs 2024-07-26T13:01:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15611.017204 secs 2024-07-26T13:01:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:08 smithi044 ceph-mon[32404]: pgmap v8692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:08 smithi116 ceph-mon[37097]: pgmap v8692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:08 smithi171 ceph-mon[37789]: pgmap v8692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:10 smithi116 ceph-mon[37097]: pgmap v8693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:10 smithi171 ceph-mon[37789]: pgmap v8693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:10.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:10 smithi044 ceph-mon[32404]: pgmap v8693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15616.017367 secs 2024-07-26T13:01:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:12 smithi171 ceph-mon[37789]: pgmap v8694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15616.017367 secs 2024-07-26T13:01:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:12 smithi044 ceph-mon[32404]: pgmap v8694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15616.017367 secs 2024-07-26T13:01:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:12 smithi116 ceph-mon[37097]: pgmap v8694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:13 smithi171 ceph-mon[37789]: pgmap v8695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:13 smithi044 ceph-mon[32404]: pgmap v8695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:13 smithi116 ceph-mon[37097]: pgmap v8695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:15.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:15.543+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:01:15.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:15.543+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:15.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:15.543+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:01:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:15 smithi171 ceph-mon[37789]: pgmap v8696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:15 smithi044 ceph-mon[32404]: pgmap v8696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:15 smithi116 ceph-mon[37097]: pgmap v8696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15621.017522 secs 2024-07-26T13:01:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15621.017522 secs 2024-07-26T13:01:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15621.017522 secs 2024-07-26T13:01:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:17 smithi044 ceph-mon[32404]: pgmap v8697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:17 smithi116 ceph-mon[37097]: pgmap v8697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:17 smithi171 ceph-mon[37789]: pgmap v8697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:19 smithi044 ceph-mon[32404]: pgmap v8698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:19 smithi116 ceph-mon[37097]: pgmap v8698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:19 smithi171 ceph-mon[37789]: pgmap v8698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15626.017667 secs 2024-07-26T13:01:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:21 smithi044 ceph-mon[32404]: pgmap v8699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15626.017667 secs 2024-07-26T13:01:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:21 smithi116 ceph-mon[37097]: pgmap v8699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15626.017667 secs 2024-07-26T13:01:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:21 smithi171 ceph-mon[37789]: pgmap v8699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:23 smithi044 ceph-mon[32404]: pgmap v8700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:23 smithi116 ceph-mon[37097]: pgmap v8700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:23 smithi171 ceph-mon[37789]: pgmap v8700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:25 smithi044 ceph-mon[32404]: pgmap v8701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:25 smithi116 ceph-mon[37097]: pgmap v8701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:25 smithi171 ceph-mon[37789]: pgmap v8701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:01:27.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15631.017810 secs 2024-07-26T13:01:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15631.017810 secs 2024-07-26T13:01:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15631.017810 secs 2024-07-26T13:01:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:27 smithi044 ceph-mon[32404]: pgmap v8702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:28.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:27 smithi116 ceph-mon[37097]: pgmap v8702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:27 smithi171 ceph-mon[37789]: pgmap v8702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:29 smithi044 ceph-mon[32404]: pgmap v8703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:29 smithi116 ceph-mon[37097]: pgmap v8703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:29 smithi171 ceph-mon[37789]: pgmap v8703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:30.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:30.544+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:01:30.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:30.545+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:01:30.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:30.544+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15636.017950 secs 2024-07-26T13:01:32.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:31 smithi044 ceph-mon[32404]: pgmap v8704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:32.292 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:32.293 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15636.017950 secs 2024-07-26T13:01:32.293 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:31 smithi116 ceph-mon[37097]: pgmap v8704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:32.293 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15636.017950 secs 2024-07-26T13:01:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:31 smithi171 ceph-mon[37789]: pgmap v8704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:32 smithi044 ceph-mon[32404]: pgmap v8705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:01:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:32 smithi044 ceph-mon[32404]: pgmap v8706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:01:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:32 smithi116 ceph-mon[37097]: pgmap v8705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:01:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:32 smithi116 ceph-mon[37097]: pgmap v8706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:01:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:32 smithi171 ceph-mon[37789]: pgmap v8705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:01:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:32 smithi171 ceph-mon[37789]: pgmap v8706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:33.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:01:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:34 smithi171 ceph-mon[37789]: pgmap v8707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:34 smithi044 ceph-mon[32404]: pgmap v8707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:34 smithi116 ceph-mon[37097]: pgmap v8707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:36 smithi044 ceph-mon[32404]: pgmap v8708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:01:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:36 smithi116 ceph-mon[37097]: pgmap v8708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:01:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:36 smithi171 ceph-mon[37789]: pgmap v8708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:01:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15641.018071 secs 2024-07-26T13:01:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15641.018071 secs 2024-07-26T13:01:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15641.018071 secs 2024-07-26T13:01:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:38 smithi044 ceph-mon[32404]: pgmap v8709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:38 smithi116 ceph-mon[37097]: pgmap v8709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:38 smithi171 ceph-mon[37789]: pgmap v8709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:40 smithi044 ceph-mon[32404]: pgmap v8710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:40 smithi116 ceph-mon[37097]: pgmap v8710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:40 smithi171 ceph-mon[37789]: pgmap v8710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15646.018205 secs 2024-07-26T13:01:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15646.018205 secs 2024-07-26T13:01:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15646.018205 secs 2024-07-26T13:01:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:42 smithi044 ceph-mon[32404]: pgmap v8711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:42 smithi116 ceph-mon[37097]: pgmap v8711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:42 smithi171 ceph-mon[37789]: pgmap v8711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:44 smithi044 ceph-mon[32404]: pgmap v8712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:44 smithi116 ceph-mon[37097]: pgmap v8712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:44 smithi171 ceph-mon[37789]: pgmap v8712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:45.550+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:01:45.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:45.550+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:45.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:01:45.550+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:01:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:46 smithi044 ceph-mon[32404]: pgmap v8713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:46 smithi116 ceph-mon[37097]: pgmap v8713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:46 smithi171 ceph-mon[37789]: pgmap v8713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15651.018343 secs 2024-07-26T13:01:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15651.018343 secs 2024-07-26T13:01:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15651.018343 secs 2024-07-26T13:01:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:48 smithi044 ceph-mon[32404]: pgmap v8714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:48 smithi116 ceph-mon[37097]: pgmap v8714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:48 smithi171 ceph-mon[37789]: pgmap v8714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:51 smithi044 ceph-mon[32404]: pgmap v8715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:51 smithi116 ceph-mon[37097]: pgmap v8715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:51 smithi171 ceph-mon[37789]: pgmap v8715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15656.018480 secs 2024-07-26T13:01:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15656.018480 secs 2024-07-26T13:01:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15656.018480 secs 2024-07-26T13:01:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:53 smithi044 ceph-mon[32404]: pgmap v8716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:53 smithi116 ceph-mon[37097]: pgmap v8716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:53 smithi171 ceph-mon[37789]: pgmap v8716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:55 smithi044 ceph-mon[32404]: pgmap v8717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:55 smithi116 ceph-mon[37097]: pgmap v8717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:55 smithi171 ceph-mon[37789]: pgmap v8717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15661.018612 secs 2024-07-26T13:01:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:57 smithi044 ceph-mon[32404]: pgmap v8718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15661.018612 secs 2024-07-26T13:01:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:57 smithi116 ceph-mon[37097]: pgmap v8718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15661.018612 secs 2024-07-26T13:01:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:57 smithi171 ceph-mon[37789]: pgmap v8718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:01:59 smithi044 ceph-mon[32404]: pgmap v8719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:01:59 smithi116 ceph-mon[37097]: pgmap v8719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:01:59 smithi171 ceph-mon[37789]: pgmap v8719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:00.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:00.551+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:02:00.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:00.551+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:02:00.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:00.551+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:01 smithi044 ceph-mon[32404]: pgmap v8720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:01 smithi116 ceph-mon[37097]: pgmap v8720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:01 smithi171 ceph-mon[37789]: pgmap v8720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:02.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15666.018748 secs 2024-07-26T13:02:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15666.018748 secs 2024-07-26T13:02:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15666.018748 secs 2024-07-26T13:02:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:03 smithi044 ceph-mon[32404]: pgmap v8721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:03 smithi116 ceph-mon[37097]: pgmap v8721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:03 smithi171 ceph-mon[37789]: pgmap v8721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:05 smithi044 ceph-mon[32404]: pgmap v8722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:05 smithi116 ceph-mon[37097]: pgmap v8722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:05 smithi171 ceph-mon[37789]: pgmap v8722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15671.018889 secs 2024-07-26T13:02:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:07 smithi044 ceph-mon[32404]: pgmap v8723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15671.018889 secs 2024-07-26T13:02:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:07 smithi116 ceph-mon[37097]: pgmap v8723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15671.018889 secs 2024-07-26T13:02:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:07 smithi171 ceph-mon[37789]: pgmap v8723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:09 smithi044 ceph-mon[32404]: pgmap v8724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:09 smithi116 ceph-mon[37097]: pgmap v8724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:09 smithi171 ceph-mon[37789]: pgmap v8724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:11 smithi044 ceph-mon[32404]: pgmap v8725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:11 smithi116 ceph-mon[37097]: pgmap v8725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:11 smithi171 ceph-mon[37789]: pgmap v8725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15676.019024 secs 2024-07-26T13:02:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15676.019024 secs 2024-07-26T13:02:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15676.019024 secs 2024-07-26T13:02:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:13 smithi044 ceph-mon[32404]: pgmap v8726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:13 smithi116 ceph-mon[37097]: pgmap v8726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:13 smithi171 ceph-mon[37789]: pgmap v8726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:15 smithi044 ceph-mon[32404]: pgmap v8727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:15 smithi116 ceph-mon[37097]: pgmap v8727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:15.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:15.555+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:02:15.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:15.555+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:02:15.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:15.555+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:15 smithi171 ceph-mon[37789]: pgmap v8727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:17.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:17.508+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:02:17.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:17.508+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:02:17.510 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15681.019166 secs 2024-07-26T13:02:17.511 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:17 smithi044 ceph-mon[32404]: pgmap v8728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:17.509+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:02:17.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:17.509+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:02:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15681.019166 secs 2024-07-26T13:02:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:17 smithi116 ceph-mon[37097]: pgmap v8728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15681.019166 secs 2024-07-26T13:02:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:17 smithi171 ceph-mon[37789]: pgmap v8728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:19 smithi044 ceph-mon[32404]: pgmap v8729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:19 smithi116 ceph-mon[37097]: pgmap v8729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:19 smithi171 ceph-mon[37789]: pgmap v8729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:21 smithi044 ceph-mon[32404]: pgmap v8730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:21 smithi116 ceph-mon[37097]: pgmap v8730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:21 smithi171 ceph-mon[37789]: pgmap v8730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15686.019304 secs 2024-07-26T13:02:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15686.019304 secs 2024-07-26T13:02:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15686.019304 secs 2024-07-26T13:02:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:23 smithi044 ceph-mon[32404]: pgmap v8731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:23 smithi116 ceph-mon[37097]: pgmap v8731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:23 smithi171 ceph-mon[37789]: pgmap v8731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:25 smithi044 ceph-mon[32404]: pgmap v8732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:25 smithi116 ceph-mon[37097]: pgmap v8732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:25 smithi171 ceph-mon[37789]: pgmap v8732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15691.019398 secs 2024-07-26T13:02:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:27 smithi044 ceph-mon[32404]: pgmap v8733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:02:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15691.019398 secs 2024-07-26T13:02:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:27 smithi116 ceph-mon[37097]: pgmap v8733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:02:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15691.019398 secs 2024-07-26T13:02:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:27 smithi171 ceph-mon[37789]: pgmap v8733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:02:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:29 smithi044 ceph-mon[32404]: pgmap v8734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:02:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:29 smithi116 ceph-mon[37097]: pgmap v8734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:02:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:29 smithi171 ceph-mon[37789]: pgmap v8734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:02:30.558 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:30.557+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:02:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:30.557+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:02:30.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:30.557+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:31 smithi044 ceph-mon[32404]: pgmap v8735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:02:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:31 smithi116 ceph-mon[37097]: pgmap v8735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:02:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:31 smithi171 ceph-mon[37789]: pgmap v8735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:02:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15696.019563 secs 2024-07-26T13:02:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15696.019563 secs 2024-07-26T13:02:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15696.019563 secs 2024-07-26T13:02:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:33 smithi044 ceph-mon[32404]: pgmap v8736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:33 smithi116 ceph-mon[37097]: pgmap v8736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:33 smithi171 ceph-mon[37789]: pgmap v8736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:02:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:33.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:34 smithi171 ceph-mon[37789]: pgmap v8737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:34 smithi171 ceph-mon[37789]: pgmap v8738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:34.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:34 smithi044 ceph-mon[32404]: pgmap v8737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:34 smithi044 ceph-mon[32404]: pgmap v8738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:34.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:34 smithi116 ceph-mon[37097]: pgmap v8737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:34 smithi116 ceph-mon[37097]: pgmap v8738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:36 smithi044 ceph-mon[32404]: pgmap v8739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:36 smithi116 ceph-mon[37097]: pgmap v8739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:36 smithi171 ceph-mon[37789]: pgmap v8739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15701.019736 secs 2024-07-26T13:02:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15701.019736 secs 2024-07-26T13:02:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15701.019736 secs 2024-07-26T13:02:38.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:38 smithi044 ceph-mon[32404]: pgmap v8740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:38 smithi116 ceph-mon[37097]: pgmap v8740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:38 smithi171 ceph-mon[37789]: pgmap v8740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:40 smithi044 ceph-mon[32404]: pgmap v8741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:40 smithi116 ceph-mon[37097]: pgmap v8741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:40.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:40 smithi171 ceph-mon[37789]: pgmap v8741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15706.019897 secs 2024-07-26T13:02:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15706.019897 secs 2024-07-26T13:02:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15706.019897 secs 2024-07-26T13:02:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:42 smithi044 ceph-mon[32404]: pgmap v8742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:42 smithi116 ceph-mon[37097]: pgmap v8742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:42 smithi171 ceph-mon[37789]: pgmap v8742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:02:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:44 smithi044 ceph-mon[32404]: pgmap v8743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:44 smithi116 ceph-mon[37097]: pgmap v8743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:44 smithi171 ceph-mon[37789]: pgmap v8743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:45.562+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:02:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:45.562+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:02:45.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:02:45.562+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:46 smithi044 ceph-mon[32404]: pgmap v8744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:46 smithi116 ceph-mon[37097]: pgmap v8744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:46 smithi171 ceph-mon[37789]: pgmap v8744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15711.020000 secs 2024-07-26T13:02:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15711.020000 secs 2024-07-26T13:02:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15711.020000 secs 2024-07-26T13:02:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:48 smithi044 ceph-mon[32404]: pgmap v8745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:48 smithi116 ceph-mon[37097]: pgmap v8745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:48 smithi171 ceph-mon[37789]: pgmap v8745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:50 smithi044 ceph-mon[32404]: pgmap v8746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:50 smithi116 ceph-mon[37097]: pgmap v8746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:50 smithi171 ceph-mon[37789]: pgmap v8746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15716.020151 secs 2024-07-26T13:02:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15716.020151 secs 2024-07-26T13:02:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15716.020151 secs 2024-07-26T13:02:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:52 smithi044 ceph-mon[32404]: pgmap v8747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:52 smithi116 ceph-mon[37097]: pgmap v8747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:52 smithi171 ceph-mon[37789]: pgmap v8747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:54 smithi044 ceph-mon[32404]: pgmap v8748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:54 smithi116 ceph-mon[37097]: pgmap v8748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:54 smithi171 ceph-mon[37789]: pgmap v8748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:02:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:56 smithi044 ceph-mon[32404]: pgmap v8749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:56 smithi116 ceph-mon[37097]: pgmap v8749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:56 smithi171 ceph-mon[37789]: pgmap v8749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:57.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15721.020315 secs 2024-07-26T13:02:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15721.020315 secs 2024-07-26T13:02:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15721.020315 secs 2024-07-26T13:02:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:02:58 smithi171 ceph-mon[37789]: pgmap v8750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:02:58 smithi044 ceph-mon[32404]: pgmap v8750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:02:58 smithi116 ceph-mon[37097]: pgmap v8750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:00 smithi044 ceph-mon[32404]: pgmap v8751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:00 smithi116 ceph-mon[37097]: pgmap v8751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:00.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:00.563+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:03:00.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:00.563+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:00.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:00.563+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:03:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:00 smithi171 ceph-mon[37789]: pgmap v8751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15726.020465 secs 2024-07-26T13:03:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15726.020465 secs 2024-07-26T13:03:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15726.020465 secs 2024-07-26T13:03:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:02 smithi044 ceph-mon[32404]: pgmap v8752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:02 smithi116 ceph-mon[37097]: pgmap v8752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:02 smithi171 ceph-mon[37789]: pgmap v8752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:04 smithi044 ceph-mon[32404]: pgmap v8753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:04 smithi116 ceph-mon[37097]: pgmap v8753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:04 smithi171 ceph-mon[37789]: pgmap v8753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:06 smithi044 ceph-mon[32404]: pgmap v8754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:06 smithi116 ceph-mon[37097]: pgmap v8754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:06 smithi171 ceph-mon[37789]: pgmap v8754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15731.020654 secs 2024-07-26T13:03:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15731.020654 secs 2024-07-26T13:03:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15731.020654 secs 2024-07-26T13:03:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:08 smithi044 ceph-mon[32404]: pgmap v8755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:08 smithi116 ceph-mon[37097]: pgmap v8755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:08 smithi171 ceph-mon[37789]: pgmap v8755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:10 smithi044 ceph-mon[32404]: pgmap v8756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:10 smithi116 ceph-mon[37097]: pgmap v8756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:10 smithi171 ceph-mon[37789]: pgmap v8756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15736.020795 secs 2024-07-26T13:03:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15736.020795 secs 2024-07-26T13:03:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15736.020795 secs 2024-07-26T13:03:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:12 smithi044 ceph-mon[32404]: pgmap v8757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:12 smithi116 ceph-mon[37097]: pgmap v8757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:12 smithi171 ceph-mon[37789]: pgmap v8757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:14 smithi044 ceph-mon[32404]: pgmap v8758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:14 smithi116 ceph-mon[37097]: pgmap v8758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:14 smithi171 ceph-mon[37789]: pgmap v8758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:15.568+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:03:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:15.568+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:15.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:15.568+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:03:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:16 smithi044 ceph-mon[32404]: pgmap v8759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:16 smithi116 ceph-mon[37097]: pgmap v8759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:16 smithi171 ceph-mon[37789]: pgmap v8759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15741.020933 secs 2024-07-26T13:03:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15741.020933 secs 2024-07-26T13:03:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15741.020933 secs 2024-07-26T13:03:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:18 smithi044 ceph-mon[32404]: pgmap v8760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:18 smithi116 ceph-mon[37097]: pgmap v8760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:18 smithi171 ceph-mon[37789]: pgmap v8760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:20 smithi044 ceph-mon[32404]: pgmap v8761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:20 smithi116 ceph-mon[37097]: pgmap v8761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:20 smithi171 ceph-mon[37789]: pgmap v8761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15746.021101 secs 2024-07-26T13:03:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15746.021101 secs 2024-07-26T13:03:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15746.021101 secs 2024-07-26T13:03:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:22 smithi044 ceph-mon[32404]: pgmap v8762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:22 smithi116 ceph-mon[37097]: pgmap v8762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:22 smithi171 ceph-mon[37789]: pgmap v8762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:24 smithi044 ceph-mon[32404]: pgmap v8763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:24 smithi116 ceph-mon[37097]: pgmap v8763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:24 smithi171 ceph-mon[37789]: pgmap v8763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:26 smithi044 ceph-mon[32404]: pgmap v8764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:26 smithi116 ceph-mon[37097]: pgmap v8764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:26 smithi171 ceph-mon[37789]: pgmap v8764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:27.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15751.021251 secs 2024-07-26T13:03:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15751.021251 secs 2024-07-26T13:03:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15751.021251 secs 2024-07-26T13:03:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:29 smithi044 ceph-mon[32404]: pgmap v8765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:29 smithi116 ceph-mon[37097]: pgmap v8765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:29 smithi171 ceph-mon[37789]: pgmap v8765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:30.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:30.570+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:03:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:30.570+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:30.572 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:30.570+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:03:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:31 smithi044 ceph-mon[32404]: pgmap v8766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:31 smithi116 ceph-mon[37097]: pgmap v8766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:31 smithi171 ceph-mon[37789]: pgmap v8766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:32.487 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15756.021338 secs 2024-07-26T13:03:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15756.021338 secs 2024-07-26T13:03:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15756.021338 secs 2024-07-26T13:03:33.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:33 smithi044 ceph-mon[32404]: pgmap v8767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:33 smithi116 ceph-mon[37097]: pgmap v8767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:33 smithi171 ceph-mon[37789]: pgmap v8767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:35.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:35 smithi044 ceph-mon[32404]: pgmap v8768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:35.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:35.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.410 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.411 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.429 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:35 smithi171 ceph-mon[37789]: pgmap v8768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:35.429 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:35.429 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.429 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.429 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.429 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:35 smithi116 ceph-mon[37097]: pgmap v8768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:35.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:35.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:35.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15761.021491 secs 2024-07-26T13:03:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: pgmap v8769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15761.021491 secs 2024-07-26T13:03:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: pgmap v8769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15761.021491 secs 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: pgmap v8769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:37.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:39 smithi044 ceph-mon[32404]: pgmap v8770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:39 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:40.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:39 smithi116 ceph-mon[37097]: pgmap v8770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:39 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:40.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:39 smithi171 ceph-mon[37789]: pgmap v8770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:39 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:40.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:40 smithi044 ceph-mon[32404]: pgmap v8771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:03:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:40 smithi044 ceph-mon[32404]: pgmap v8772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:40 smithi116 ceph-mon[37097]: pgmap v8771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:03:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:40 smithi116 ceph-mon[37097]: pgmap v8772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:40 smithi171 ceph-mon[37789]: pgmap v8771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:03:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:40 smithi171 ceph-mon[37789]: pgmap v8772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15766.021669 secs 2024-07-26T13:03:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15766.021669 secs 2024-07-26T13:03:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15766.021669 secs 2024-07-26T13:03:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:42 smithi044 ceph-mon[32404]: pgmap v8773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:42 smithi116 ceph-mon[37097]: pgmap v8773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:42 smithi171 ceph-mon[37789]: pgmap v8773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:44 smithi044 ceph-mon[32404]: pgmap v8774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:44 smithi116 ceph-mon[37097]: pgmap v8774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:44 smithi171 ceph-mon[37789]: pgmap v8774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:45.575+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:03:45.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:45.575+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:45.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:03:45.575+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:03:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:46 smithi044 ceph-mon[32404]: pgmap v8775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15771.021836 secs 2024-07-26T13:03:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:46 smithi116 ceph-mon[37097]: pgmap v8775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15771.021836 secs 2024-07-26T13:03:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:46 smithi171 ceph-mon[37789]: pgmap v8775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15771.021836 secs 2024-07-26T13:03:49.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:49 smithi044 ceph-mon[32404]: pgmap v8776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:03:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:49 smithi116 ceph-mon[37097]: pgmap v8776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:03:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:49 smithi171 ceph-mon[37789]: pgmap v8776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:03:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:51 smithi044 ceph-mon[32404]: pgmap v8777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:51 smithi116 ceph-mon[37097]: pgmap v8777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:51 smithi171 ceph-mon[37789]: pgmap v8777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15776.021980 secs 2024-07-26T13:03:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15776.021980 secs 2024-07-26T13:03:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15776.021980 secs 2024-07-26T13:03:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:53 smithi116 ceph-mon[37097]: pgmap v8778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:53 smithi171 ceph-mon[37789]: pgmap v8778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:53 smithi044 ceph-mon[32404]: pgmap v8778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:55 smithi171 ceph-mon[37789]: pgmap v8779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:55 smithi044 ceph-mon[32404]: pgmap v8779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:55 smithi116 ceph-mon[37097]: pgmap v8779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:57.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:57 smithi116 ceph-mon[37097]: pgmap v8780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:57.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15781.022147 secs 2024-07-26T13:03:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:57 smithi171 ceph-mon[37789]: pgmap v8780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15781.022147 secs 2024-07-26T13:03:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:57 smithi044 ceph-mon[32404]: pgmap v8780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15781.022147 secs 2024-07-26T13:03:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:03:59 smithi044 ceph-mon[32404]: pgmap v8781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:03:59 smithi171 ceph-mon[37789]: pgmap v8781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:03:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:03:59 smithi116 ceph-mon[37097]: pgmap v8781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:00.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:00.576+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:04:00.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:00.576+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:00.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:00.576+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:04:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:01 smithi171 ceph-mon[37789]: pgmap v8782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:01 smithi044 ceph-mon[32404]: pgmap v8782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:01 smithi116 ceph-mon[37097]: pgmap v8782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15786.022308 secs 2024-07-26T13:04:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15786.022308 secs 2024-07-26T13:04:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15786.022308 secs 2024-07-26T13:04:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:03 smithi171 ceph-mon[37789]: pgmap v8783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:03 smithi044 ceph-mon[32404]: pgmap v8783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:03 smithi116 ceph-mon[37097]: pgmap v8783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:04 smithi044 ceph-mon[32404]: pgmap v8784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:04 smithi116 ceph-mon[37097]: pgmap v8784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:04 smithi171 ceph-mon[37789]: pgmap v8784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:06 smithi044 ceph-mon[32404]: pgmap v8785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15791.022473 secs 2024-07-26T13:04:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:06 smithi116 ceph-mon[37097]: pgmap v8785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15791.022473 secs 2024-07-26T13:04:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:06 smithi171 ceph-mon[37789]: pgmap v8785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15791.022473 secs 2024-07-26T13:04:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:08 smithi044 ceph-mon[32404]: pgmap v8786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:08 smithi116 ceph-mon[37097]: pgmap v8786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:08 smithi171 ceph-mon[37789]: pgmap v8786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:10 smithi044 ceph-mon[32404]: pgmap v8787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:10 smithi116 ceph-mon[37097]: pgmap v8787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:11.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:10 smithi171 ceph-mon[37789]: pgmap v8787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15796.022658 secs 2024-07-26T13:04:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15796.022658 secs 2024-07-26T13:04:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15796.022658 secs 2024-07-26T13:04:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:12 smithi044 ceph-mon[32404]: pgmap v8788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:12 smithi116 ceph-mon[37097]: pgmap v8788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:12 smithi171 ceph-mon[37789]: pgmap v8788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:14 smithi044 ceph-mon[32404]: pgmap v8789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:14 smithi116 ceph-mon[37097]: pgmap v8789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:14 smithi171 ceph-mon[37789]: pgmap v8789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:15.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:15.581+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:04:15.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:15.581+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:15.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:15.581+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:04:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:16 smithi044 ceph-mon[32404]: pgmap v8790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15801.023415 secs 2024-07-26T13:04:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:16 smithi116 ceph-mon[37097]: pgmap v8790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15801.023415 secs 2024-07-26T13:04:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:16 smithi171 ceph-mon[37789]: pgmap v8790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15801.023415 secs 2024-07-26T13:04:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:18 smithi044 ceph-mon[32404]: pgmap v8791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:18 smithi116 ceph-mon[37097]: pgmap v8791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:18 smithi171 ceph-mon[37789]: pgmap v8791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:20 smithi044 ceph-mon[32404]: pgmap v8792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:20 smithi116 ceph-mon[37097]: pgmap v8792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:20 smithi171 ceph-mon[37789]: pgmap v8792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15806.023500 secs 2024-07-26T13:04:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15806.023500 secs 2024-07-26T13:04:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15806.023500 secs 2024-07-26T13:04:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:22 smithi116 ceph-mon[37097]: pgmap v8793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:22 smithi171 ceph-mon[37789]: pgmap v8793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:22 smithi044 ceph-mon[32404]: pgmap v8793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:24 smithi116 ceph-mon[37097]: pgmap v8794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:24 smithi171 ceph-mon[37789]: pgmap v8794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:24 smithi044 ceph-mon[32404]: pgmap v8794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:26 smithi171 ceph-mon[37789]: pgmap v8795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15811.023683 secs 2024-07-26T13:04:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:26 smithi044 ceph-mon[32404]: pgmap v8795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15811.023683 secs 2024-07-26T13:04:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:26 smithi116 ceph-mon[37097]: pgmap v8795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15811.023683 secs 2024-07-26T13:04:29.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:28 smithi116 ceph-mon[37097]: pgmap v8796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:28 smithi171 ceph-mon[37789]: pgmap v8796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:28 smithi044 ceph-mon[32404]: pgmap v8796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:30.583+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:04:30.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:30.583+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:30.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:30.583+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:04:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:30 smithi044 ceph-mon[32404]: pgmap v8797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:30 smithi171 ceph-mon[37789]: pgmap v8797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:30 smithi116 ceph-mon[37097]: pgmap v8797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15816.023838 secs 2024-07-26T13:04:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15816.023838 secs 2024-07-26T13:04:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15816.023838 secs 2024-07-26T13:04:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:32 smithi171 ceph-mon[37789]: pgmap v8798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:32 smithi044 ceph-mon[32404]: pgmap v8798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:32 smithi116 ceph-mon[37097]: pgmap v8798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:34 smithi171 ceph-mon[37789]: pgmap v8799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:34 smithi044 ceph-mon[32404]: pgmap v8799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:34 smithi116 ceph-mon[37097]: pgmap v8799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:36 smithi171 ceph-mon[37789]: pgmap v8800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15821.023987 secs 2024-07-26T13:04:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:36 smithi044 ceph-mon[32404]: pgmap v8800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15821.023987 secs 2024-07-26T13:04:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:36 smithi116 ceph-mon[37097]: pgmap v8800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15821.023987 secs 2024-07-26T13:04:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:38 smithi171 ceph-mon[37789]: pgmap v8801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:38 smithi044 ceph-mon[32404]: pgmap v8801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:38 smithi116 ceph-mon[37097]: pgmap v8801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:40 smithi171 ceph-mon[37789]: pgmap v8802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:40 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:40 smithi044 ceph-mon[32404]: pgmap v8802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:40 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:40 smithi116 ceph-mon[37097]: pgmap v8802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:40 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15826.024119 secs 2024-07-26T13:04:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:41 smithi044 ceph-mon[32404]: pgmap v8803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:41 smithi044 ceph-mon[32404]: pgmap v8804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15826.024119 secs 2024-07-26T13:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:41 smithi116 ceph-mon[37097]: pgmap v8803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:41 smithi116 ceph-mon[37097]: pgmap v8804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15826.024119 secs 2024-07-26T13:04:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:41 smithi171 ceph-mon[37789]: pgmap v8803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:41 smithi171 ceph-mon[37789]: pgmap v8804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:43 smithi044 ceph-mon[32404]: pgmap v8805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:43 smithi116 ceph-mon[37097]: pgmap v8805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:43 smithi171 ceph-mon[37789]: pgmap v8805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:45.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:45.589+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:04:45.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:45.589+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:04:45.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:04:45.589+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:45 smithi044 ceph-mon[32404]: pgmap v8806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:45 smithi116 ceph-mon[37097]: pgmap v8806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:45 smithi171 ceph-mon[37789]: pgmap v8806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15831.024265 secs 2024-07-26T13:04:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15831.024265 secs 2024-07-26T13:04:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15831.024265 secs 2024-07-26T13:04:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:47 smithi044 ceph-mon[32404]: pgmap v8807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:47 smithi116 ceph-mon[37097]: pgmap v8807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:47 smithi171 ceph-mon[37789]: pgmap v8807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:49 smithi044 ceph-mon[32404]: pgmap v8808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:49 smithi116 ceph-mon[37097]: pgmap v8808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:49 smithi171 ceph-mon[37789]: pgmap v8808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:04:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15836.024435 secs 2024-07-26T13:04:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:51 smithi044 ceph-mon[32404]: pgmap v8809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:04:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15836.024435 secs 2024-07-26T13:04:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:51 smithi116 ceph-mon[37097]: pgmap v8809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:04:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15836.024435 secs 2024-07-26T13:04:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:51 smithi171 ceph-mon[37789]: pgmap v8809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:04:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:53 smithi044 ceph-mon[32404]: pgmap v8810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:53 smithi116 ceph-mon[37097]: pgmap v8810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:53 smithi171 ceph-mon[37789]: pgmap v8810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:55 smithi044 ceph-mon[32404]: pgmap v8811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:55 smithi116 ceph-mon[37097]: pgmap v8811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:55 smithi171 ceph-mon[37789]: pgmap v8811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:04:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15841.024569 secs 2024-07-26T13:04:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15841.024569 secs 2024-07-26T13:04:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15841.024569 secs 2024-07-26T13:04:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:57 smithi044 ceph-mon[32404]: pgmap v8812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:57 smithi116 ceph-mon[37097]: pgmap v8812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:04:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:57 smithi171 ceph-mon[37789]: pgmap v8812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:04:59 smithi044 ceph-mon[32404]: pgmap v8813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:04:59 smithi116 ceph-mon[37097]: pgmap v8813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:04:59 smithi171 ceph-mon[37789]: pgmap v8813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:00.590+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:05:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:00.590+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:05:00.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:00.590+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15846.024731 secs 2024-07-26T13:05:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:01 smithi044 ceph-mon[32404]: pgmap v8814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15846.024731 secs 2024-07-26T13:05:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:01 smithi116 ceph-mon[37097]: pgmap v8814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15846.024731 secs 2024-07-26T13:05:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:01 smithi171 ceph-mon[37789]: pgmap v8814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:03 smithi044 ceph-mon[32404]: pgmap v8815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:03 smithi116 ceph-mon[37097]: pgmap v8815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:03 smithi171 ceph-mon[37789]: pgmap v8815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:06 smithi044 ceph-mon[32404]: pgmap v8816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:06 smithi116 ceph-mon[37097]: pgmap v8816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:06 smithi171 ceph-mon[37789]: pgmap v8816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15851.024864 secs 2024-07-26T13:05:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15851.024864 secs 2024-07-26T13:05:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15851.024864 secs 2024-07-26T13:05:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:08 smithi116 ceph-mon[37097]: pgmap v8817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:08 smithi171 ceph-mon[37789]: pgmap v8817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:08 smithi044 ceph-mon[32404]: pgmap v8817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:10 smithi171 ceph-mon[37789]: pgmap v8818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:10 smithi044 ceph-mon[32404]: pgmap v8818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:10 smithi116 ceph-mon[37097]: pgmap v8818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15856.024975 secs 2024-07-26T13:05:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:12 smithi171 ceph-mon[37789]: pgmap v8819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15856.024975 secs 2024-07-26T13:05:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:12 smithi044 ceph-mon[32404]: pgmap v8819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15856.024975 secs 2024-07-26T13:05:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:12 smithi116 ceph-mon[37097]: pgmap v8819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:14 smithi171 ceph-mon[37789]: pgmap v8820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:14 smithi044 ceph-mon[32404]: pgmap v8820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:14 smithi116 ceph-mon[37097]: pgmap v8820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:15.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:15.596+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:05:15.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:15.596+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:05:15.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:15.596+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:16 smithi171 ceph-mon[37789]: pgmap v8821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:16 smithi044 ceph-mon[32404]: pgmap v8821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:16 smithi116 ceph-mon[37097]: pgmap v8821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15861.025092 secs 2024-07-26T13:05:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15861.025092 secs 2024-07-26T13:05:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15861.025092 secs 2024-07-26T13:05:17.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.540+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:05:17.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.540+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:05:17.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.540+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:05:17.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.540+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:05:17.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.540+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:05:17.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.540+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:05:17.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.541+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:05:17.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:17.541+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:05:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:18 smithi171 ceph-mon[37789]: pgmap v8822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:18 smithi044 ceph-mon[32404]: pgmap v8822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:18 smithi116 ceph-mon[37097]: pgmap v8822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:20 smithi171 ceph-mon[37789]: pgmap v8823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:20 smithi044 ceph-mon[32404]: pgmap v8823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:20 smithi116 ceph-mon[37097]: pgmap v8823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15866.025225 secs 2024-07-26T13:05:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:22 smithi171 ceph-mon[37789]: pgmap v8824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15866.025225 secs 2024-07-26T13:05:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:22 smithi044 ceph-mon[32404]: pgmap v8824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15866.025225 secs 2024-07-26T13:05:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:22 smithi116 ceph-mon[37097]: pgmap v8824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:24 smithi171 ceph-mon[37789]: pgmap v8825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:24 smithi044 ceph-mon[32404]: pgmap v8825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:24 smithi116 ceph-mon[37097]: pgmap v8825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:26 smithi171 ceph-mon[37789]: pgmap v8826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:26 smithi044 ceph-mon[32404]: pgmap v8826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:26 smithi116 ceph-mon[37097]: pgmap v8826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:05:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15871.025335 secs 2024-07-26T13:05:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:27 smithi044 ceph-mon[32404]: pgmap v8827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15871.025335 secs 2024-07-26T13:05:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:27 smithi116 ceph-mon[37097]: pgmap v8827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15871.025335 secs 2024-07-26T13:05:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:27 smithi171 ceph-mon[37789]: pgmap v8827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:05:30.225 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:29 smithi116 ceph-mon[37097]: pgmap v8828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:29 smithi044 ceph-mon[32404]: pgmap v8828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:29 smithi171 ceph-mon[37789]: pgmap v8828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:30.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:30.597+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:05:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:30.597+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:30.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:30.597+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:05:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15876.025472 secs 2024-07-26T13:05:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:31 smithi044 ceph-mon[32404]: pgmap v8829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:05:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15876.025472 secs 2024-07-26T13:05:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:31 smithi116 ceph-mon[37097]: pgmap v8829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:05:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15876.025472 secs 2024-07-26T13:05:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:31 smithi171 ceph-mon[37789]: pgmap v8829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:05:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:33 smithi044 ceph-mon[32404]: pgmap v8830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:33 smithi116 ceph-mon[37097]: pgmap v8830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:33 smithi171 ceph-mon[37789]: pgmap v8830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:35 smithi044 ceph-mon[32404]: pgmap v8831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:35 smithi116 ceph-mon[37097]: pgmap v8831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:35 smithi171 ceph-mon[37789]: pgmap v8831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15881.025608 secs 2024-07-26T13:05:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15881.025608 secs 2024-07-26T13:05:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15881.025608 secs 2024-07-26T13:05:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:37 smithi044 ceph-mon[32404]: pgmap v8832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:38.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:37 smithi116 ceph-mon[37097]: pgmap v8832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:37 smithi171 ceph-mon[37789]: pgmap v8832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:39 smithi044 ceph-mon[32404]: pgmap v8833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:39 smithi116 ceph-mon[37097]: pgmap v8833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:39 smithi171 ceph-mon[37789]: pgmap v8833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:05:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15886.025749 secs 2024-07-26T13:05:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:41 smithi044 ceph-mon[32404]: pgmap v8834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15886.025749 secs 2024-07-26T13:05:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:41 smithi116 ceph-mon[37097]: pgmap v8834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15886.025749 secs 2024-07-26T13:05:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:41 smithi171 ceph-mon[37789]: pgmap v8834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:43.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:43.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:44 smithi044 ceph-mon[32404]: pgmap v8835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:44 smithi116 ceph-mon[37097]: pgmap v8835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:44 smithi171 ceph-mon[37789]: pgmap v8835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:45.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:45.603+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:05:45.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:45.603+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:45.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:05:45.603+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:05:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:46 smithi116 ceph-mon[37097]: pgmap v8836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:46 smithi171 ceph-mon[37789]: pgmap v8836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:46 smithi044 ceph-mon[32404]: pgmap v8836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15891.025847 secs 2024-07-26T13:05:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15891.025847 secs 2024-07-26T13:05:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15891.025847 secs 2024-07-26T13:05:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:48 smithi171 ceph-mon[37789]: pgmap v8837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:48 smithi044 ceph-mon[32404]: pgmap v8837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:48 smithi116 ceph-mon[37097]: pgmap v8837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:50 smithi171 ceph-mon[37789]: pgmap v8838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:50 smithi044 ceph-mon[32404]: pgmap v8838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:50 smithi116 ceph-mon[37097]: pgmap v8838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:05:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:52 smithi171 ceph-mon[37789]: pgmap v8839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15896.025946 secs 2024-07-26T13:05:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:52 smithi044 ceph-mon[32404]: pgmap v8839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15896.025946 secs 2024-07-26T13:05:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:52 smithi116 ceph-mon[37097]: pgmap v8839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15896.025946 secs 2024-07-26T13:05:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:54 smithi171 ceph-mon[37789]: pgmap v8840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:05:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:54 smithi044 ceph-mon[32404]: pgmap v8840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:05:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:54 smithi116 ceph-mon[37097]: pgmap v8840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:05:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:56 smithi171 ceph-mon[37789]: pgmap v8841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:56.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:56 smithi044 ceph-mon[32404]: pgmap v8841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:56 smithi116 ceph-mon[37097]: pgmap v8841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15901.026057 secs 2024-07-26T13:05:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15901.026057 secs 2024-07-26T13:05:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15901.026057 secs 2024-07-26T13:05:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:05:58 smithi171 ceph-mon[37789]: pgmap v8842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:05:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:05:58 smithi044 ceph-mon[32404]: pgmap v8842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:05:58.551 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:05:58 smithi116 ceph-mon[37097]: pgmap v8842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:06:00.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:00 smithi116 ceph-mon[37097]: pgmap v8843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:06:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:00 smithi171 ceph-mon[37789]: pgmap v8843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:06:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:00 smithi044 ceph-mon[32404]: pgmap v8843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:06:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:00.604+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:06:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:00.604+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:00.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:00.604+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:06:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:02 smithi171 ceph-mon[37789]: pgmap v8844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:06:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15906.026193 secs 2024-07-26T13:06:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:02 smithi044 ceph-mon[32404]: pgmap v8844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:06:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15906.026193 secs 2024-07-26T13:06:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:02 smithi116 ceph-mon[37097]: pgmap v8844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:06:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15906.026193 secs 2024-07-26T13:06:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:04 smithi171 ceph-mon[37789]: pgmap v8845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:06:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:04 smithi044 ceph-mon[32404]: pgmap v8845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:06:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:04 smithi116 ceph-mon[37097]: pgmap v8845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:06:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:06 smithi044 ceph-mon[32404]: pgmap v8846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:06 smithi116 ceph-mon[37097]: pgmap v8846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:06 smithi171 ceph-mon[37789]: pgmap v8846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15911.026328 secs 2024-07-26T13:06:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:07 smithi044 ceph-mon[32404]: pgmap v8847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15911.026328 secs 2024-07-26T13:06:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:07 smithi116 ceph-mon[37097]: pgmap v8847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15911.026328 secs 2024-07-26T13:06:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:07 smithi171 ceph-mon[37789]: pgmap v8847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:09 smithi044 ceph-mon[32404]: pgmap v8848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:09 smithi116 ceph-mon[37097]: pgmap v8848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:09 smithi171 ceph-mon[37789]: pgmap v8848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:11 smithi044 ceph-mon[32404]: pgmap v8849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:11 smithi116 ceph-mon[37097]: pgmap v8849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:11 smithi171 ceph-mon[37789]: pgmap v8849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15916.026466 secs 2024-07-26T13:06:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15916.026466 secs 2024-07-26T13:06:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15916.026466 secs 2024-07-26T13:06:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:13 smithi044 ceph-mon[32404]: pgmap v8850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:13 smithi116 ceph-mon[37097]: pgmap v8850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:13 smithi171 ceph-mon[37789]: pgmap v8850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:15 smithi044 ceph-mon[32404]: pgmap v8851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:15 smithi116 ceph-mon[37097]: pgmap v8851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:15 smithi171 ceph-mon[37789]: pgmap v8851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:15.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:15.610+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:06:15.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:15.610+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:15.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:15.610+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:06:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15921.026605 secs 2024-07-26T13:06:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:17 smithi044 ceph-mon[32404]: pgmap v8852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15921.026605 secs 2024-07-26T13:06:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:17 smithi116 ceph-mon[37097]: pgmap v8852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15921.026605 secs 2024-07-26T13:06:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:17 smithi171 ceph-mon[37789]: pgmap v8852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:19 smithi044 ceph-mon[32404]: pgmap v8853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:19 smithi116 ceph-mon[37097]: pgmap v8853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:19 smithi171 ceph-mon[37789]: pgmap v8853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:21 smithi044 ceph-mon[32404]: pgmap v8854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:21 smithi116 ceph-mon[37097]: pgmap v8854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:21 smithi171 ceph-mon[37789]: pgmap v8854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15926.026743 secs 2024-07-26T13:06:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15926.026743 secs 2024-07-26T13:06:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15926.026743 secs 2024-07-26T13:06:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:23 smithi044 ceph-mon[32404]: pgmap v8855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:23 smithi116 ceph-mon[37097]: pgmap v8855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:23 smithi171 ceph-mon[37789]: pgmap v8855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:25 smithi044 ceph-mon[32404]: pgmap v8856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:25.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:25 smithi116 ceph-mon[37097]: pgmap v8856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:25 smithi171 ceph-mon[37789]: pgmap v8856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15931.026880 secs 2024-07-26T13:06:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:27 smithi116 ceph-mon[37097]: pgmap v8857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15931.026880 secs 2024-07-26T13:06:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:27 smithi171 ceph-mon[37789]: pgmap v8857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15931.026880 secs 2024-07-26T13:06:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:27 smithi044 ceph-mon[32404]: pgmap v8857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:29 smithi171 ceph-mon[37789]: pgmap v8858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:29 smithi044 ceph-mon[32404]: pgmap v8858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:29 smithi116 ceph-mon[37097]: pgmap v8858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:30.610+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:06:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:30.610+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:30.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:30.610+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:06:31.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:31 smithi116 ceph-mon[37097]: pgmap v8859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:31 smithi171 ceph-mon[37789]: pgmap v8859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:31 smithi044 ceph-mon[32404]: pgmap v8859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15936.027019 secs 2024-07-26T13:06:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15936.027019 secs 2024-07-26T13:06:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15936.027019 secs 2024-07-26T13:06:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:33 smithi171 ceph-mon[37789]: pgmap v8860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:33 smithi044 ceph-mon[32404]: pgmap v8860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:33 smithi116 ceph-mon[37097]: pgmap v8860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:35 smithi171 ceph-mon[37789]: pgmap v8861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:35 smithi044 ceph-mon[32404]: pgmap v8861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:35 smithi116 ceph-mon[37097]: pgmap v8861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15941.027151 secs 2024-07-26T13:06:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:37 smithi171 ceph-mon[37789]: pgmap v8862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.601 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15941.027151 secs 2024-07-26T13:06:37.601 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:37 smithi044 ceph-mon[32404]: pgmap v8862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15941.027151 secs 2024-07-26T13:06:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:37 smithi116 ceph-mon[37097]: pgmap v8862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:39 smithi171 ceph-mon[37789]: pgmap v8863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:39 smithi044 ceph-mon[32404]: pgmap v8863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:39 smithi116 ceph-mon[37097]: pgmap v8863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:41 smithi171 ceph-mon[37789]: pgmap v8864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:41 smithi044 ceph-mon[32404]: pgmap v8864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:41 smithi116 ceph-mon[37097]: pgmap v8864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15946.027287 secs 2024-07-26T13:06:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15946.027287 secs 2024-07-26T13:06:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15946.027287 secs 2024-07-26T13:06:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:43 smithi171 ceph-mon[37789]: pgmap v8865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:43 smithi044 ceph-mon[32404]: pgmap v8865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:43.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:43 smithi116 ceph-mon[37097]: pgmap v8865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:44 smithi044 ceph-mon[32404]: pgmap v8866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:06:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:44 smithi044 ceph-mon[32404]: pgmap v8867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:44.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:06:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:44 smithi116 ceph-mon[37097]: pgmap v8866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:06:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:44 smithi116 ceph-mon[37097]: pgmap v8867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:06:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:44 smithi171 ceph-mon[37789]: pgmap v8866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:06:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:44 smithi171 ceph-mon[37789]: pgmap v8867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:44.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:06:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:45.616+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:06:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:45.616+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:45.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:06:45.616+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:06:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:46 smithi044 ceph-mon[32404]: pgmap v8868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:46 smithi116 ceph-mon[37097]: pgmap v8868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:46.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:46 smithi171 ceph-mon[37789]: pgmap v8868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15951.027448 secs 2024-07-26T13:06:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15951.027448 secs 2024-07-26T13:06:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15951.027448 secs 2024-07-26T13:06:48.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:48 smithi044 ceph-mon[32404]: pgmap v8869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:48 smithi116 ceph-mon[37097]: pgmap v8869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:48 smithi171 ceph-mon[37789]: pgmap v8869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:50 smithi044 ceph-mon[32404]: pgmap v8870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:50 smithi116 ceph-mon[37097]: pgmap v8870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:50 smithi171 ceph-mon[37789]: pgmap v8870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15956.027600 secs 2024-07-26T13:06:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:52 smithi044 ceph-mon[32404]: pgmap v8871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15956.027600 secs 2024-07-26T13:06:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:52 smithi116 ceph-mon[37097]: pgmap v8871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15956.027600 secs 2024-07-26T13:06:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:52 smithi171 ceph-mon[37789]: pgmap v8871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:06:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:54 smithi044 ceph-mon[32404]: pgmap v8872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:54 smithi116 ceph-mon[37097]: pgmap v8872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:54 smithi171 ceph-mon[37789]: pgmap v8872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:56 smithi044 ceph-mon[32404]: pgmap v8873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15961.027717 secs 2024-07-26T13:06:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:56 smithi116 ceph-mon[37097]: pgmap v8873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15961.027717 secs 2024-07-26T13:06:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:56 smithi171 ceph-mon[37789]: pgmap v8873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15961.027717 secs 2024-07-26T13:06:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:06:58 smithi044 ceph-mon[32404]: pgmap v8874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:06:58 smithi116 ceph-mon[37097]: pgmap v8874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:06:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:06:58 smithi171 ceph-mon[37789]: pgmap v8874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:00.618+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:07:00.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:00.618+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:00.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:00.618+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:07:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:00 smithi044 ceph-mon[32404]: pgmap v8875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:00 smithi116 ceph-mon[37097]: pgmap v8875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:00 smithi171 ceph-mon[37789]: pgmap v8875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15966.027881 secs 2024-07-26T13:07:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15966.027881 secs 2024-07-26T13:07:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15966.027881 secs 2024-07-26T13:07:02.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:02 smithi171 ceph-mon[37789]: pgmap v8876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:02 smithi044 ceph-mon[32404]: pgmap v8876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:02 smithi116 ceph-mon[37097]: pgmap v8876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:04 smithi044 ceph-mon[32404]: pgmap v8877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:04 smithi116 ceph-mon[37097]: pgmap v8877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:04 smithi171 ceph-mon[37789]: pgmap v8877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:06 smithi044 ceph-mon[32404]: pgmap v8878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15971.028010 secs 2024-07-26T13:07:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:06 smithi116 ceph-mon[37097]: pgmap v8878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15971.028010 secs 2024-07-26T13:07:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:06 smithi171 ceph-mon[37789]: pgmap v8878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15971.028010 secs 2024-07-26T13:07:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:08 smithi044 ceph-mon[32404]: pgmap v8879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:08 smithi116 ceph-mon[37097]: pgmap v8879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:08 smithi171 ceph-mon[37789]: pgmap v8879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:10 smithi044 ceph-mon[32404]: pgmap v8880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:10 smithi116 ceph-mon[37097]: pgmap v8880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:10 smithi171 ceph-mon[37789]: pgmap v8880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15976.028160 secs 2024-07-26T13:07:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15976.028160 secs 2024-07-26T13:07:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15976.028160 secs 2024-07-26T13:07:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:12 smithi044 ceph-mon[32404]: pgmap v8881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:07:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:12 smithi116 ceph-mon[37097]: pgmap v8881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:07:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:12 smithi171 ceph-mon[37789]: pgmap v8881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:07:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:14 smithi044 ceph-mon[32404]: pgmap v8882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:14 smithi116 ceph-mon[37097]: pgmap v8882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:14 smithi171 ceph-mon[37789]: pgmap v8882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:15.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:15.622+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:07:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:15.622+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:07:15.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:15.622+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:16 smithi044 ceph-mon[32404]: pgmap v8883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15981.028308 secs 2024-07-26T13:07:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:16 smithi116 ceph-mon[37097]: pgmap v8883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15981.028308 secs 2024-07-26T13:07:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:16 smithi171 ceph-mon[37789]: pgmap v8883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15981.028308 secs 2024-07-26T13:07:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:18 smithi116 ceph-mon[37097]: pgmap v8884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:18 smithi171 ceph-mon[37789]: pgmap v8884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:18 smithi044 ceph-mon[32404]: pgmap v8884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:20 smithi116 ceph-mon[37097]: pgmap v8885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:20 smithi171 ceph-mon[37789]: pgmap v8885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:20 smithi044 ceph-mon[32404]: pgmap v8885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15986.028439 secs 2024-07-26T13:07:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15986.028439 secs 2024-07-26T13:07:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15986.028439 secs 2024-07-26T13:07:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:22 smithi171 ceph-mon[37789]: pgmap v8886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:22 smithi044 ceph-mon[32404]: pgmap v8886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:22 smithi116 ceph-mon[37097]: pgmap v8886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:24 smithi044 ceph-mon[32404]: pgmap v8887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:24 smithi171 ceph-mon[37789]: pgmap v8887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:24 smithi116 ceph-mon[37097]: pgmap v8887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:26 smithi171 ceph-mon[37789]: pgmap v8888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15991.028577 secs 2024-07-26T13:07:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:26 smithi044 ceph-mon[32404]: pgmap v8888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15991.028577 secs 2024-07-26T13:07:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:26 smithi116 ceph-mon[37097]: pgmap v8888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15991.028577 secs 2024-07-26T13:07:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:28 smithi171 ceph-mon[37789]: pgmap v8889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:28 smithi044 ceph-mon[32404]: pgmap v8889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:28 smithi116 ceph-mon[37097]: pgmap v8889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:30.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:30.622+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:07:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:30.622+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:07:30.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:30.622+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:30 smithi044 ceph-mon[32404]: pgmap v8890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:31.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:30 smithi116 ceph-mon[37097]: pgmap v8890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:30 smithi171 ceph-mon[37789]: pgmap v8890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 15996.028713 secs 2024-07-26T13:07:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 15996.028713 secs 2024-07-26T13:07:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 15996.028713 secs 2024-07-26T13:07:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:32 smithi171 ceph-mon[37789]: pgmap v8891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:32 smithi044 ceph-mon[32404]: pgmap v8891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:32 smithi116 ceph-mon[37097]: pgmap v8891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:34 smithi171 ceph-mon[37789]: pgmap v8892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:34 smithi044 ceph-mon[32404]: pgmap v8892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:34 smithi116 ceph-mon[37097]: pgmap v8892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:36 smithi044 ceph-mon[32404]: pgmap v8893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16001.028848 secs 2024-07-26T13:07:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:36 smithi116 ceph-mon[37097]: pgmap v8893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16001.028848 secs 2024-07-26T13:07:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:36 smithi171 ceph-mon[37789]: pgmap v8893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16001.028848 secs 2024-07-26T13:07:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:38 smithi044 ceph-mon[32404]: pgmap v8894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:38 smithi116 ceph-mon[37097]: pgmap v8894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:38 smithi171 ceph-mon[37789]: pgmap v8894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:07:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:40 smithi044 ceph-mon[32404]: pgmap v8895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:40 smithi116 ceph-mon[37097]: pgmap v8895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:40 smithi171 ceph-mon[37789]: pgmap v8895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16006.028939 secs 2024-07-26T13:07:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16006.028939 secs 2024-07-26T13:07:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16006.028939 secs 2024-07-26T13:07:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:42 smithi044 ceph-mon[32404]: pgmap v8896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:42 smithi116 ceph-mon[37097]: pgmap v8896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:42 smithi171 ceph-mon[37789]: pgmap v8896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:44 smithi044 ceph-mon[32404]: pgmap v8897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:44 smithi116 ceph-mon[37097]: pgmap v8897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:07:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:44 smithi171 ceph-mon[37789]: pgmap v8897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:07:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:45.625+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:07:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:45.625+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:07:45.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:07:45.625+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:45 smithi044 ceph-mon[32404]: pgmap v8898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:07:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:45 smithi044 ceph-mon[32404]: pgmap v8899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:45 smithi116 ceph-mon[37097]: pgmap v8898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:07:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:45 smithi116 ceph-mon[37097]: pgmap v8899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:45 smithi171 ceph-mon[37789]: pgmap v8898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:07:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:45 smithi171 ceph-mon[37789]: pgmap v8899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:47.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16011.029074 secs 2024-07-26T13:07:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16011.029074 secs 2024-07-26T13:07:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16011.029074 secs 2024-07-26T13:07:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:47 smithi044 ceph-mon[32404]: pgmap v8900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:47 smithi116 ceph-mon[37097]: pgmap v8900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:47 smithi171 ceph-mon[37789]: pgmap v8900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:49 smithi044 ceph-mon[32404]: pgmap v8901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:49 smithi116 ceph-mon[37097]: pgmap v8901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:49 smithi171 ceph-mon[37789]: pgmap v8901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16016.029196 secs 2024-07-26T13:07:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:51 smithi044 ceph-mon[32404]: pgmap v8902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:07:52.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16016.029196 secs 2024-07-26T13:07:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:51 smithi116 ceph-mon[37097]: pgmap v8902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:07:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16016.029196 secs 2024-07-26T13:07:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:51 smithi171 ceph-mon[37789]: pgmap v8902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:07:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:53 smithi044 ceph-mon[32404]: pgmap v8903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:54.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:53 smithi116 ceph-mon[37097]: pgmap v8903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:53 smithi171 ceph-mon[37789]: pgmap v8903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:55 smithi044 ceph-mon[32404]: pgmap v8904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:55 smithi116 ceph-mon[37097]: pgmap v8904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:55 smithi171 ceph-mon[37789]: pgmap v8904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16021.029337 secs 2024-07-26T13:07:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16021.029337 secs 2024-07-26T13:07:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16021.029337 secs 2024-07-26T13:07:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:57 smithi044 ceph-mon[32404]: pgmap v8905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:57 smithi116 ceph-mon[37097]: pgmap v8905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:07:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:57 smithi171 ceph-mon[37789]: pgmap v8905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:07:59 smithi044 ceph-mon[32404]: pgmap v8906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:07:59 smithi116 ceph-mon[37097]: pgmap v8906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:07:59 smithi171 ceph-mon[37789]: pgmap v8906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:00.626+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:08:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:00.626+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:08:00.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:00.626+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16026.029471 secs 2024-07-26T13:08:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:01 smithi044 ceph-mon[32404]: pgmap v8907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:08:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16026.029471 secs 2024-07-26T13:08:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:01 smithi116 ceph-mon[37097]: pgmap v8907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:08:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16026.029471 secs 2024-07-26T13:08:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:01 smithi171 ceph-mon[37789]: pgmap v8907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:08:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:03 smithi044 ceph-mon[32404]: pgmap v8908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:03 smithi116 ceph-mon[37097]: pgmap v8908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:03 smithi171 ceph-mon[37789]: pgmap v8908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:05 smithi044 ceph-mon[32404]: pgmap v8909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:05 smithi116 ceph-mon[37097]: pgmap v8909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:05 smithi171 ceph-mon[37789]: pgmap v8909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16031.029664 secs 2024-07-26T13:08:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16031.029664 secs 2024-07-26T13:08:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16031.029664 secs 2024-07-26T13:08:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:07 smithi044 ceph-mon[32404]: pgmap v8910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:07 smithi116 ceph-mon[37097]: pgmap v8910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:07 smithi171 ceph-mon[37789]: pgmap v8910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:09 smithi044 ceph-mon[32404]: pgmap v8911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:09 smithi116 ceph-mon[37097]: pgmap v8911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:09 smithi171 ceph-mon[37789]: pgmap v8911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:12.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16036.029825 secs 2024-07-26T13:08:12.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:12 smithi044 ceph-mon[32404]: pgmap v8912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:08:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16036.029825 secs 2024-07-26T13:08:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:12 smithi116 ceph-mon[37097]: pgmap v8912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:08:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16036.029825 secs 2024-07-26T13:08:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:12 smithi171 ceph-mon[37789]: pgmap v8912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:08:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:14 smithi044 ceph-mon[32404]: pgmap v8913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:14 smithi116 ceph-mon[37097]: pgmap v8913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:14 smithi171 ceph-mon[37789]: pgmap v8913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:15.632+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:08:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:15.632+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:08:15.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:15.632+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:16 smithi044 ceph-mon[32404]: pgmap v8914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:08:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:16 smithi116 ceph-mon[37097]: pgmap v8914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:08:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:16 smithi171 ceph-mon[37789]: pgmap v8914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:08:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16041.029981 secs 2024-07-26T13:08:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16041.029981 secs 2024-07-26T13:08:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16041.029981 secs 2024-07-26T13:08:17.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:17.574+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:08:17.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:17.574+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:08:17.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:17.574+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:08:17.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:17.574+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:08:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:18 smithi171 ceph-mon[37789]: pgmap v8915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:18 smithi044 ceph-mon[32404]: pgmap v8915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:18.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:18 smithi116 ceph-mon[37097]: pgmap v8915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:08:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:20 smithi171 ceph-mon[37789]: pgmap v8916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:20 smithi044 ceph-mon[32404]: pgmap v8916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:20 smithi116 ceph-mon[37097]: pgmap v8916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16046.030147 secs 2024-07-26T13:08:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:22 smithi171 ceph-mon[37789]: pgmap v8917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16046.030147 secs 2024-07-26T13:08:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:22 smithi044 ceph-mon[32404]: pgmap v8917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16046.030147 secs 2024-07-26T13:08:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:22 smithi116 ceph-mon[37097]: pgmap v8917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:24 smithi171 ceph-mon[37789]: pgmap v8918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:24 smithi044 ceph-mon[32404]: pgmap v8918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:24 smithi116 ceph-mon[37097]: pgmap v8918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:26 smithi171 ceph-mon[37789]: pgmap v8919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:26 smithi044 ceph-mon[32404]: pgmap v8919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:26 smithi116 ceph-mon[37097]: pgmap v8919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16051.030231 secs 2024-07-26T13:08:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16051.030231 secs 2024-07-26T13:08:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16051.030231 secs 2024-07-26T13:08:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:28 smithi171 ceph-mon[37789]: pgmap v8920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:28 smithi044 ceph-mon[32404]: pgmap v8920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:28 smithi116 ceph-mon[37097]: pgmap v8920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:30 smithi171 ceph-mon[37789]: pgmap v8921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:30 smithi044 ceph-mon[32404]: pgmap v8921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:30.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:30 smithi116 ceph-mon[37097]: pgmap v8921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:30.632+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:08:30.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:30.632+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:08:30.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:30.632+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16056.030393 secs 2024-07-26T13:08:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:32 smithi171 ceph-mon[37789]: pgmap v8922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16056.030393 secs 2024-07-26T13:08:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:32 smithi044 ceph-mon[32404]: pgmap v8922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16056.030393 secs 2024-07-26T13:08:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:32 smithi116 ceph-mon[37097]: pgmap v8922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:34 smithi171 ceph-mon[37789]: pgmap v8923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:34 smithi044 ceph-mon[32404]: pgmap v8923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:34 smithi116 ceph-mon[37097]: pgmap v8923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:36 smithi044 ceph-mon[32404]: pgmap v8924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:36 smithi116 ceph-mon[37097]: pgmap v8924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:36 smithi171 ceph-mon[37789]: pgmap v8924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16061.030549 secs 2024-07-26T13:08:37.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16061.030549 secs 2024-07-26T13:08:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16061.030549 secs 2024-07-26T13:08:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:38 smithi044 ceph-mon[32404]: pgmap v8925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:38 smithi116 ceph-mon[37097]: pgmap v8925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:38 smithi171 ceph-mon[37789]: pgmap v8925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:40 smithi044 ceph-mon[32404]: pgmap v8926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:40 smithi116 ceph-mon[37097]: pgmap v8926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:40 smithi171 ceph-mon[37789]: pgmap v8926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:42.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16066.030706 secs 2024-07-26T13:08:42.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:42 smithi044 ceph-mon[32404]: pgmap v8927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16066.030706 secs 2024-07-26T13:08:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:42 smithi116 ceph-mon[37097]: pgmap v8927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16066.030706 secs 2024-07-26T13:08:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:42 smithi171 ceph-mon[37789]: pgmap v8927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:44 smithi044 ceph-mon[32404]: pgmap v8928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:44.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:44 smithi116 ceph-mon[37097]: pgmap v8928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:44 smithi171 ceph-mon[37789]: pgmap v8928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:45.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:45.638+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:08:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:45.638+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:08:45.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:08:45.638+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:46 smithi044 ceph-mon[32404]: pgmap v8929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:46 smithi044 ceph-mon[32404]: pgmap v8930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:46 smithi044 ceph-mon[32404]: pgmap v8931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:46.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:46 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:08:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:46 smithi116 ceph-mon[37097]: pgmap v8929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:46 smithi116 ceph-mon[37097]: pgmap v8930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:46 smithi116 ceph-mon[37097]: pgmap v8931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:46.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:46 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:08:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:46 smithi171 ceph-mon[37789]: pgmap v8929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:08:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:46 smithi171 ceph-mon[37789]: pgmap v8930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:46.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:46 smithi171 ceph-mon[37789]: pgmap v8931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:46.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:46 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:08:47.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16071.030800 secs 2024-07-26T13:08:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16071.030800 secs 2024-07-26T13:08:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16071.030800 secs 2024-07-26T13:08:48.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:48 smithi044 ceph-mon[32404]: pgmap v8932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:48 smithi116 ceph-mon[37097]: pgmap v8932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:48 smithi171 ceph-mon[37789]: pgmap v8932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:51 smithi044 ceph-mon[32404]: pgmap v8933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:51 smithi116 ceph-mon[37097]: pgmap v8933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:51 smithi171 ceph-mon[37789]: pgmap v8933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16076.030906 secs 2024-07-26T13:08:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16076.030906 secs 2024-07-26T13:08:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16076.030906 secs 2024-07-26T13:08:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:53 smithi116 ceph-mon[37097]: pgmap v8934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:53 smithi171 ceph-mon[37789]: pgmap v8934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:53 smithi044 ceph-mon[32404]: pgmap v8934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:55 smithi116 ceph-mon[37097]: pgmap v8935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:55 smithi171 ceph-mon[37789]: pgmap v8935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:55 smithi044 ceph-mon[32404]: pgmap v8935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16081.031082 secs 2024-07-26T13:08:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:57 smithi171 ceph-mon[37789]: pgmap v8936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16081.031082 secs 2024-07-26T13:08:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:57 smithi044 ceph-mon[32404]: pgmap v8936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16081.031082 secs 2024-07-26T13:08:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:57 smithi116 ceph-mon[37097]: pgmap v8936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:08:59 smithi171 ceph-mon[37789]: pgmap v8937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:08:59 smithi044 ceph-mon[32404]: pgmap v8937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:08:59 smithi116 ceph-mon[37097]: pgmap v8937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:00.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:00.639+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:09:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:00.639+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:09:00.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:00.639+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:01 smithi171 ceph-mon[37789]: pgmap v8938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:01 smithi044 ceph-mon[32404]: pgmap v8938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:01 smithi116 ceph-mon[37097]: pgmap v8938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16086.031244 secs 2024-07-26T13:09:02.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16086.031244 secs 2024-07-26T13:09:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16086.031244 secs 2024-07-26T13:09:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:03 smithi171 ceph-mon[37789]: pgmap v8939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:03 smithi044 ceph-mon[32404]: pgmap v8939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:03 smithi116 ceph-mon[37097]: pgmap v8939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:05 smithi171 ceph-mon[37789]: pgmap v8940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:05 smithi044 ceph-mon[32404]: pgmap v8940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:05 smithi116 ceph-mon[37097]: pgmap v8940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16091.031386 secs 2024-07-26T13:09:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:07 smithi171 ceph-mon[37789]: pgmap v8941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16091.031386 secs 2024-07-26T13:09:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:07 smithi044 ceph-mon[32404]: pgmap v8941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16091.031386 secs 2024-07-26T13:09:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:07 smithi116 ceph-mon[37097]: pgmap v8941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:09 smithi171 ceph-mon[37789]: pgmap v8942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:09 smithi044 ceph-mon[32404]: pgmap v8942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:09 smithi116 ceph-mon[37097]: pgmap v8942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:11 smithi171 ceph-mon[37789]: pgmap v8943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:11 smithi044 ceph-mon[32404]: pgmap v8943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:11 smithi116 ceph-mon[37097]: pgmap v8943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16096.031537 secs 2024-07-26T13:09:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16096.031537 secs 2024-07-26T13:09:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16096.031537 secs 2024-07-26T13:09:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:13 smithi044 ceph-mon[32404]: pgmap v8944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:13 smithi116 ceph-mon[37097]: pgmap v8944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:13 smithi171 ceph-mon[37789]: pgmap v8944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:15 smithi044 ceph-mon[32404]: pgmap v8945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:15 smithi116 ceph-mon[37097]: pgmap v8945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:15 smithi171 ceph-mon[37789]: pgmap v8945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:15.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:15.645+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:09:15.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:15.645+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:15.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:15.645+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:09:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16101.031699 secs 2024-07-26T13:09:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:17 smithi044 ceph-mon[32404]: pgmap v8946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:17.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16101.031699 secs 2024-07-26T13:09:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:17 smithi116 ceph-mon[37097]: pgmap v8946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16101.031699 secs 2024-07-26T13:09:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:17 smithi171 ceph-mon[37789]: pgmap v8946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:19 smithi044 ceph-mon[32404]: pgmap v8947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:19 smithi116 ceph-mon[37097]: pgmap v8947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:19 smithi171 ceph-mon[37789]: pgmap v8947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:21 smithi044 ceph-mon[32404]: pgmap v8948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:21 smithi116 ceph-mon[37097]: pgmap v8948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:21 smithi171 ceph-mon[37789]: pgmap v8948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16106.031798 secs 2024-07-26T13:09:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16106.031798 secs 2024-07-26T13:09:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16106.031798 secs 2024-07-26T13:09:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:23 smithi044 ceph-mon[32404]: pgmap v8949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:23.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:23 smithi116 ceph-mon[37097]: pgmap v8949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:23 smithi171 ceph-mon[37789]: pgmap v8949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:24 smithi044 ceph-mon[32404]: pgmap v8950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:24 smithi116 ceph-mon[37097]: pgmap v8950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:24 smithi171 ceph-mon[37789]: pgmap v8950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16111.031929 secs 2024-07-26T13:09:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:27 smithi171 ceph-mon[37789]: pgmap v8951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16111.031929 secs 2024-07-26T13:09:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:27 smithi044 ceph-mon[32404]: pgmap v8951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16111.031929 secs 2024-07-26T13:09:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:27 smithi116 ceph-mon[37097]: pgmap v8951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:29 smithi171 ceph-mon[37789]: pgmap v8952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:29 smithi044 ceph-mon[32404]: pgmap v8952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:29 smithi116 ceph-mon[37097]: pgmap v8952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:30.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:30.646+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:09:30.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:30.646+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:30.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:30.646+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:09:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:31 smithi171 ceph-mon[37789]: pgmap v8953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:31 smithi044 ceph-mon[32404]: pgmap v8953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:31 smithi116 ceph-mon[37097]: pgmap v8953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16116.032095 secs 2024-07-26T13:09:32.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16116.032095 secs 2024-07-26T13:09:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16116.032095 secs 2024-07-26T13:09:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:33 smithi171 ceph-mon[37789]: pgmap v8954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:33 smithi044 ceph-mon[32404]: pgmap v8954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:33 smithi116 ceph-mon[37097]: pgmap v8954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:35 smithi171 ceph-mon[37789]: pgmap v8955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:35 smithi044 ceph-mon[32404]: pgmap v8955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:35 smithi116 ceph-mon[37097]: pgmap v8955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16121.032235 secs 2024-07-26T13:09:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:37 smithi171 ceph-mon[37789]: pgmap v8956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16121.032235 secs 2024-07-26T13:09:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:37 smithi044 ceph-mon[32404]: pgmap v8956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16121.032235 secs 2024-07-26T13:09:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:37 smithi116 ceph-mon[37097]: pgmap v8956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:39 smithi171 ceph-mon[37789]: pgmap v8957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:39 smithi044 ceph-mon[32404]: pgmap v8957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:39.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:39 smithi116 ceph-mon[37097]: pgmap v8957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:41 smithi171 ceph-mon[37789]: pgmap v8958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:41 smithi044 ceph-mon[32404]: pgmap v8958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:41 smithi116 ceph-mon[37097]: pgmap v8958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16126.032406 secs 2024-07-26T13:09:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16126.032406 secs 2024-07-26T13:09:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16126.032406 secs 2024-07-26T13:09:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:43 smithi044 ceph-mon[32404]: pgmap v8959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:43 smithi116 ceph-mon[37097]: pgmap v8959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:43 smithi171 ceph-mon[37789]: pgmap v8959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:45 smithi044 ceph-mon[32404]: pgmap v8960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:45 smithi116 ceph-mon[37097]: pgmap v8960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:45 smithi171 ceph-mon[37789]: pgmap v8960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:45.648 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:45.647+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:09:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:45.647+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:09:45.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:09:45.647+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:47.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16131.032566 secs 2024-07-26T13:09:47.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:47 smithi116 ceph-mon[37097]: pgmap v8961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:47.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:47 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16131.032566 secs 2024-07-26T13:09:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:47 smithi044 ceph-mon[32404]: pgmap v8961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:47 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16131.032566 secs 2024-07-26T13:09:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:47 smithi171 ceph-mon[37789]: pgmap v8961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:47 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:49.392 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:49 smithi116 ceph-mon[37097]: pgmap v8962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:49.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:49 smithi044 ceph-mon[32404]: pgmap v8962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:49 smithi171 ceph-mon[37789]: pgmap v8962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:50 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:50 smithi044 ceph-mon[32404]: pgmap v8963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:51.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:50 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:50 smithi116 ceph-mon[37097]: pgmap v8963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:50 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:50 smithi171 ceph-mon[37789]: pgmap v8963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:09:52.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16136.032671 secs 2024-07-26T13:09:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16136.032671 secs 2024-07-26T13:09:52.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16136.032671 secs 2024-07-26T13:09:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:52 smithi044 ceph-mon[32404]: pgmap v8964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:52 smithi116 ceph-mon[37097]: pgmap v8964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:52 smithi171 ceph-mon[37789]: pgmap v8964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:09:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: pgmap v8965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: pgmap v8966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:54.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:09:54.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:54.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:54.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: pgmap v8965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:09:54.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: pgmap v8966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:54.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:54.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: pgmap v8965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:09:54.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: pgmap v8966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:54.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:56 smithi044 ceph-mon[32404]: pgmap v8967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:56.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:56 smithi116 ceph-mon[37097]: pgmap v8967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:56 smithi171 ceph-mon[37789]: pgmap v8967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16141.032834 secs 2024-07-26T13:09:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16141.032834 secs 2024-07-26T13:09:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16141.032834 secs 2024-07-26T13:09:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:58 smithi044 ceph-mon[32404]: pgmap v8968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:58 smithi116 ceph-mon[37097]: pgmap v8968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:58 smithi171 ceph-mon[37789]: pgmap v8968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:09:59 smithi044 ceph-mon[32404]: pgmap v8969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:09:59 smithi116 ceph-mon[37097]: pgmap v8969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:09:59 smithi171 ceph-mon[37789]: pgmap v8969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:00.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:00.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:00.648+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:10:00.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:00.648+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:10:00.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:00.648+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:01 smithi044 ceph-mon[32404]: pgmap v8970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:01 smithi116 ceph-mon[37097]: pgmap v8970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:01 smithi171 ceph-mon[37789]: pgmap v8970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:02.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16146.032995 secs 2024-07-26T13:10:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16146.032995 secs 2024-07-26T13:10:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16146.032995 secs 2024-07-26T13:10:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:03 smithi044 ceph-mon[32404]: pgmap v8971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:03 smithi116 ceph-mon[37097]: pgmap v8971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:03 smithi171 ceph-mon[37789]: pgmap v8971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:06 smithi044 ceph-mon[32404]: pgmap v8972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:06 smithi116 ceph-mon[37097]: pgmap v8972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:06 smithi171 ceph-mon[37789]: pgmap v8972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16151.033153 secs 2024-07-26T13:10:07.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16151.033153 secs 2024-07-26T13:10:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16151.033153 secs 2024-07-26T13:10:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:08 smithi044 ceph-mon[32404]: pgmap v8973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:08 smithi116 ceph-mon[37097]: pgmap v8973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:08 smithi171 ceph-mon[37789]: pgmap v8973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:09 smithi044 ceph-mon[32404]: pgmap v8974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:09 smithi116 ceph-mon[37097]: pgmap v8974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:09 smithi171 ceph-mon[37789]: pgmap v8974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16156.033308 secs 2024-07-26T13:10:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:12 smithi044 ceph-mon[32404]: pgmap v8975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:12.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16156.033308 secs 2024-07-26T13:10:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:12 smithi116 ceph-mon[37097]: pgmap v8975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16156.033308 secs 2024-07-26T13:10:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:12 smithi171 ceph-mon[37789]: pgmap v8975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:14 smithi044 ceph-mon[32404]: pgmap v8976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:14 smithi116 ceph-mon[37097]: pgmap v8976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:14 smithi171 ceph-mon[37789]: pgmap v8976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:15.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:15.649+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:10:15.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:15.649+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:15.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:15.649+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:10:16.420 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:16 smithi044 ceph-mon[32404]: pgmap v8977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:16 smithi116 ceph-mon[37097]: pgmap v8977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:16 smithi171 ceph-mon[37789]: pgmap v8977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16161.033469 secs 2024-07-26T13:10:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:17 smithi044 ceph-mon[32404]: pgmap v8978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16161.033469 secs 2024-07-26T13:10:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:17 smithi116 ceph-mon[37097]: pgmap v8978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16161.033469 secs 2024-07-26T13:10:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:17 smithi171 ceph-mon[37789]: pgmap v8978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:20.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:20 smithi044 ceph-mon[32404]: pgmap v8979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:20 smithi116 ceph-mon[37097]: pgmap v8979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:20 smithi171 ceph-mon[37789]: pgmap v8979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16166.033654 secs 2024-07-26T13:10:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:22 smithi044 ceph-mon[32404]: pgmap v8980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16166.033654 secs 2024-07-26T13:10:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:22 smithi116 ceph-mon[37097]: pgmap v8980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16166.033654 secs 2024-07-26T13:10:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:22 smithi171 ceph-mon[37789]: pgmap v8980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:23 smithi044 ceph-mon[32404]: pgmap v8981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:23 smithi116 ceph-mon[37097]: pgmap v8981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:23 smithi171 ceph-mon[37789]: pgmap v8981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:26 smithi044 ceph-mon[32404]: pgmap v8982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:26 smithi116 ceph-mon[37097]: pgmap v8982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:26 smithi171 ceph-mon[37789]: pgmap v8982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16171.033666 secs 2024-07-26T13:10:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16171.033666 secs 2024-07-26T13:10:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16171.033666 secs 2024-07-26T13:10:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:28 smithi044 ceph-mon[32404]: pgmap v8983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:28.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:28 smithi116 ceph-mon[37097]: pgmap v8983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:28 smithi171 ceph-mon[37789]: pgmap v8983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:29 smithi044 ceph-mon[32404]: pgmap v8984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:29.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:29 smithi116 ceph-mon[37097]: pgmap v8984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:29 smithi171 ceph-mon[37789]: pgmap v8984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:30.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:30.650+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:10:30.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:30.650+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:10:30.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:30.650+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:32.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16176.033830 secs 2024-07-26T13:10:32.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:32 smithi116 ceph-mon[37097]: pgmap v8985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16176.033830 secs 2024-07-26T13:10:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:32 smithi044 ceph-mon[32404]: pgmap v8985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16176.033830 secs 2024-07-26T13:10:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:32 smithi171 ceph-mon[37789]: pgmap v8985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:33 smithi044 ceph-mon[32404]: pgmap v8986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:33.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:33 smithi116 ceph-mon[37097]: pgmap v8986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:33 smithi171 ceph-mon[37789]: pgmap v8986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:36 smithi044 ceph-mon[32404]: pgmap v8987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:36.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:36 smithi116 ceph-mon[37097]: pgmap v8987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:36 smithi171 ceph-mon[37789]: pgmap v8987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16181.033925 secs 2024-07-26T13:10:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:37 smithi044 ceph-mon[32404]: pgmap v8988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16181.033925 secs 2024-07-26T13:10:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:37 smithi116 ceph-mon[37097]: pgmap v8988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16181.033925 secs 2024-07-26T13:10:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:37 smithi171 ceph-mon[37789]: pgmap v8988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:40 smithi044 ceph-mon[32404]: pgmap v8989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:40.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:40 smithi116 ceph-mon[37097]: pgmap v8989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:40 smithi171 ceph-mon[37789]: pgmap v8989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:41 smithi044 ceph-mon[32404]: pgmap v8990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:41 smithi116 ceph-mon[37097]: pgmap v8990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:41 smithi171 ceph-mon[37789]: pgmap v8990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16186.034073 secs 2024-07-26T13:10:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16186.034073 secs 2024-07-26T13:10:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16186.034073 secs 2024-07-26T13:10:43.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:43 smithi116 ceph-mon[37097]: pgmap v8991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:43 smithi044 ceph-mon[32404]: pgmap v8991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:43 smithi171 ceph-mon[37789]: pgmap v8991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:45 smithi044 ceph-mon[32404]: pgmap v8992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:45 smithi116 ceph-mon[37097]: pgmap v8992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:45 smithi171 ceph-mon[37789]: pgmap v8992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:45.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:45.655+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:10:45.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:45.655+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:10:45.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:10:45.655+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16191.034236 secs 2024-07-26T13:10:46.945 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16191.034236 secs 2024-07-26T13:10:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16191.034236 secs 2024-07-26T13:10:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:47 smithi044 ceph-mon[32404]: pgmap v8993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:47 smithi116 ceph-mon[37097]: pgmap v8993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:47 smithi171 ceph-mon[37789]: pgmap v8993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:49 smithi044 ceph-mon[32404]: pgmap v8994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:49 smithi116 ceph-mon[37097]: pgmap v8994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:49 smithi171 ceph-mon[37789]: pgmap v8994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:51 smithi044 ceph-mon[32404]: pgmap v8995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:51 smithi116 ceph-mon[37097]: pgmap v8995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:51 smithi171 ceph-mon[37789]: pgmap v8995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16196.034362 secs 2024-07-26T13:10:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16196.034362 secs 2024-07-26T13:10:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16196.034362 secs 2024-07-26T13:10:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:53 smithi171 ceph-mon[37789]: pgmap v8996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:54.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:53 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:53 smithi044 ceph-mon[32404]: pgmap v8996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:54.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:53 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:53 smithi116 ceph-mon[37097]: pgmap v8996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:53 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:10:55.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:54 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:10:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:10:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:54 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:10:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:54 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:10:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:55 smithi171 ceph-mon[37789]: pgmap v8997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:55 smithi044 ceph-mon[32404]: pgmap v8997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:55 smithi116 ceph-mon[37097]: pgmap v8997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16201.034516 secs 2024-07-26T13:10:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16201.034516 secs 2024-07-26T13:10:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16201.034516 secs 2024-07-26T13:10:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: pgmap v8998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:10:58.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:10:58.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:58.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:58.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: pgmap v8998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:10:58.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:10:58.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:58.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:58.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: pgmap v8998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:10:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:10:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:10:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:10:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:58.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:59 smithi044 ceph-mon[32404]: pgmap v8999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:59.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:10:59 smithi044 ceph-mon[32404]: pgmap v9000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:59 smithi116 ceph-mon[37097]: pgmap v8999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:10:59 smithi116 ceph-mon[37097]: pgmap v9000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:59 smithi171 ceph-mon[37789]: pgmap v8999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:59.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:10:59 smithi171 ceph-mon[37789]: pgmap v9000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:00.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:00.656+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:11:00.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:00.656+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:11:00.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:00.656+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:01 smithi044 ceph-mon[32404]: pgmap v9001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:01 smithi116 ceph-mon[37097]: pgmap v9001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:01 smithi171 ceph-mon[37789]: pgmap v9001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:02.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16206.034691 secs 2024-07-26T13:11:02.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16206.034691 secs 2024-07-26T13:11:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16206.034691 secs 2024-07-26T13:11:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:03 smithi044 ceph-mon[32404]: pgmap v9002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T13:11:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:03 smithi116 ceph-mon[37097]: pgmap v9002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T13:11:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:03 smithi171 ceph-mon[37789]: pgmap v9002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T13:11:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:05 smithi044 ceph-mon[32404]: pgmap v9003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:05 smithi116 ceph-mon[37097]: pgmap v9003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:05 smithi171 ceph-mon[37789]: pgmap v9003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:07 smithi044 ceph-mon[32404]: pgmap v9004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16211.034847 secs 2024-07-26T13:11:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:07 smithi116 ceph-mon[37097]: pgmap v9004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16211.034847 secs 2024-07-26T13:11:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:07 smithi171 ceph-mon[37789]: pgmap v9004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16211.034847 secs 2024-07-26T13:11:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:09 smithi044 ceph-mon[32404]: pgmap v9005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:09.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:09 smithi116 ceph-mon[37097]: pgmap v9005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:09 smithi171 ceph-mon[37789]: pgmap v9005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:10 smithi044 ceph-mon[32404]: pgmap v9006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:10 smithi116 ceph-mon[37097]: pgmap v9006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:10 smithi171 ceph-mon[37789]: pgmap v9006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16216.035009 secs 2024-07-26T13:11:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:12 smithi044 ceph-mon[32404]: pgmap v9007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16216.035009 secs 2024-07-26T13:11:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:12 smithi116 ceph-mon[37097]: pgmap v9007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:12.540 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16216.035009 secs 2024-07-26T13:11:12.540 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:12 smithi171 ceph-mon[37789]: pgmap v9007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:14 smithi044 ceph-mon[32404]: pgmap v9008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:14.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:14 smithi116 ceph-mon[37097]: pgmap v9008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:14 smithi171 ceph-mon[37789]: pgmap v9008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.768+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900029000) for more than 900000000 us, fault. 2024-07-26T13:11:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.768+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58069040 comp rx=0 tx=0).stop 2024-07-26T13:11:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.768+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:11:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.768+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:11:14.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.768+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:11:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.769+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:11:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:14.771+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fcf580318f0 tx=0x7fcf58069040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:11:15.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:15.657+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:11:15.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:15.657+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:11:15.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:15.657+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:16 smithi044 ceph-mon[32404]: pgmap v9009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:16 smithi116 ceph-mon[37097]: pgmap v9009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:16 smithi171 ceph-mon[37789]: pgmap v9009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16221.035109 secs 2024-07-26T13:11:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16221.035109 secs 2024-07-26T13:11:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16221.035109 secs 2024-07-26T13:11:17.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:17.606+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:11:17.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:17.606+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:11:17.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:17.606+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:11:17.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:17.606+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:11:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:18 smithi044 ceph-mon[32404]: pgmap v9010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:18 smithi116 ceph-mon[37097]: pgmap v9010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:18 smithi171 ceph-mon[37789]: pgmap v9010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:20 smithi044 ceph-mon[32404]: pgmap v9011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:20 smithi116 ceph-mon[37097]: pgmap v9011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:20 smithi171 ceph-mon[37789]: pgmap v9011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16226.035213 secs 2024-07-26T13:11:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:22 smithi044 ceph-mon[32404]: pgmap v9012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16226.035213 secs 2024-07-26T13:11:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:22 smithi116 ceph-mon[37097]: pgmap v9012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16226.035213 secs 2024-07-26T13:11:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:22 smithi171 ceph-mon[37789]: pgmap v9012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:24 smithi116 ceph-mon[37097]: pgmap v9013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:24 smithi171 ceph-mon[37789]: pgmap v9013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:24 smithi044 ceph-mon[32404]: pgmap v9013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:26 smithi171 ceph-mon[37789]: pgmap v9014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:26 smithi044 ceph-mon[32404]: pgmap v9014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:26 smithi116 ceph-mon[37097]: pgmap v9014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16231.035312 secs 2024-07-26T13:11:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16231.035312 secs 2024-07-26T13:11:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16231.035312 secs 2024-07-26T13:11:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:28 smithi171 ceph-mon[37789]: pgmap v9015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:28 smithi044 ceph-mon[32404]: pgmap v9015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:28 smithi116 ceph-mon[37097]: pgmap v9015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:30 smithi171 ceph-mon[37789]: pgmap v9016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:30.658+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:11:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:30.658+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:11:30.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:30.658+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:30.660 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:30 smithi044 ceph-mon[32404]: pgmap v9016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:30 smithi116 ceph-mon[37097]: pgmap v9016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:32.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16236.035450 secs 2024-07-26T13:11:32.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:32 smithi116 ceph-mon[37097]: pgmap v9017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16236.035450 secs 2024-07-26T13:11:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:32 smithi171 ceph-mon[37789]: pgmap v9017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16236.035450 secs 2024-07-26T13:11:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:32 smithi044 ceph-mon[32404]: pgmap v9017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:34 smithi171 ceph-mon[37789]: pgmap v9018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:34 smithi044 ceph-mon[32404]: pgmap v9018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:34 smithi116 ceph-mon[37097]: pgmap v9018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:36 smithi171 ceph-mon[37789]: pgmap v9019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:36 smithi044 ceph-mon[32404]: pgmap v9019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:36 smithi116 ceph-mon[37097]: pgmap v9019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16241.035591 secs 2024-07-26T13:11:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16241.035591 secs 2024-07-26T13:11:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16241.035591 secs 2024-07-26T13:11:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:38 smithi171 ceph-mon[37789]: pgmap v9020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:38 smithi044 ceph-mon[32404]: pgmap v9020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:38 smithi116 ceph-mon[37097]: pgmap v9020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:40.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:40 smithi171 ceph-mon[37789]: pgmap v9021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:40.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:40 smithi044 ceph-mon[32404]: pgmap v9021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:40 smithi116 ceph-mon[37097]: pgmap v9021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16246.035753 secs 2024-07-26T13:11:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:42 smithi044 ceph-mon[32404]: pgmap v9022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16246.035753 secs 2024-07-26T13:11:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:42 smithi116 ceph-mon[37097]: pgmap v9022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16246.035753 secs 2024-07-26T13:11:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:42 smithi171 ceph-mon[37789]: pgmap v9022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:44 smithi044 ceph-mon[32404]: pgmap v9023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:44 smithi116 ceph-mon[37097]: pgmap v9023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:44 smithi171 ceph-mon[37789]: pgmap v9023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:45.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:45.663+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:11:45.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:45.663+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:45.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:11:45.663+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:11:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:46 smithi044 ceph-mon[32404]: pgmap v9024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:46 smithi116 ceph-mon[37097]: pgmap v9024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:46 smithi171 ceph-mon[37789]: pgmap v9024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:47.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16251.035855 secs 2024-07-26T13:11:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16251.035855 secs 2024-07-26T13:11:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16251.035855 secs 2024-07-26T13:11:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:48 smithi044 ceph-mon[32404]: pgmap v9025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:48 smithi116 ceph-mon[37097]: pgmap v9025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:48 smithi171 ceph-mon[37789]: pgmap v9025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:50 smithi044 ceph-mon[32404]: pgmap v9026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:50 smithi116 ceph-mon[37097]: pgmap v9026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:50 smithi171 ceph-mon[37789]: pgmap v9026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16256.035990 secs 2024-07-26T13:11:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:52 smithi044 ceph-mon[32404]: pgmap v9027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16256.035990 secs 2024-07-26T13:11:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:52 smithi116 ceph-mon[37097]: pgmap v9027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16256.035990 secs 2024-07-26T13:11:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:52 smithi171 ceph-mon[37789]: pgmap v9027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:54 smithi044 ceph-mon[32404]: pgmap v9028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:54 smithi116 ceph-mon[37097]: pgmap v9028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:54 smithi171 ceph-mon[37789]: pgmap v9028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:56 smithi044 ceph-mon[32404]: pgmap v9029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:56 smithi116 ceph-mon[37097]: pgmap v9029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:56 smithi171 ceph-mon[37789]: pgmap v9029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16261.036147 secs 2024-07-26T13:11:57.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16261.036147 secs 2024-07-26T13:11:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16261.036147 secs 2024-07-26T13:11:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:58 smithi044 ceph-mon[32404]: pgmap v9030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:58 smithi116 ceph-mon[37097]: pgmap v9030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:58 smithi171 ceph-mon[37789]: pgmap v9030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:11:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:59 smithi044 ceph-mon[32404]: pgmap v9031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:59 smithi044 ceph-mon[32404]: pgmap v9032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:11:59.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:11:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:59 smithi116 ceph-mon[37097]: pgmap v9031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:59 smithi116 ceph-mon[37097]: pgmap v9032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:11:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:11:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:11:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:59 smithi171 ceph-mon[37789]: pgmap v9031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:59 smithi171 ceph-mon[37789]: pgmap v9032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:11:59.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:11:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:00.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:00.665+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:12:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:00.665+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:12:00.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:00.665+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:01 smithi044 ceph-mon[32404]: pgmap v9033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:12:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:01 smithi116 ceph-mon[37097]: pgmap v9033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:12:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:01 smithi171 ceph-mon[37789]: pgmap v9033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:12:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16266.036300 secs 2024-07-26T13:12:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16266.036300 secs 2024-07-26T13:12:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16266.036300 secs 2024-07-26T13:12:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:03 smithi171 ceph-mon[37789]: pgmap v9034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:12:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:03 smithi044 ceph-mon[32404]: pgmap v9034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:12:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:03 smithi116 ceph-mon[37097]: pgmap v9034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:12:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:05 smithi171 ceph-mon[37789]: pgmap v9035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:12:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:05 smithi044 ceph-mon[32404]: pgmap v9035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:12:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:05 smithi116 ceph-mon[37097]: pgmap v9035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:12:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16271.036456 secs 2024-07-26T13:12:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:07 smithi171 ceph-mon[37789]: pgmap v9036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:12:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16271.036456 secs 2024-07-26T13:12:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:07 smithi044 ceph-mon[32404]: pgmap v9036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:12:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16271.036456 secs 2024-07-26T13:12:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:07 smithi116 ceph-mon[37097]: pgmap v9036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:12:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:09 smithi171 ceph-mon[37789]: pgmap v9037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:09 smithi044 ceph-mon[32404]: pgmap v9037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:09 smithi116 ceph-mon[37097]: pgmap v9037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:11 smithi171 ceph-mon[37789]: pgmap v9038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16276.036591 secs 2024-07-26T13:12:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:11 smithi044 ceph-mon[32404]: pgmap v9038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16276.036591 secs 2024-07-26T13:12:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:11 smithi116 ceph-mon[37097]: pgmap v9038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16276.036591 secs 2024-07-26T13:12:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:13 smithi171 ceph-mon[37789]: pgmap v9039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:13 smithi044 ceph-mon[32404]: pgmap v9039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:13 smithi116 ceph-mon[37097]: pgmap v9039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:15.670+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:12:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:15.670+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:15.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:15.670+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:12:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:15 smithi171 ceph-mon[37789]: pgmap v9040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:16.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:15 smithi044 ceph-mon[32404]: pgmap v9040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:15 smithi116 ceph-mon[37097]: pgmap v9040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16281.036728 secs 2024-07-26T13:12:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16281.036728 secs 2024-07-26T13:12:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16281.036728 secs 2024-07-26T13:12:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:17 smithi171 ceph-mon[37789]: pgmap v9041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:17.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:17 smithi044 ceph-mon[32404]: pgmap v9041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:17 smithi116 ceph-mon[37097]: pgmap v9041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:19 smithi171 ceph-mon[37789]: pgmap v9042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:19 smithi044 ceph-mon[32404]: pgmap v9042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:19 smithi116 ceph-mon[37097]: pgmap v9042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:21 smithi171 ceph-mon[37789]: pgmap v9043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16286.036902 secs 2024-07-26T13:12:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:21 smithi044 ceph-mon[32404]: pgmap v9043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16286.036902 secs 2024-07-26T13:12:22.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:21 smithi116 ceph-mon[37097]: pgmap v9043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16286.036902 secs 2024-07-26T13:12:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:23 smithi044 ceph-mon[32404]: pgmap v9044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:24.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:23 smithi116 ceph-mon[37097]: pgmap v9044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:23 smithi171 ceph-mon[37789]: pgmap v9044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:25 smithi044 ceph-mon[32404]: pgmap v9045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:25 smithi116 ceph-mon[37097]: pgmap v9045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:25 smithi171 ceph-mon[37789]: pgmap v9045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16291.037062 secs 2024-07-26T13:12:27.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16291.037062 secs 2024-07-26T13:12:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16291.037062 secs 2024-07-26T13:12:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:27 smithi044 ceph-mon[32404]: pgmap v9046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:27 smithi116 ceph-mon[37097]: pgmap v9046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:27 smithi171 ceph-mon[37789]: pgmap v9046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:29 smithi044 ceph-mon[32404]: pgmap v9047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:29 smithi116 ceph-mon[37097]: pgmap v9047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:29 smithi171 ceph-mon[37789]: pgmap v9047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:30.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:30.672+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:12:30.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:30.672+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:30.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:30.672+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:12:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:31 smithi044 ceph-mon[32404]: pgmap v9048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16296.037214 secs 2024-07-26T13:12:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:31 smithi116 ceph-mon[37097]: pgmap v9048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16296.037214 secs 2024-07-26T13:12:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:31 smithi171 ceph-mon[37789]: pgmap v9048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16296.037214 secs 2024-07-26T13:12:33.890 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:33 smithi171 ceph-mon[37789]: pgmap v9049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:33 smithi044 ceph-mon[32404]: pgmap v9049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:33 smithi116 ceph-mon[37097]: pgmap v9049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:35 smithi044 ceph-mon[32404]: pgmap v9050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:35 smithi116 ceph-mon[37097]: pgmap v9050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:35 smithi171 ceph-mon[37789]: pgmap v9050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16301.037365 secs 2024-07-26T13:12:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16301.037365 secs 2024-07-26T13:12:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16301.037365 secs 2024-07-26T13:12:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:37 smithi044 ceph-mon[32404]: pgmap v9051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:37 smithi116 ceph-mon[37097]: pgmap v9051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:37 smithi171 ceph-mon[37789]: pgmap v9051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:39 smithi044 ceph-mon[32404]: pgmap v9052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:39 smithi116 ceph-mon[37097]: pgmap v9052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:39 smithi171 ceph-mon[37789]: pgmap v9052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:41 smithi044 ceph-mon[32404]: pgmap v9053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16306.037530 secs 2024-07-26T13:12:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:41 smithi116 ceph-mon[37097]: pgmap v9053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16306.037530 secs 2024-07-26T13:12:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:41 smithi171 ceph-mon[37789]: pgmap v9053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16306.037530 secs 2024-07-26T13:12:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:43 smithi044 ceph-mon[32404]: pgmap v9054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:43 smithi116 ceph-mon[37097]: pgmap v9054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:43 smithi171 ceph-mon[37789]: pgmap v9054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:45.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:45.677+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:12:45.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:45.677+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:45.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:12:45.677+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:12:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:45 smithi044 ceph-mon[32404]: pgmap v9055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:45 smithi116 ceph-mon[37097]: pgmap v9055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:45 smithi171 ceph-mon[37789]: pgmap v9055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16311.037687 secs 2024-07-26T13:12:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16311.037687 secs 2024-07-26T13:12:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16311.037687 secs 2024-07-26T13:12:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:47 smithi044 ceph-mon[32404]: pgmap v9056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:47 smithi116 ceph-mon[37097]: pgmap v9056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:47 smithi171 ceph-mon[37789]: pgmap v9056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:49 smithi044 ceph-mon[32404]: pgmap v9057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:50.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:49 smithi116 ceph-mon[37097]: pgmap v9057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:49 smithi171 ceph-mon[37789]: pgmap v9057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:51 smithi044 ceph-mon[32404]: pgmap v9058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16316.037828 secs 2024-07-26T13:12:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:51 smithi116 ceph-mon[37097]: pgmap v9058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16316.037828 secs 2024-07-26T13:12:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:51 smithi171 ceph-mon[37789]: pgmap v9058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16316.037828 secs 2024-07-26T13:12:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:53 smithi044 ceph-mon[32404]: pgmap v9059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:54.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:53 smithi116 ceph-mon[37097]: pgmap v9059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:53 smithi171 ceph-mon[37789]: pgmap v9059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:12:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:55 smithi044 ceph-mon[32404]: pgmap v9060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:56.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:55 smithi116 ceph-mon[37097]: pgmap v9060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:55 smithi171 ceph-mon[37789]: pgmap v9060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:12:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16321.037990 secs 2024-07-26T13:12:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16321.037990 secs 2024-07-26T13:12:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16321.037990 secs 2024-07-26T13:12:58.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:57 smithi044 ceph-mon[32404]: pgmap v9061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:58.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:57 smithi116 ceph-mon[37097]: pgmap v9061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:57 smithi171 ceph-mon[37789]: pgmap v9061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:59 smithi044 ceph-mon[32404]: pgmap v9062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:12:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:59 smithi116 ceph-mon[37097]: pgmap v9062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:12:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:59 smithi171 ceph-mon[37789]: pgmap v9062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:12:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:00.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:00.677+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:13:00.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:00.677+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:00.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:00.677+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:13:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:00 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:00 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:00 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:01 smithi044 ceph-mon[32404]: pgmap v9063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:13:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:01 smithi044 ceph-mon[32404]: pgmap v9064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16326.038138 secs 2024-07-26T13:13:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:01 smithi116 ceph-mon[37097]: pgmap v9063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:13:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:01 smithi116 ceph-mon[37097]: pgmap v9064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16326.038138 secs 2024-07-26T13:13:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:01 smithi171 ceph-mon[37789]: pgmap v9063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:13:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:01 smithi171 ceph-mon[37789]: pgmap v9064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16326.038138 secs 2024-07-26T13:13:04.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:03 smithi044 ceph-mon[32404]: pgmap v9065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:03 smithi116 ceph-mon[37097]: pgmap v9065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:03 smithi171 ceph-mon[37789]: pgmap v9065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:05 smithi044 ceph-mon[32404]: pgmap v9066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:05 smithi116 ceph-mon[37097]: pgmap v9066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:05 smithi171 ceph-mon[37789]: pgmap v9066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16331.038390 secs 2024-07-26T13:13:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16331.038390 secs 2024-07-26T13:13:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16331.038390 secs 2024-07-26T13:13:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:07 smithi044 ceph-mon[32404]: pgmap v9067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:13:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:07 smithi116 ceph-mon[37097]: pgmap v9067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:13:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:07 smithi171 ceph-mon[37789]: pgmap v9067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:13:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:09 smithi044 ceph-mon[32404]: pgmap v9068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:10.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:09 smithi116 ceph-mon[37097]: pgmap v9068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:09 smithi171 ceph-mon[37789]: pgmap v9068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:11 smithi044 ceph-mon[32404]: pgmap v9069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:12.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16336.038561 secs 2024-07-26T13:13:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:11 smithi116 ceph-mon[37097]: pgmap v9069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16336.038561 secs 2024-07-26T13:13:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:11 smithi171 ceph-mon[37789]: pgmap v9069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16336.038561 secs 2024-07-26T13:13:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:13 smithi044 ceph-mon[32404]: pgmap v9070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:14.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:13 smithi116 ceph-mon[37097]: pgmap v9070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:13 smithi171 ceph-mon[37789]: pgmap v9070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:15.681+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:13:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:15.681+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:15.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:15.681+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:13:16.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:15 smithi044 ceph-mon[32404]: pgmap v9071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:15 smithi116 ceph-mon[37097]: pgmap v9071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:15 smithi171 ceph-mon[37789]: pgmap v9071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16341.038726 secs 2024-07-26T13:13:17.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16341.038726 secs 2024-07-26T13:13:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16341.038726 secs 2024-07-26T13:13:18.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:17 smithi044 ceph-mon[32404]: pgmap v9072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:17 smithi116 ceph-mon[37097]: pgmap v9072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:17 smithi171 ceph-mon[37789]: pgmap v9072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:20.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:19 smithi044 ceph-mon[32404]: pgmap v9073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:19 smithi116 ceph-mon[37097]: pgmap v9073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:19 smithi171 ceph-mon[37789]: pgmap v9073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:21 smithi044 ceph-mon[32404]: pgmap v9074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16346.038901 secs 2024-07-26T13:13:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:21 smithi116 ceph-mon[37097]: pgmap v9074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16346.038901 secs 2024-07-26T13:13:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:21 smithi171 ceph-mon[37789]: pgmap v9074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16346.038901 secs 2024-07-26T13:13:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:23 smithi044 ceph-mon[32404]: pgmap v9075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:13:24.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:23 smithi116 ceph-mon[37097]: pgmap v9075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:13:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:23 smithi171 ceph-mon[37789]: pgmap v9075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:13:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:25 smithi044 ceph-mon[32404]: pgmap v9076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:13:26.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:25 smithi116 ceph-mon[37097]: pgmap v9076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:13:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:25 smithi171 ceph-mon[37789]: pgmap v9076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:13:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16351.039037 secs 2024-07-26T13:13:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16351.039037 secs 2024-07-26T13:13:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16351.039037 secs 2024-07-26T13:13:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:27 smithi044 ceph-mon[32404]: pgmap v9077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:13:28.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:27 smithi116 ceph-mon[37097]: pgmap v9077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:13:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:27 smithi171 ceph-mon[37789]: pgmap v9077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:13:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:29 smithi044 ceph-mon[32404]: pgmap v9078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:13:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:29 smithi116 ceph-mon[37097]: pgmap v9078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:13:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:29 smithi171 ceph-mon[37789]: pgmap v9078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:13:30.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:30.682+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:13:30.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:30.683+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:13:30.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:30.683+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:31 smithi044 ceph-mon[32404]: pgmap v9079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:13:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16356.039193 secs 2024-07-26T13:13:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:31 smithi116 ceph-mon[37097]: pgmap v9079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:13:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16356.039193 secs 2024-07-26T13:13:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:31 smithi171 ceph-mon[37789]: pgmap v9079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:13:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16356.039193 secs 2024-07-26T13:13:32.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:32.630+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:13:32.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:32.630+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:13:32.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:32.630+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:13:32.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:32.630+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:13:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:33 smithi044 ceph-mon[32404]: pgmap v9080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:13:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:33 smithi116 ceph-mon[37097]: pgmap v9080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:13:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:33 smithi171 ceph-mon[37789]: pgmap v9080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:13:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:35 smithi044 ceph-mon[32404]: pgmap v9081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:35 smithi116 ceph-mon[37097]: pgmap v9081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:35 smithi171 ceph-mon[37789]: pgmap v9081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16361.039332 secs 2024-07-26T13:13:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16361.039332 secs 2024-07-26T13:13:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16361.039332 secs 2024-07-26T13:13:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:37 smithi044 ceph-mon[32404]: pgmap v9082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:38.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:37 smithi116 ceph-mon[37097]: pgmap v9082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:37 smithi171 ceph-mon[37789]: pgmap v9082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:39 smithi044 ceph-mon[32404]: pgmap v9083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:39 smithi116 ceph-mon[37097]: pgmap v9083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:39 smithi171 ceph-mon[37789]: pgmap v9083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:42 smithi044 ceph-mon[32404]: pgmap v9084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16366.039451 secs 2024-07-26T13:13:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:42 smithi116 ceph-mon[37097]: pgmap v9084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16366.039451 secs 2024-07-26T13:13:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:42 smithi171 ceph-mon[37789]: pgmap v9084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16366.039451 secs 2024-07-26T13:13:44.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:44 smithi044 ceph-mon[32404]: pgmap v9085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:44.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:44 smithi116 ceph-mon[37097]: pgmap v9085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:44 smithi171 ceph-mon[37789]: pgmap v9085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:45.689+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:13:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:45.689+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:13:45.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:13:45.689+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:46 smithi116 ceph-mon[37097]: pgmap v9086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:46 smithi171 ceph-mon[37789]: pgmap v9086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:46 smithi044 ceph-mon[32404]: pgmap v9086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16371.039548 secs 2024-07-26T13:13:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16371.039548 secs 2024-07-26T13:13:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16371.039548 secs 2024-07-26T13:13:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:48 smithi171 ceph-mon[37789]: pgmap v9087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:48.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:48 smithi044 ceph-mon[32404]: pgmap v9087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:48 smithi116 ceph-mon[37097]: pgmap v9087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:50 smithi171 ceph-mon[37789]: pgmap v9088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:50 smithi044 ceph-mon[32404]: pgmap v9088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:50.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:50 smithi116 ceph-mon[37097]: pgmap v9088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:52 smithi171 ceph-mon[37789]: pgmap v9089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16376.039707 secs 2024-07-26T13:13:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:52 smithi044 ceph-mon[32404]: pgmap v9089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16376.039707 secs 2024-07-26T13:13:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:52 smithi116 ceph-mon[37097]: pgmap v9089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16376.039707 secs 2024-07-26T13:13:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:54 smithi171 ceph-mon[37789]: pgmap v9090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:54 smithi044 ceph-mon[32404]: pgmap v9090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:54 smithi116 ceph-mon[37097]: pgmap v9090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:13:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:56 smithi171 ceph-mon[37789]: pgmap v9091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:56 smithi044 ceph-mon[32404]: pgmap v9091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:56.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:56 smithi116 ceph-mon[37097]: pgmap v9091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:13:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16381.039839 secs 2024-07-26T13:13:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16381.039839 secs 2024-07-26T13:13:57.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16381.039839 secs 2024-07-26T13:13:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:13:58 smithi171 ceph-mon[37789]: pgmap v9092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:13:58 smithi044 ceph-mon[32404]: pgmap v9092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:13:58 smithi116 ceph-mon[37097]: pgmap v9092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:00 smithi171 ceph-mon[37789]: pgmap v9093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:14:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:00 smithi044 ceph-mon[32404]: pgmap v9093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:14:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:00 smithi116 ceph-mon[37097]: pgmap v9093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:14:00.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:00.689+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:14:00.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:00.689+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:00.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:00.689+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:14:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:02 smithi044 ceph-mon[32404]: pgmap v9094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:14:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16386.039940 secs 2024-07-26T13:14:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:02 smithi171 ceph-mon[37789]: pgmap v9094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:14:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16386.039940 secs 2024-07-26T13:14:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:02 smithi116 ceph-mon[37097]: pgmap v9094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:14:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16386.039940 secs 2024-07-26T13:14:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:03 smithi044 ceph-mon[32404]: pgmap v9095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:14:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:03 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:03 smithi116 ceph-mon[37097]: pgmap v9095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:14:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:03 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:03 smithi171 ceph-mon[37789]: pgmap v9095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:14:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:03 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:05 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:05 smithi171 ceph-mon[37789]: pgmap v9096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:14:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:05 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:05 smithi044 ceph-mon[32404]: pgmap v9096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:14:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:05 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:05 smithi116 ceph-mon[37097]: pgmap v9096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:14:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:06 smithi171 ceph-mon[37789]: pgmap v9097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:14:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:06 smithi171 ceph-mon[37789]: pgmap v9098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:06 smithi044 ceph-mon[32404]: pgmap v9097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:14:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:06 smithi044 ceph-mon[32404]: pgmap v9098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:06 smithi116 ceph-mon[37097]: pgmap v9097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:14:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:06 smithi116 ceph-mon[37097]: pgmap v9098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:14:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16391.040043 secs 2024-07-26T13:14:07.649 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16391.040043 secs 2024-07-26T13:14:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16391.040043 secs 2024-07-26T13:14:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:08 smithi044 ceph-mon[32404]: pgmap v9099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:08 smithi116 ceph-mon[37097]: pgmap v9099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:08 smithi171 ceph-mon[37789]: pgmap v9099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:10 smithi044 ceph-mon[32404]: pgmap v9100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:14:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:10 smithi116 ceph-mon[37097]: pgmap v9100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:14:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:10 smithi171 ceph-mon[37789]: pgmap v9100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:14:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16396.040194 secs 2024-07-26T13:14:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:12 smithi044 ceph-mon[32404]: pgmap v9101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16396.040194 secs 2024-07-26T13:14:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:12 smithi116 ceph-mon[37097]: pgmap v9101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16396.040194 secs 2024-07-26T13:14:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:12 smithi171 ceph-mon[37789]: pgmap v9101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:14 smithi044 ceph-mon[32404]: pgmap v9102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:14:14.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:14 smithi116 ceph-mon[37097]: pgmap v9102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:14:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:14 smithi171 ceph-mon[37789]: pgmap v9102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:14:15.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:15.694+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:14:15.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:15.694+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:15.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:15.694+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:14:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:16 smithi044 ceph-mon[32404]: pgmap v9103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:16.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:16 smithi116 ceph-mon[37097]: pgmap v9103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:16 smithi171 ceph-mon[37789]: pgmap v9103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:17.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:17.638+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:14:17.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:17.638+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:14:17.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:17.639+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:14:17.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:17.639+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:14:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16401.040357 secs 2024-07-26T13:14:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16401.040357 secs 2024-07-26T13:14:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16401.040357 secs 2024-07-26T13:14:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:18 smithi044 ceph-mon[32404]: pgmap v9104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:18.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:18 smithi116 ceph-mon[37097]: pgmap v9104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:18 smithi171 ceph-mon[37789]: pgmap v9104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:20 smithi044 ceph-mon[32404]: pgmap v9105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:20.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:20 smithi116 ceph-mon[37097]: pgmap v9105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:20 smithi171 ceph-mon[37789]: pgmap v9105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16406.040535 secs 2024-07-26T13:14:22.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:22 smithi116 ceph-mon[37097]: pgmap v9106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16406.040535 secs 2024-07-26T13:14:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:22 smithi171 ceph-mon[37789]: pgmap v9106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:23.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16406.040535 secs 2024-07-26T13:14:23.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:22 smithi044 ceph-mon[32404]: pgmap v9106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:24 smithi116 ceph-mon[37097]: pgmap v9107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:24 smithi171 ceph-mon[37789]: pgmap v9107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:24 smithi044 ceph-mon[32404]: pgmap v9107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:26 smithi171 ceph-mon[37789]: pgmap v9108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:26 smithi044 ceph-mon[32404]: pgmap v9108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:27.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:26 smithi116 ceph-mon[37097]: pgmap v9108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16411.040711 secs 2024-07-26T13:14:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16411.040711 secs 2024-07-26T13:14:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16411.040711 secs 2024-07-26T13:14:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:28 smithi171 ceph-mon[37789]: pgmap v9109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:28 smithi044 ceph-mon[32404]: pgmap v9109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:29.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:28 smithi116 ceph-mon[37097]: pgmap v9109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:30.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:30.695+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:14:30.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:30.695+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:14:30.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:30.695+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:30 smithi171 ceph-mon[37789]: pgmap v9110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:30 smithi044 ceph-mon[32404]: pgmap v9110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:31.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:30 smithi116 ceph-mon[37097]: pgmap v9110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16416.040866 secs 2024-07-26T13:14:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:32 smithi171 ceph-mon[37789]: pgmap v9111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16416.040866 secs 2024-07-26T13:14:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:32 smithi044 ceph-mon[32404]: pgmap v9111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16416.040866 secs 2024-07-26T13:14:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:32 smithi116 ceph-mon[37097]: pgmap v9111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:34.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:34 smithi116 ceph-mon[37097]: pgmap v9112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:34 smithi171 ceph-mon[37789]: pgmap v9112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:34 smithi044 ceph-mon[32404]: pgmap v9112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:36 smithi171 ceph-mon[37789]: pgmap v9113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:36 smithi044 ceph-mon[32404]: pgmap v9113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:36 smithi116 ceph-mon[37097]: pgmap v9113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16421.041043 secs 2024-07-26T13:14:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16421.041043 secs 2024-07-26T13:14:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16421.041043 secs 2024-07-26T13:14:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:38 smithi171 ceph-mon[37789]: pgmap v9114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:38.881 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:38 smithi044 ceph-mon[32404]: pgmap v9114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:38 smithi116 ceph-mon[37097]: pgmap v9114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:40 smithi171 ceph-mon[37789]: pgmap v9115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:40 smithi044 ceph-mon[32404]: pgmap v9115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:41.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:40 smithi116 ceph-mon[37097]: pgmap v9115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16426.041207 secs 2024-07-26T13:14:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16426.041207 secs 2024-07-26T13:14:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16426.041207 secs 2024-07-26T13:14:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:42 smithi044 ceph-mon[32404]: pgmap v9116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:43.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:42 smithi116 ceph-mon[37097]: pgmap v9116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:42 smithi171 ceph-mon[37789]: pgmap v9116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:44 smithi044 ceph-mon[32404]: pgmap v9117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:45.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:44 smithi116 ceph-mon[37097]: pgmap v9117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:44 smithi171 ceph-mon[37789]: pgmap v9117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:45.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:45.701+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:14:45.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:45.701+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:45.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:14:45.701+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:14:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:46 smithi044 ceph-mon[32404]: pgmap v9118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16431.041381 secs 2024-07-26T13:14:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:46 smithi116 ceph-mon[37097]: pgmap v9118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16431.041381 secs 2024-07-26T13:14:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:46 smithi171 ceph-mon[37789]: pgmap v9118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16431.041381 secs 2024-07-26T13:14:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:48 smithi044 ceph-mon[32404]: pgmap v9119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:49.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:48 smithi116 ceph-mon[37097]: pgmap v9119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:48 smithi171 ceph-mon[37789]: pgmap v9119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:50 smithi044 ceph-mon[32404]: pgmap v9120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:50 smithi116 ceph-mon[37097]: pgmap v9120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:50 smithi171 ceph-mon[37789]: pgmap v9120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:14:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16436.041565 secs 2024-07-26T13:14:52.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16436.041565 secs 2024-07-26T13:14:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16436.041565 secs 2024-07-26T13:14:53.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:52 smithi044 ceph-mon[32404]: pgmap v9121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:52 smithi116 ceph-mon[37097]: pgmap v9121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:52 smithi171 ceph-mon[37789]: pgmap v9121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:54 smithi044 ceph-mon[32404]: pgmap v9122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:54 smithi116 ceph-mon[37097]: pgmap v9122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:54 smithi171 ceph-mon[37789]: pgmap v9122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:14:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:56 smithi044 ceph-mon[32404]: pgmap v9123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16441.041746 secs 2024-07-26T13:14:57.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:56 smithi116 ceph-mon[37097]: pgmap v9123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16441.041746 secs 2024-07-26T13:14:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:56 smithi171 ceph-mon[37789]: pgmap v9123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16441.041746 secs 2024-07-26T13:14:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:14:58 smithi044 ceph-mon[32404]: pgmap v9124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:14:58 smithi116 ceph-mon[37097]: pgmap v9124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:14:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:14:58 smithi171 ceph-mon[37789]: pgmap v9124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:00.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:00.701+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:15:00.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:00.701+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:15:00.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:00.701+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:00 smithi044 ceph-mon[32404]: pgmap v9125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:00 smithi116 ceph-mon[37097]: pgmap v9125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:00 smithi171 ceph-mon[37789]: pgmap v9125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16446.041846 secs 2024-07-26T13:15:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16446.041846 secs 2024-07-26T13:15:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16446.041846 secs 2024-07-26T13:15:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:02 smithi044 ceph-mon[32404]: pgmap v9126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:03.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:02 smithi116 ceph-mon[37097]: pgmap v9126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:02 smithi171 ceph-mon[37789]: pgmap v9126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:04 smithi044 ceph-mon[32404]: pgmap v9127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:15:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:04 smithi116 ceph-mon[37097]: pgmap v9127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:15:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:04 smithi171 ceph-mon[37789]: pgmap v9127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:15:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:05 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:06.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:05 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:05 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:06 smithi044 ceph-mon[32404]: pgmap v9128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16451.041981 secs 2024-07-26T13:15:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:07.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:06 smithi116 ceph-mon[37097]: pgmap v9128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16451.041981 secs 2024-07-26T13:15:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:07.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:06 smithi171 ceph-mon[37789]: pgmap v9128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16451.041981 secs 2024-07-26T13:15:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:07 smithi044 ceph-mon[32404]: pgmap v9129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:08.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:07 smithi044 ceph-mon[32404]: pgmap v9130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:15:08.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:07 smithi116 ceph-mon[37097]: pgmap v9129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:07 smithi116 ceph-mon[37097]: pgmap v9130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:15:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:07 smithi171 ceph-mon[37789]: pgmap v9129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:08.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:07 smithi171 ceph-mon[37789]: pgmap v9130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:15:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:09 smithi044 ceph-mon[32404]: pgmap v9131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:09 smithi116 ceph-mon[37097]: pgmap v9131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:09 smithi171 ceph-mon[37789]: pgmap v9131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16456.042138 secs 2024-07-26T13:15:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:11 smithi044 ceph-mon[32404]: pgmap v9132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:12.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16456.042138 secs 2024-07-26T13:15:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:11 smithi116 ceph-mon[37097]: pgmap v9132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16456.042138 secs 2024-07-26T13:15:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:11 smithi171 ceph-mon[37789]: pgmap v9132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:14 smithi044 ceph-mon[32404]: pgmap v9133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:14 smithi116 ceph-mon[37097]: pgmap v9133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:14 smithi171 ceph-mon[37789]: pgmap v9133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:15.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:15.706+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:15:15.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:15.707+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:15:15.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:15.707+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:16 smithi044 ceph-mon[32404]: pgmap v9134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:16 smithi116 ceph-mon[37097]: pgmap v9134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:16 smithi171 ceph-mon[37789]: pgmap v9134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:15:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16461.042272 secs 2024-07-26T13:15:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16461.042272 secs 2024-07-26T13:15:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16461.042272 secs 2024-07-26T13:15:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:18 smithi171 ceph-mon[37789]: pgmap v9135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:18.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:18 smithi044 ceph-mon[32404]: pgmap v9135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:18 smithi116 ceph-mon[37097]: pgmap v9135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:20 smithi171 ceph-mon[37789]: pgmap v9136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:20 smithi044 ceph-mon[32404]: pgmap v9136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:20 smithi116 ceph-mon[37097]: pgmap v9136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16466.042430 secs 2024-07-26T13:15:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:22 smithi171 ceph-mon[37789]: pgmap v9137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16466.042430 secs 2024-07-26T13:15:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:22 smithi044 ceph-mon[32404]: pgmap v9137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16466.042430 secs 2024-07-26T13:15:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:22 smithi116 ceph-mon[37097]: pgmap v9137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:24 smithi171 ceph-mon[37789]: pgmap v9138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:24 smithi044 ceph-mon[32404]: pgmap v9138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:24 smithi116 ceph-mon[37097]: pgmap v9138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:26 smithi171 ceph-mon[37789]: pgmap v9139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:26 smithi044 ceph-mon[32404]: pgmap v9139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:26 smithi116 ceph-mon[37097]: pgmap v9139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16471.042590 secs 2024-07-26T13:15:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16471.042590 secs 2024-07-26T13:15:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16471.042590 secs 2024-07-26T13:15:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:28 smithi171 ceph-mon[37789]: pgmap v9140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:28 smithi044 ceph-mon[32404]: pgmap v9140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:28.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:28 smithi116 ceph-mon[37097]: pgmap v9140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:30 smithi171 ceph-mon[37789]: pgmap v9141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:30 smithi044 ceph-mon[32404]: pgmap v9141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:30 smithi116 ceph-mon[37097]: pgmap v9141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:30.708+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:15:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:30.708+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:15:30.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:30.708+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16476.042757 secs 2024-07-26T13:15:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:32 smithi171 ceph-mon[37789]: pgmap v9142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16476.042757 secs 2024-07-26T13:15:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:32 smithi044 ceph-mon[32404]: pgmap v9142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16476.042757 secs 2024-07-26T13:15:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:32 smithi116 ceph-mon[37097]: pgmap v9142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:34 smithi171 ceph-mon[37789]: pgmap v9143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:34 smithi044 ceph-mon[32404]: pgmap v9143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:34 smithi116 ceph-mon[37097]: pgmap v9143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:36 smithi171 ceph-mon[37789]: pgmap v9144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:36 smithi044 ceph-mon[32404]: pgmap v9144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:36 smithi116 ceph-mon[37097]: pgmap v9144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16481.042921 secs 2024-07-26T13:15:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16481.042921 secs 2024-07-26T13:15:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16481.042921 secs 2024-07-26T13:15:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:38 smithi044 ceph-mon[32404]: pgmap v9145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:38 smithi116 ceph-mon[37097]: pgmap v9145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:38 smithi171 ceph-mon[37789]: pgmap v9145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:40 smithi044 ceph-mon[32404]: pgmap v9146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:40 smithi116 ceph-mon[37097]: pgmap v9146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:40 smithi171 ceph-mon[37789]: pgmap v9146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16486.043051 secs 2024-07-26T13:15:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:42 smithi044 ceph-mon[32404]: pgmap v9147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16486.043051 secs 2024-07-26T13:15:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:42 smithi116 ceph-mon[37097]: pgmap v9147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16486.043051 secs 2024-07-26T13:15:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:42 smithi171 ceph-mon[37789]: pgmap v9147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:44 smithi044 ceph-mon[32404]: pgmap v9148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:44 smithi116 ceph-mon[37097]: pgmap v9148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:44 smithi171 ceph-mon[37789]: pgmap v9148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:45.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:45.713+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:15:45.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:45.713+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:15:45.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:15:45.713+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:46 smithi044 ceph-mon[32404]: pgmap v9149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:46 smithi116 ceph-mon[37097]: pgmap v9149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:46 smithi171 ceph-mon[37789]: pgmap v9149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16491.043215 secs 2024-07-26T13:15:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16491.043215 secs 2024-07-26T13:15:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16491.043215 secs 2024-07-26T13:15:48.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:48 smithi044 ceph-mon[32404]: pgmap v9150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:48 smithi116 ceph-mon[37097]: pgmap v9150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:48 smithi171 ceph-mon[37789]: pgmap v9150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:50.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:50 smithi044 ceph-mon[32404]: pgmap v9151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:50 smithi116 ceph-mon[37097]: pgmap v9151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:50 smithi171 ceph-mon[37789]: pgmap v9151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16496.043344 secs 2024-07-26T13:15:52.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:52 smithi044 ceph-mon[32404]: pgmap v9152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16496.043344 secs 2024-07-26T13:15:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:52 smithi116 ceph-mon[37097]: pgmap v9152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16496.043344 secs 2024-07-26T13:15:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:52 smithi171 ceph-mon[37789]: pgmap v9152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:54 smithi044 ceph-mon[32404]: pgmap v9153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:54 smithi116 ceph-mon[37097]: pgmap v9153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:54 smithi171 ceph-mon[37789]: pgmap v9153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:56 smithi044 ceph-mon[32404]: pgmap v9154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:56 smithi116 ceph-mon[37097]: pgmap v9154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:56 smithi171 ceph-mon[37789]: pgmap v9154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:15:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16501.043492 secs 2024-07-26T13:15:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16501.043492 secs 2024-07-26T13:15:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16501.043492 secs 2024-07-26T13:15:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:15:58 smithi044 ceph-mon[32404]: pgmap v9155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:15:58 smithi116 ceph-mon[37097]: pgmap v9155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:15:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:15:58 smithi171 ceph-mon[37789]: pgmap v9155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:00 smithi044 ceph-mon[32404]: pgmap v9156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:00 smithi116 ceph-mon[37097]: pgmap v9156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:00 smithi171 ceph-mon[37789]: pgmap v9156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:00.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:00.714+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:16:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:00.714+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:16:00.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:00.714+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16506.043661 secs 2024-07-26T13:16:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:02 smithi044 ceph-mon[32404]: pgmap v9157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:02.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16506.043661 secs 2024-07-26T13:16:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:02 smithi116 ceph-mon[37097]: pgmap v9157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16506.043661 secs 2024-07-26T13:16:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:02 smithi171 ceph-mon[37789]: pgmap v9157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:04 smithi044 ceph-mon[32404]: pgmap v9158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:04.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:04 smithi116 ceph-mon[37097]: pgmap v9158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:04 smithi171 ceph-mon[37789]: pgmap v9158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:06 smithi044 ceph-mon[32404]: pgmap v9159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:06 smithi116 ceph-mon[37097]: pgmap v9159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:06 smithi171 ceph-mon[37789]: pgmap v9159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16511.043804 secs 2024-07-26T13:16:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16511.043804 secs 2024-07-26T13:16:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16511.043804 secs 2024-07-26T13:16:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:08 smithi044 ceph-mon[32404]: pgmap v9160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:08.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:08 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:08.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:08 smithi116 ceph-mon[37097]: pgmap v9160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:08 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:08 smithi171 ceph-mon[37789]: pgmap v9160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:08.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:08 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:09 smithi044 ceph-mon[32404]: pgmap v9161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:16:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:09 smithi044 ceph-mon[32404]: pgmap v9162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:09 smithi116 ceph-mon[37097]: pgmap v9161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:16:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:09 smithi116 ceph-mon[37097]: pgmap v9162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:09 smithi171 ceph-mon[37789]: pgmap v9161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:16:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:09 smithi171 ceph-mon[37789]: pgmap v9162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:10 smithi044 ceph-mon[32404]: pgmap v9163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:16:10.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:10 smithi116 ceph-mon[37097]: pgmap v9163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:16:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:10 smithi171 ceph-mon[37789]: pgmap v9163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:16:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16516.043970 secs 2024-07-26T13:16:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16516.043970 secs 2024-07-26T13:16:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16516.043970 secs 2024-07-26T13:16:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:12 smithi044 ceph-mon[32404]: pgmap v9164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:16:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:12 smithi116 ceph-mon[37097]: pgmap v9164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:16:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:12 smithi171 ceph-mon[37789]: pgmap v9164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:16:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:15 smithi171 ceph-mon[37789]: pgmap v9165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:15 smithi044 ceph-mon[32404]: pgmap v9165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:15 smithi116 ceph-mon[37097]: pgmap v9165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:15.721+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:16:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:15.721+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:16:15.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:15.721+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16521.044129 secs 2024-07-26T13:16:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:17 smithi171 ceph-mon[37789]: pgmap v9166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T13:16:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16521.044129 secs 2024-07-26T13:16:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:17 smithi044 ceph-mon[32404]: pgmap v9166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T13:16:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16521.044129 secs 2024-07-26T13:16:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:17 smithi116 ceph-mon[37097]: pgmap v9166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T13:16:19.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:19 smithi044 ceph-mon[32404]: pgmap v9167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:16:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:19 smithi171 ceph-mon[37789]: pgmap v9167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:16:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:19 smithi116 ceph-mon[37097]: pgmap v9167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:16:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:21 smithi171 ceph-mon[37789]: pgmap v9168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:21 smithi044 ceph-mon[32404]: pgmap v9168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:21.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:21 smithi116 ceph-mon[37097]: pgmap v9168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16526.044297 secs 2024-07-26T13:16:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16526.044297 secs 2024-07-26T13:16:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16526.044297 secs 2024-07-26T13:16:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:23 smithi171 ceph-mon[37789]: pgmap v9169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:23 smithi044 ceph-mon[32404]: pgmap v9169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:23.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:23 smithi116 ceph-mon[37097]: pgmap v9169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:25 smithi171 ceph-mon[37789]: pgmap v9170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:25 smithi044 ceph-mon[32404]: pgmap v9170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:25 smithi116 ceph-mon[37097]: pgmap v9170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16531.044446 secs 2024-07-26T13:16:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:27 smithi171 ceph-mon[37789]: pgmap v9171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16531.044446 secs 2024-07-26T13:16:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:27 smithi044 ceph-mon[32404]: pgmap v9171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16531.044446 secs 2024-07-26T13:16:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:27 smithi116 ceph-mon[37097]: pgmap v9171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:29 smithi044 ceph-mon[32404]: pgmap v9172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:29 smithi116 ceph-mon[37097]: pgmap v9172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:29 smithi171 ceph-mon[37789]: pgmap v9172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:30.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:30.722+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:16:30.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:30.722+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:16:30.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:30.722+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:31 smithi044 ceph-mon[32404]: pgmap v9173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:31 smithi116 ceph-mon[37097]: pgmap v9173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:31 smithi171 ceph-mon[37789]: pgmap v9173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16536.044568 secs 2024-07-26T13:16:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16536.044568 secs 2024-07-26T13:16:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16536.044568 secs 2024-07-26T13:16:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:33 smithi044 ceph-mon[32404]: pgmap v9174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:33 smithi116 ceph-mon[37097]: pgmap v9174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:33 smithi171 ceph-mon[37789]: pgmap v9174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:35 smithi044 ceph-mon[32404]: pgmap v9175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:35 smithi116 ceph-mon[37097]: pgmap v9175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:35 smithi171 ceph-mon[37789]: pgmap v9175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:37.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16541.044726 secs 2024-07-26T13:16:37.389 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:37 smithi171 ceph-mon[37789]: pgmap v9176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16541.044726 secs 2024-07-26T13:16:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:37 smithi044 ceph-mon[32404]: pgmap v9176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16541.044726 secs 2024-07-26T13:16:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:37 smithi116 ceph-mon[37097]: pgmap v9176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:39 smithi044 ceph-mon[32404]: pgmap v9177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:39 smithi116 ceph-mon[37097]: pgmap v9177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:39 smithi171 ceph-mon[37789]: pgmap v9177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:41 smithi044 ceph-mon[32404]: pgmap v9178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:41 smithi116 ceph-mon[37097]: pgmap v9178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:41 smithi171 ceph-mon[37789]: pgmap v9178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16546.044863 secs 2024-07-26T13:16:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16546.044863 secs 2024-07-26T13:16:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16546.044863 secs 2024-07-26T13:16:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:43 smithi044 ceph-mon[32404]: pgmap v9179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:43.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:43 smithi116 ceph-mon[37097]: pgmap v9179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:43 smithi171 ceph-mon[37789]: pgmap v9179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:45 smithi044 ceph-mon[32404]: pgmap v9180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:45 smithi116 ceph-mon[37097]: pgmap v9180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:45 smithi171 ceph-mon[37789]: pgmap v9180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:45.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:45.727+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:16:45.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:45.727+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:16:45.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:16:45.727+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16551.045001 secs 2024-07-26T13:16:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:47 smithi044 ceph-mon[32404]: pgmap v9181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16551.045001 secs 2024-07-26T13:16:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:47 smithi116 ceph-mon[37097]: pgmap v9181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16551.045001 secs 2024-07-26T13:16:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:47 smithi171 ceph-mon[37789]: pgmap v9181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:49.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:49 smithi044 ceph-mon[32404]: pgmap v9182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:49.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:49 smithi116 ceph-mon[37097]: pgmap v9182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:49 smithi171 ceph-mon[37789]: pgmap v9182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:51 smithi044 ceph-mon[32404]: pgmap v9183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:51 smithi116 ceph-mon[37097]: pgmap v9183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:51 smithi171 ceph-mon[37789]: pgmap v9183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16556.045151 secs 2024-07-26T13:16:52.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16556.045151 secs 2024-07-26T13:16:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16556.045151 secs 2024-07-26T13:16:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:53 smithi044 ceph-mon[32404]: pgmap v9184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:53 smithi116 ceph-mon[37097]: pgmap v9184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:53 smithi171 ceph-mon[37789]: pgmap v9184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:55.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:55 smithi044 ceph-mon[32404]: pgmap v9185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:55.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:55 smithi116 ceph-mon[37097]: pgmap v9185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:55 smithi171 ceph-mon[37789]: pgmap v9185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:16:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16561.045298 secs 2024-07-26T13:16:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:57 smithi044 ceph-mon[32404]: pgmap v9186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16561.045298 secs 2024-07-26T13:16:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:57 smithi116 ceph-mon[37097]: pgmap v9186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16561.045298 secs 2024-07-26T13:16:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:57 smithi171 ceph-mon[37789]: pgmap v9186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:16:59 smithi044 ceph-mon[32404]: pgmap v9187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:16:59 smithi116 ceph-mon[37097]: pgmap v9187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:16:59 smithi171 ceph-mon[37789]: pgmap v9187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:00.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:00.728+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:17:00.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:00.728+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:17:00.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:00.728+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:01 smithi044 ceph-mon[32404]: pgmap v9188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:01 smithi116 ceph-mon[37097]: pgmap v9188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:01 smithi171 ceph-mon[37789]: pgmap v9188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16566.045465 secs 2024-07-26T13:17:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16566.045465 secs 2024-07-26T13:17:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16566.045465 secs 2024-07-26T13:17:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:03 smithi044 ceph-mon[32404]: pgmap v9189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:03 smithi116 ceph-mon[37097]: pgmap v9189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:03 smithi171 ceph-mon[37789]: pgmap v9189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:05 smithi044 ceph-mon[32404]: pgmap v9190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:05.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:05 smithi116 ceph-mon[37097]: pgmap v9190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:05 smithi171 ceph-mon[37789]: pgmap v9190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16571.045655 secs 2024-07-26T13:17:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:07 smithi044 ceph-mon[32404]: pgmap v9191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16571.045655 secs 2024-07-26T13:17:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:07 smithi116 ceph-mon[37097]: pgmap v9191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16571.045655 secs 2024-07-26T13:17:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:07 smithi171 ceph-mon[37789]: pgmap v9191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:08.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:09 smithi044 ceph-mon[32404]: pgmap v9192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:09 smithi044 ceph-mon[32404]: pgmap v9193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:09 smithi044 ceph-mon[32404]: pgmap v9194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:09.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:17:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:09 smithi116 ceph-mon[37097]: pgmap v9192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:09 smithi116 ceph-mon[37097]: pgmap v9193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:17:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:09 smithi116 ceph-mon[37097]: pgmap v9194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:17:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:09 smithi171 ceph-mon[37789]: pgmap v9192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:09 smithi171 ceph-mon[37789]: pgmap v9193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:17:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:09 smithi171 ceph-mon[37789]: pgmap v9194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:09.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:17:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:11 smithi171 ceph-mon[37789]: pgmap v9195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:11 smithi044 ceph-mon[32404]: pgmap v9195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:11 smithi116 ceph-mon[37097]: pgmap v9195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16576.045675 secs 2024-07-26T13:17:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16576.045675 secs 2024-07-26T13:17:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16576.045675 secs 2024-07-26T13:17:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:13 smithi171 ceph-mon[37789]: pgmap v9196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:13 smithi044 ceph-mon[32404]: pgmap v9196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:13 smithi116 ceph-mon[37097]: pgmap v9196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:15 smithi171 ceph-mon[37789]: pgmap v9197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:15.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:15.733+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:17:15.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:15.733+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:15.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:15.733+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:17:15.735 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:15 smithi044 ceph-mon[32404]: pgmap v9197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:15 smithi116 ceph-mon[37097]: pgmap v9197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:17:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16581.045764 secs 2024-07-26T13:17:17.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16581.045764 secs 2024-07-26T13:17:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16581.045764 secs 2024-07-26T13:17:17.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:17.671+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:17:17.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:17.671+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:17:17.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:17.677+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:17:17.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:17.677+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:17:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:17 smithi044 ceph-mon[32404]: pgmap v9198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:18.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:17 smithi116 ceph-mon[37097]: pgmap v9198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:17 smithi171 ceph-mon[37789]: pgmap v9198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:19 smithi171 ceph-mon[37789]: pgmap v9199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:19 smithi044 ceph-mon[32404]: pgmap v9199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:19 smithi116 ceph-mon[37097]: pgmap v9199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:21 smithi171 ceph-mon[37789]: pgmap v9200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:21 smithi044 ceph-mon[32404]: pgmap v9200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:21 smithi116 ceph-mon[37097]: pgmap v9200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16586.045900 secs 2024-07-26T13:17:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16586.045900 secs 2024-07-26T13:17:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16586.045900 secs 2024-07-26T13:17:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:23 smithi171 ceph-mon[37789]: pgmap v9201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:23 smithi044 ceph-mon[32404]: pgmap v9201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:23.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:23 smithi116 ceph-mon[37097]: pgmap v9201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:25 smithi171 ceph-mon[37789]: pgmap v9202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:25 smithi044 ceph-mon[32404]: pgmap v9202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:25 smithi116 ceph-mon[37097]: pgmap v9202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16591.045996 secs 2024-07-26T13:17:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16591.045996 secs 2024-07-26T13:17:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16591.045996 secs 2024-07-26T13:17:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:27 smithi044 ceph-mon[32404]: pgmap v9203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:27 smithi116 ceph-mon[37097]: pgmap v9203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:27 smithi171 ceph-mon[37789]: pgmap v9203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:29 smithi171 ceph-mon[37789]: pgmap v9204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:29 smithi044 ceph-mon[32404]: pgmap v9204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:29.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:29 smithi116 ceph-mon[37097]: pgmap v9204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:30.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:30.734+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:17:30.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:30.735+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:17:30.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:30.735+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:31 smithi171 ceph-mon[37789]: pgmap v9205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:31 smithi044 ceph-mon[32404]: pgmap v9205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:31.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:31 smithi116 ceph-mon[37097]: pgmap v9205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16596.046143 secs 2024-07-26T13:17:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16596.046143 secs 2024-07-26T13:17:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16596.046143 secs 2024-07-26T13:17:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:34 smithi171 ceph-mon[37789]: pgmap v9206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:34 smithi044 ceph-mon[32404]: pgmap v9206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:34 smithi116 ceph-mon[37097]: pgmap v9206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:36 smithi171 ceph-mon[37789]: pgmap v9207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:36 smithi044 ceph-mon[32404]: pgmap v9207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:36.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:36 smithi116 ceph-mon[37097]: pgmap v9207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16601.046250 secs 2024-07-26T13:17:37.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16601.046250 secs 2024-07-26T13:17:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16601.046250 secs 2024-07-26T13:17:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:38 smithi171 ceph-mon[37789]: pgmap v9208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:38 smithi044 ceph-mon[32404]: pgmap v9208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:38.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:38 smithi116 ceph-mon[37097]: pgmap v9208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:40.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:40 smithi044 ceph-mon[32404]: pgmap v9209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:40 smithi116 ceph-mon[37097]: pgmap v9209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:40 smithi171 ceph-mon[37789]: pgmap v9209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16606.046391 secs 2024-07-26T13:17:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:42 smithi044 ceph-mon[32404]: pgmap v9210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16606.046391 secs 2024-07-26T13:17:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:42 smithi116 ceph-mon[37097]: pgmap v9210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16606.046391 secs 2024-07-26T13:17:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:42 smithi171 ceph-mon[37789]: pgmap v9210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:44 smithi044 ceph-mon[32404]: pgmap v9211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:44 smithi116 ceph-mon[37097]: pgmap v9211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:44 smithi171 ceph-mon[37789]: pgmap v9211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:45.739+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:17:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:45.739+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:45.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:17:45.739+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:17:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:46 smithi044 ceph-mon[32404]: pgmap v9212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:46.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:46 smithi116 ceph-mon[37097]: pgmap v9212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:46 smithi171 ceph-mon[37789]: pgmap v9212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16611.046533 secs 2024-07-26T13:17:47.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16611.046533 secs 2024-07-26T13:17:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16611.046533 secs 2024-07-26T13:17:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:48 smithi044 ceph-mon[32404]: pgmap v9213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:48 smithi116 ceph-mon[37097]: pgmap v9213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:48 smithi171 ceph-mon[37789]: pgmap v9213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:50 smithi044 ceph-mon[32404]: pgmap v9214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:50 smithi116 ceph-mon[37097]: pgmap v9214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:50 smithi171 ceph-mon[37789]: pgmap v9214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16616.046690 secs 2024-07-26T13:17:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:52 smithi044 ceph-mon[32404]: pgmap v9215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16616.046690 secs 2024-07-26T13:17:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:52 smithi116 ceph-mon[37097]: pgmap v9215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16616.046690 secs 2024-07-26T13:17:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:52 smithi171 ceph-mon[37789]: pgmap v9215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:54 smithi044 ceph-mon[32404]: pgmap v9216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:54 smithi116 ceph-mon[37097]: pgmap v9216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:54 smithi171 ceph-mon[37789]: pgmap v9216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:56 smithi044 ceph-mon[32404]: pgmap v9217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:56.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:56 smithi116 ceph-mon[37097]: pgmap v9217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:56 smithi171 ceph-mon[37789]: pgmap v9217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16621.046849 secs 2024-07-26T13:17:57.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16621.046849 secs 2024-07-26T13:17:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16621.046849 secs 2024-07-26T13:17:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:17:58 smithi044 ceph-mon[32404]: pgmap v9218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:17:58 smithi116 ceph-mon[37097]: pgmap v9218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:17:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:17:58 smithi171 ceph-mon[37789]: pgmap v9218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:00.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:00.740+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:18:00.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:00.740+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:00.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:00.740+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:18:00.743 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:00 smithi044 ceph-mon[32404]: pgmap v9219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:00 smithi116 ceph-mon[37097]: pgmap v9219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:00 smithi171 ceph-mon[37789]: pgmap v9219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16626.047000 secs 2024-07-26T13:18:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:02 smithi044 ceph-mon[32404]: pgmap v9220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:02.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16626.047000 secs 2024-07-26T13:18:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:02 smithi116 ceph-mon[37097]: pgmap v9220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16626.047000 secs 2024-07-26T13:18:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:02 smithi171 ceph-mon[37789]: pgmap v9220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:04 smithi044 ceph-mon[32404]: pgmap v9221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:04 smithi116 ceph-mon[37097]: pgmap v9221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:04 smithi171 ceph-mon[37789]: pgmap v9221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:06 smithi044 ceph-mon[32404]: pgmap v9222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:06.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:06 smithi116 ceph-mon[37097]: pgmap v9222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:06 smithi171 ceph-mon[37789]: pgmap v9222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:07.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16631.047141 secs 2024-07-26T13:18:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16631.047141 secs 2024-07-26T13:18:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16631.047141 secs 2024-07-26T13:18:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:08 smithi044 ceph-mon[32404]: pgmap v9223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:08.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:08 smithi116 ceph-mon[37097]: pgmap v9223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:08 smithi171 ceph-mon[37789]: pgmap v9223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:10 smithi044 ceph-mon[32404]: pgmap v9224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:10.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:10 smithi116 ceph-mon[37097]: pgmap v9224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:10 smithi171 ceph-mon[37789]: pgmap v9224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:11 smithi044 ceph-mon[32404]: pgmap v9225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:18:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:11 smithi044 ceph-mon[32404]: pgmap v9226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:18:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:11 smithi116 ceph-mon[37097]: pgmap v9225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:18:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:11 smithi116 ceph-mon[37097]: pgmap v9226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:18:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:11 smithi171 ceph-mon[37789]: pgmap v9225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:18:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:11 smithi171 ceph-mon[37789]: pgmap v9226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:18:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16636.047293 secs 2024-07-26T13:18:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16636.047293 secs 2024-07-26T13:18:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16636.047293 secs 2024-07-26T13:18:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:13 smithi044 ceph-mon[32404]: pgmap v9227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:13 smithi116 ceph-mon[37097]: pgmap v9227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:13 smithi171 ceph-mon[37789]: pgmap v9227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:15.745+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:18:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:15.745+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:18:15.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:15.745+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:15 smithi171 ceph-mon[37789]: pgmap v9228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:15.969 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:15 smithi044 ceph-mon[32404]: pgmap v9228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:15 smithi116 ceph-mon[37097]: pgmap v9228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16641.047451 secs 2024-07-26T13:18:17.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:17 smithi171 ceph-mon[37789]: pgmap v9229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16641.047451 secs 2024-07-26T13:18:18.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:17 smithi044 ceph-mon[32404]: pgmap v9229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:18.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16641.047451 secs 2024-07-26T13:18:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:17 smithi116 ceph-mon[37097]: pgmap v9229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:18:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:19 smithi171 ceph-mon[37789]: pgmap v9230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:19 smithi044 ceph-mon[32404]: pgmap v9230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:20.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:19 smithi116 ceph-mon[37097]: pgmap v9230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:21.820 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:21 smithi044 ceph-mon[32404]: pgmap v9231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:21 smithi171 ceph-mon[37789]: pgmap v9231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:21 smithi116 ceph-mon[37097]: pgmap v9231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16646.047642 secs 2024-07-26T13:18:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16646.047642 secs 2024-07-26T13:18:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16646.047642 secs 2024-07-26T13:18:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:23 smithi171 ceph-mon[37789]: pgmap v9232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:23 smithi044 ceph-mon[32404]: pgmap v9232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:24.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:23 smithi116 ceph-mon[37097]: pgmap v9232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:25 smithi044 ceph-mon[32404]: pgmap v9233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:25 smithi116 ceph-mon[37097]: pgmap v9233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:25 smithi171 ceph-mon[37789]: pgmap v9233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16651.047775 secs 2024-07-26T13:18:28.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:27 smithi044 ceph-mon[32404]: pgmap v9234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16651.047775 secs 2024-07-26T13:18:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:27 smithi116 ceph-mon[37097]: pgmap v9234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16651.047775 secs 2024-07-26T13:18:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:27 smithi171 ceph-mon[37789]: pgmap v9234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:29 smithi044 ceph-mon[32404]: pgmap v9235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:30.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:29 smithi116 ceph-mon[37097]: pgmap v9235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:29 smithi171 ceph-mon[37789]: pgmap v9235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:30.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:30.747+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:18:30.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:30.747+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:18:30.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:30.747+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:31.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:31 smithi044 ceph-mon[32404]: pgmap v9236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:31 smithi116 ceph-mon[37097]: pgmap v9236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:31 smithi171 ceph-mon[37789]: pgmap v9236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16656.047937 secs 2024-07-26T13:18:33.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16656.047937 secs 2024-07-26T13:18:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16656.047937 secs 2024-07-26T13:18:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:33 smithi044 ceph-mon[32404]: pgmap v9237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:33 smithi116 ceph-mon[37097]: pgmap v9237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:33 smithi171 ceph-mon[37789]: pgmap v9237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:35 smithi044 ceph-mon[32404]: pgmap v9238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:35 smithi116 ceph-mon[37097]: pgmap v9238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:35 smithi171 ceph-mon[37789]: pgmap v9238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16661.048076 secs 2024-07-26T13:18:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16661.048076 secs 2024-07-26T13:18:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16661.048076 secs 2024-07-26T13:18:37.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:37 smithi116 ceph-mon[37097]: pgmap v9239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:37.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:37 smithi171 ceph-mon[37789]: pgmap v9239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:37 smithi044 ceph-mon[32404]: pgmap v9239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:39 smithi044 ceph-mon[32404]: pgmap v9240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:39 smithi116 ceph-mon[37097]: pgmap v9240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:39 smithi171 ceph-mon[37789]: pgmap v9240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:41 smithi044 ceph-mon[32404]: pgmap v9241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16666.048226 secs 2024-07-26T13:18:42.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:41 smithi116 ceph-mon[37097]: pgmap v9241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16666.048226 secs 2024-07-26T13:18:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:41 smithi171 ceph-mon[37789]: pgmap v9241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16666.048226 secs 2024-07-26T13:18:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:43 smithi044 ceph-mon[32404]: pgmap v9242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:43 smithi116 ceph-mon[37097]: pgmap v9242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:43 smithi171 ceph-mon[37789]: pgmap v9242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:45.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:45.753+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:18:45.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:45.753+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:18:45.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:18:45.753+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:45 smithi044 ceph-mon[32404]: pgmap v9243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:45 smithi116 ceph-mon[37097]: pgmap v9243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:45 smithi171 ceph-mon[37789]: pgmap v9243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16671.048399 secs 2024-07-26T13:18:47.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16671.048399 secs 2024-07-26T13:18:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16671.048399 secs 2024-07-26T13:18:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:47 smithi044 ceph-mon[32404]: pgmap v9244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:47 smithi116 ceph-mon[37097]: pgmap v9244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:47 smithi171 ceph-mon[37789]: pgmap v9244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:49 smithi044 ceph-mon[32404]: pgmap v9245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:49 smithi116 ceph-mon[37097]: pgmap v9245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:49 smithi171 ceph-mon[37789]: pgmap v9245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:18:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:51 smithi044 ceph-mon[32404]: pgmap v9246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16676.048532 secs 2024-07-26T13:18:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:51 smithi116 ceph-mon[37097]: pgmap v9246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16676.048532 secs 2024-07-26T13:18:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:51 smithi171 ceph-mon[37789]: pgmap v9246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16676.048532 secs 2024-07-26T13:18:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:54 smithi171 ceph-mon[37789]: pgmap v9247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:54 smithi044 ceph-mon[32404]: pgmap v9247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:54 smithi116 ceph-mon[37097]: pgmap v9247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:56 smithi171 ceph-mon[37789]: pgmap v9248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:56 smithi044 ceph-mon[32404]: pgmap v9248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:56.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:56 smithi116 ceph-mon[37097]: pgmap v9248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16681.048667 secs 2024-07-26T13:18:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16681.048667 secs 2024-07-26T13:18:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16681.048667 secs 2024-07-26T13:18:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:18:58 smithi171 ceph-mon[37789]: pgmap v9249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:18:58 smithi044 ceph-mon[32404]: pgmap v9249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:18:58 smithi116 ceph-mon[37097]: pgmap v9249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:00 smithi171 ceph-mon[37789]: pgmap v9250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:19:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:00 smithi044 ceph-mon[32404]: pgmap v9250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:19:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:00 smithi116 ceph-mon[37097]: pgmap v9250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:19:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:00.754+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:19:00.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:00.754+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:00.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:00.754+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:19:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:02 smithi171 ceph-mon[37789]: pgmap v9251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16686.048828 secs 2024-07-26T13:19:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:02 smithi044 ceph-mon[32404]: pgmap v9251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16686.048828 secs 2024-07-26T13:19:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:02 smithi116 ceph-mon[37097]: pgmap v9251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16686.048828 secs 2024-07-26T13:19:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:04 smithi171 ceph-mon[37789]: pgmap v9252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:04 smithi044 ceph-mon[32404]: pgmap v9252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:04 smithi116 ceph-mon[37097]: pgmap v9252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:06 smithi171 ceph-mon[37789]: pgmap v9253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:06 smithi044 ceph-mon[32404]: pgmap v9253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:06 smithi116 ceph-mon[37097]: pgmap v9253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:07.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16691.048960 secs 2024-07-26T13:19:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16691.048960 secs 2024-07-26T13:19:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16691.048960 secs 2024-07-26T13:19:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:08 smithi044 ceph-mon[32404]: pgmap v9254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:08 smithi116 ceph-mon[37097]: pgmap v9254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:08 smithi171 ceph-mon[37789]: pgmap v9254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:10 smithi044 ceph-mon[32404]: pgmap v9255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:19:10.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:10 smithi116 ceph-mon[37097]: pgmap v9255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:19:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:10 smithi171 ceph-mon[37789]: pgmap v9255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:19:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:11.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:12 smithi044 ceph-mon[32404]: pgmap v9256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16696.049109 secs 2024-07-26T13:19:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:19:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:12 smithi116 ceph-mon[37097]: pgmap v9256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16696.049109 secs 2024-07-26T13:19:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:19:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:12 smithi171 ceph-mon[37789]: pgmap v9256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16696.049109 secs 2024-07-26T13:19:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:19:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:13 smithi044 ceph-mon[32404]: pgmap v9257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:13.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:13 smithi116 ceph-mon[37097]: pgmap v9257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:13 smithi171 ceph-mon[37789]: pgmap v9257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:15 smithi044 ceph-mon[32404]: pgmap v9258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:15 smithi116 ceph-mon[37097]: pgmap v9258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:15 smithi171 ceph-mon[37789]: pgmap v9258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:15.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:15.758+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:19:15.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:15.758+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:15.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:15.758+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:19:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16701.049268 secs 2024-07-26T13:19:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:17 smithi044 ceph-mon[32404]: pgmap v9259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:17.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16701.049268 secs 2024-07-26T13:19:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:17 smithi116 ceph-mon[37097]: pgmap v9259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16701.049268 secs 2024-07-26T13:19:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:17 smithi171 ceph-mon[37789]: pgmap v9259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:19 smithi044 ceph-mon[32404]: pgmap v9260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:19 smithi116 ceph-mon[37097]: pgmap v9260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:19 smithi171 ceph-mon[37789]: pgmap v9260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:21 smithi044 ceph-mon[32404]: pgmap v9261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:21 smithi116 ceph-mon[37097]: pgmap v9261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:21 smithi171 ceph-mon[37789]: pgmap v9261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16706.049431 secs 2024-07-26T13:19:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16706.049431 secs 2024-07-26T13:19:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16706.049431 secs 2024-07-26T13:19:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:23 smithi044 ceph-mon[32404]: pgmap v9262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:23 smithi116 ceph-mon[37097]: pgmap v9262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:23 smithi171 ceph-mon[37789]: pgmap v9262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:19:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:25 smithi044 ceph-mon[32404]: pgmap v9263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:25 smithi116 ceph-mon[37097]: pgmap v9263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:25 smithi171 ceph-mon[37789]: pgmap v9263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16711.049590 secs 2024-07-26T13:19:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:27 smithi044 ceph-mon[32404]: pgmap v9264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:27.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16711.049590 secs 2024-07-26T13:19:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:27 smithi116 ceph-mon[37097]: pgmap v9264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16711.049590 secs 2024-07-26T13:19:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:27 smithi171 ceph-mon[37789]: pgmap v9264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:29 smithi044 ceph-mon[32404]: pgmap v9265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:29 smithi116 ceph-mon[37097]: pgmap v9265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:29 smithi171 ceph-mon[37789]: pgmap v9265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:30.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:30.760+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:19:30.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:30.760+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:30.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:30.760+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:19:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:31 smithi044 ceph-mon[32404]: pgmap v9266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:31 smithi116 ceph-mon[37097]: pgmap v9266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:31 smithi171 ceph-mon[37789]: pgmap v9266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16716.049742 secs 2024-07-26T13:19:32.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16716.049742 secs 2024-07-26T13:19:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16716.049742 secs 2024-07-26T13:19:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:33 smithi044 ceph-mon[32404]: pgmap v9267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:33 smithi116 ceph-mon[37097]: pgmap v9267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:33 smithi171 ceph-mon[37789]: pgmap v9267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:35 smithi044 ceph-mon[32404]: pgmap v9268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:35 smithi116 ceph-mon[37097]: pgmap v9268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:35 smithi171 ceph-mon[37789]: pgmap v9268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16721.049887 secs 2024-07-26T13:19:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:37 smithi044 ceph-mon[32404]: pgmap v9269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16721.049887 secs 2024-07-26T13:19:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:37 smithi116 ceph-mon[37097]: pgmap v9269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16721.049887 secs 2024-07-26T13:19:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:37 smithi171 ceph-mon[37789]: pgmap v9269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:39 smithi044 ceph-mon[32404]: pgmap v9270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:39 smithi116 ceph-mon[37097]: pgmap v9270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:39 smithi171 ceph-mon[37789]: pgmap v9270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:19:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:41 smithi044 ceph-mon[32404]: pgmap v9271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:41.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:41 smithi116 ceph-mon[37097]: pgmap v9271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:41 smithi171 ceph-mon[37789]: pgmap v9271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:19:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16726.050018 secs 2024-07-26T13:19:42.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16726.050018 secs 2024-07-26T13:19:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16726.050018 secs 2024-07-26T13:19:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:43 smithi044 ceph-mon[32404]: pgmap v9272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:43.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:43 smithi116 ceph-mon[37097]: pgmap v9272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:43 smithi171 ceph-mon[37789]: pgmap v9272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:45 smithi044 ceph-mon[32404]: pgmap v9273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:45 smithi116 ceph-mon[37097]: pgmap v9273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:45 smithi171 ceph-mon[37789]: pgmap v9273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:45.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:45.766+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:19:45.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:45.766+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:45.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:19:45.766+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:19:47.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16731.050162 secs 2024-07-26T13:19:47.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:47 smithi044 ceph-mon[32404]: pgmap v9274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16731.050162 secs 2024-07-26T13:19:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:47 smithi116 ceph-mon[37097]: pgmap v9274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16731.050162 secs 2024-07-26T13:19:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:47 smithi171 ceph-mon[37789]: pgmap v9274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:49 smithi044 ceph-mon[32404]: pgmap v9275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:19:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:49 smithi116 ceph-mon[37097]: pgmap v9275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:19:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:49 smithi171 ceph-mon[37789]: pgmap v9275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:19:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:51 smithi116 ceph-mon[37097]: pgmap v9276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:19:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:51 smithi171 ceph-mon[37789]: pgmap v9276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:19:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:51 smithi044 ceph-mon[32404]: pgmap v9276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:19:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16736.050314 secs 2024-07-26T13:19:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16736.050314 secs 2024-07-26T13:19:52.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16736.050314 secs 2024-07-26T13:19:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:53 smithi171 ceph-mon[37789]: pgmap v9277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:53 smithi044 ceph-mon[32404]: pgmap v9277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:53 smithi116 ceph-mon[37097]: pgmap v9277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:54 smithi171 ceph-mon[37789]: pgmap v9278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:19:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:54 smithi044 ceph-mon[32404]: pgmap v9278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:19:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:54 smithi116 ceph-mon[37097]: pgmap v9278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:19:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16741.050452 secs 2024-07-26T13:19:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:56 smithi044 ceph-mon[32404]: pgmap v9279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:19:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16741.050452 secs 2024-07-26T13:19:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:56 smithi116 ceph-mon[37097]: pgmap v9279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:19:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16741.050452 secs 2024-07-26T13:19:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:56 smithi171 ceph-mon[37789]: pgmap v9279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:19:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:19:58 smithi044 ceph-mon[32404]: pgmap v9280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:19:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:19:58 smithi116 ceph-mon[37097]: pgmap v9280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:19:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:19:58 smithi171 ceph-mon[37789]: pgmap v9280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:20:00.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:00.767+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:20:00.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:00.767+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:00.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:00.767+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:20:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:00 smithi044 ceph-mon[32404]: pgmap v9281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:00 smithi116 ceph-mon[37097]: pgmap v9281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:00 smithi171 ceph-mon[37789]: pgmap v9281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16746.050672 secs 2024-07-26T13:20:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16746.050672 secs 2024-07-26T13:20:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16746.050672 secs 2024-07-26T13:20:03.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:02 smithi044 ceph-mon[32404]: pgmap v9282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:02 smithi116 ceph-mon[37097]: pgmap v9282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:02 smithi171 ceph-mon[37789]: pgmap v9282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:04 smithi044 ceph-mon[32404]: pgmap v9283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:04 smithi116 ceph-mon[37097]: pgmap v9283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:04 smithi171 ceph-mon[37789]: pgmap v9283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16751.050767 secs 2024-07-26T13:20:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:06 smithi044 ceph-mon[32404]: pgmap v9284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16751.050767 secs 2024-07-26T13:20:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:06 smithi116 ceph-mon[37097]: pgmap v9284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16751.050767 secs 2024-07-26T13:20:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:06 smithi171 ceph-mon[37789]: pgmap v9284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:08 smithi044 ceph-mon[32404]: pgmap v9285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:08 smithi116 ceph-mon[37097]: pgmap v9285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:08 smithi171 ceph-mon[37789]: pgmap v9285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:10 smithi044 ceph-mon[32404]: pgmap v9286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:10 smithi116 ceph-mon[37097]: pgmap v9286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:10 smithi171 ceph-mon[37789]: pgmap v9286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:20:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16756.050934 secs 2024-07-26T13:20:12.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16756.050934 secs 2024-07-26T13:20:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16756.050934 secs 2024-07-26T13:20:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:12 smithi044 ceph-mon[32404]: pgmap v9287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:13.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:13.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:12 smithi116 ceph-mon[37097]: pgmap v9287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:12 smithi171 ceph-mon[37789]: pgmap v9287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:13.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:20:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:20:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: pgmap v9288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: pgmap v9289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:14.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:13 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:20:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:20:14.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:14.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:14.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: pgmap v9288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:14.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: pgmap v9289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:14.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:13 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:20:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: pgmap v9288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:14.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: pgmap v9289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:14.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:13 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:20:15.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:15.772+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:20:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:15.772+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:20:15.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:15.772+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:16 smithi044 ceph-mon[32404]: pgmap v9290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:16 smithi116 ceph-mon[37097]: pgmap v9290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:16 smithi171 ceph-mon[37789]: pgmap v9290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16761.051085 secs 2024-07-26T13:20:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16761.051085 secs 2024-07-26T13:20:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16761.051085 secs 2024-07-26T13:20:17.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:17.704+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:20:17.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:17.704+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:20:17.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:17.704+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:20:17.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:17.704+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:20:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:18 smithi044 ceph-mon[32404]: pgmap v9291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:18 smithi116 ceph-mon[37097]: pgmap v9291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:18 smithi171 ceph-mon[37789]: pgmap v9291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:20 smithi044 ceph-mon[32404]: pgmap v9292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:20 smithi116 ceph-mon[37097]: pgmap v9292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:20 smithi171 ceph-mon[37789]: pgmap v9292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16766.051257 secs 2024-07-26T13:20:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:22 smithi044 ceph-mon[32404]: pgmap v9293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:20:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16766.051257 secs 2024-07-26T13:20:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:22 smithi116 ceph-mon[37097]: pgmap v9293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:20:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16766.051257 secs 2024-07-26T13:20:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:22 smithi171 ceph-mon[37789]: pgmap v9293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:20:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:24 smithi044 ceph-mon[32404]: pgmap v9294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:24 smithi116 ceph-mon[37097]: pgmap v9294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:24 smithi171 ceph-mon[37789]: pgmap v9294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:26 smithi044 ceph-mon[32404]: pgmap v9295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:26.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:26 smithi116 ceph-mon[37097]: pgmap v9295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:26 smithi171 ceph-mon[37789]: pgmap v9295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16771.051418 secs 2024-07-26T13:20:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16771.051418 secs 2024-07-26T13:20:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16771.051418 secs 2024-07-26T13:20:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:28 smithi044 ceph-mon[32404]: pgmap v9296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:28 smithi116 ceph-mon[37097]: pgmap v9296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:28 smithi171 ceph-mon[37789]: pgmap v9296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:30 smithi044 ceph-mon[32404]: pgmap v9297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:30 smithi116 ceph-mon[37097]: pgmap v9297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:30 smithi171 ceph-mon[37789]: pgmap v9297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:30.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:30.773+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:20:30.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:30.773+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:20:30.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:30.773+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16776.051584 secs 2024-07-26T13:20:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:32 smithi044 ceph-mon[32404]: pgmap v9298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16776.051584 secs 2024-07-26T13:20:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:32 smithi116 ceph-mon[37097]: pgmap v9298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16776.051584 secs 2024-07-26T13:20:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:32 smithi171 ceph-mon[37789]: pgmap v9298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:34 smithi044 ceph-mon[32404]: pgmap v9299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:34 smithi116 ceph-mon[37097]: pgmap v9299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:34 smithi171 ceph-mon[37789]: pgmap v9299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:36 smithi044 ceph-mon[32404]: pgmap v9300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:36.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:36 smithi116 ceph-mon[37097]: pgmap v9300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:36 smithi171 ceph-mon[37789]: pgmap v9300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16781.051754 secs 2024-07-26T13:20:37.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16781.051754 secs 2024-07-26T13:20:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16781.051754 secs 2024-07-26T13:20:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:38 smithi044 ceph-mon[32404]: pgmap v9301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:38 smithi116 ceph-mon[37097]: pgmap v9301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:38 smithi171 ceph-mon[37789]: pgmap v9301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:40 smithi044 ceph-mon[32404]: pgmap v9302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:40.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:40 smithi116 ceph-mon[37097]: pgmap v9302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:40 smithi171 ceph-mon[37789]: pgmap v9302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16786.051907 secs 2024-07-26T13:20:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:42 smithi044 ceph-mon[32404]: pgmap v9303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16786.051907 secs 2024-07-26T13:20:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:42 smithi116 ceph-mon[37097]: pgmap v9303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16786.051907 secs 2024-07-26T13:20:42.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:42 smithi171 ceph-mon[37789]: pgmap v9303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:44 smithi044 ceph-mon[32404]: pgmap v9304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:44 smithi116 ceph-mon[37097]: pgmap v9304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:44 smithi171 ceph-mon[37789]: pgmap v9304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:45.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:45.778+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:20:45.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:45.779+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:45.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:20:45.779+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:20:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:46 smithi044 ceph-mon[32404]: pgmap v9305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:46.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:46 smithi116 ceph-mon[37097]: pgmap v9305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:46 smithi171 ceph-mon[37789]: pgmap v9305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16791.052065 secs 2024-07-26T13:20:47.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16791.052065 secs 2024-07-26T13:20:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16791.052065 secs 2024-07-26T13:20:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:48 smithi044 ceph-mon[32404]: pgmap v9306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:48.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:48 smithi116 ceph-mon[37097]: pgmap v9306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:48 smithi171 ceph-mon[37789]: pgmap v9306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:50 smithi116 ceph-mon[37097]: pgmap v9307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:50 smithi171 ceph-mon[37789]: pgmap v9307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:50 smithi044 ceph-mon[32404]: pgmap v9307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:20:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16796.052218 secs 2024-07-26T13:20:52.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:52 smithi171 ceph-mon[37789]: pgmap v9308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16796.052218 secs 2024-07-26T13:20:52.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:52 smithi044 ceph-mon[32404]: pgmap v9308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16796.052218 secs 2024-07-26T13:20:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:52 smithi116 ceph-mon[37097]: pgmap v9308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:54 smithi171 ceph-mon[37789]: pgmap v9309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:20:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:54 smithi044 ceph-mon[32404]: pgmap v9309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:20:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:54 smithi116 ceph-mon[37097]: pgmap v9309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:20:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:56 smithi171 ceph-mon[37789]: pgmap v9310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:20:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:56 smithi044 ceph-mon[32404]: pgmap v9310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:20:56.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:56 smithi116 ceph-mon[37097]: pgmap v9310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:20:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16801.052377 secs 2024-07-26T13:20:57.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:20:57 smithi171 ceph-mon[37789]: pgmap v9311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:20:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16801.052377 secs 2024-07-26T13:20:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:20:57 smithi044 ceph-mon[32404]: pgmap v9311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:20:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16801.052377 secs 2024-07-26T13:20:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:20:57 smithi116 ceph-mon[37097]: pgmap v9311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:21:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:00 smithi044 ceph-mon[32404]: pgmap v9312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:21:00.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:00 smithi116 ceph-mon[37097]: pgmap v9312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:21:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:00 smithi171 ceph-mon[37789]: pgmap v9312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:21:00.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:00.780+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:21:00.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:00.780+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:21:00.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:00.780+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16806.052538 secs 2024-07-26T13:21:02.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:02 smithi044 ceph-mon[32404]: pgmap v9313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:21:02.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16806.052538 secs 2024-07-26T13:21:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:02 smithi116 ceph-mon[37097]: pgmap v9313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:21:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16806.052538 secs 2024-07-26T13:21:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:02 smithi171 ceph-mon[37789]: pgmap v9313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:21:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:04 smithi044 ceph-mon[32404]: pgmap v9314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:21:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:04 smithi116 ceph-mon[37097]: pgmap v9314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:21:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:04 smithi171 ceph-mon[37789]: pgmap v9314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:21:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:06 smithi044 ceph-mon[32404]: pgmap v9315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:06 smithi116 ceph-mon[37097]: pgmap v9315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:06 smithi171 ceph-mon[37789]: pgmap v9315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16811.052704 secs 2024-07-26T13:21:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16811.052704 secs 2024-07-26T13:21:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16811.052704 secs 2024-07-26T13:21:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:08 smithi044 ceph-mon[32404]: pgmap v9316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:08.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:08 smithi116 ceph-mon[37097]: pgmap v9316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:08 smithi171 ceph-mon[37789]: pgmap v9316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:10 smithi044 ceph-mon[32404]: pgmap v9317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:10 smithi116 ceph-mon[37097]: pgmap v9317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:10 smithi171 ceph-mon[37789]: pgmap v9317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16816.052875 secs 2024-07-26T13:21:12.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:12 smithi044 ceph-mon[32404]: pgmap v9318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16816.052875 secs 2024-07-26T13:21:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:12 smithi116 ceph-mon[37097]: pgmap v9318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16816.052875 secs 2024-07-26T13:21:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:12 smithi171 ceph-mon[37789]: pgmap v9318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: pgmap v9319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:14.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: pgmap v9319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: pgmap v9319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:14.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:21:15.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:21:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:15.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:21:15.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:15.786+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:21:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:15.786+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:21:15.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:15.786+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:16 smithi044 ceph-mon[32404]: pgmap v9320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:21:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:16 smithi044 ceph-mon[32404]: pgmap v9321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:16 smithi116 ceph-mon[37097]: pgmap v9320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:21:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:16 smithi116 ceph-mon[37097]: pgmap v9321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:16 smithi171 ceph-mon[37789]: pgmap v9320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:21:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:16 smithi171 ceph-mon[37789]: pgmap v9321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16821.053039 secs 2024-07-26T13:21:17.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16821.053039 secs 2024-07-26T13:21:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16821.053039 secs 2024-07-26T13:21:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:18 smithi044 ceph-mon[32404]: pgmap v9322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:21:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:18 smithi116 ceph-mon[37097]: pgmap v9322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:21:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:18 smithi171 ceph-mon[37789]: pgmap v9322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:21:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:20 smithi044 ceph-mon[32404]: pgmap v9323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:20.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:20 smithi116 ceph-mon[37097]: pgmap v9323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:20 smithi171 ceph-mon[37789]: pgmap v9323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:22 smithi044 ceph-mon[32404]: pgmap v9324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:22.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16826.053206 secs 2024-07-26T13:21:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:22 smithi116 ceph-mon[37097]: pgmap v9324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16826.053206 secs 2024-07-26T13:21:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:22 smithi171 ceph-mon[37789]: pgmap v9324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:21:22.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16826.053206 secs 2024-07-26T13:21:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:24 smithi044 ceph-mon[32404]: pgmap v9325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:21:24.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:24 smithi116 ceph-mon[37097]: pgmap v9325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:21:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:24 smithi171 ceph-mon[37789]: pgmap v9325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:21:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:26 smithi116 ceph-mon[37097]: pgmap v9326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:26 smithi171 ceph-mon[37789]: pgmap v9326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:26 smithi044 ceph-mon[32404]: pgmap v9326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16831.053367 secs 2024-07-26T13:21:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16831.053367 secs 2024-07-26T13:21:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16831.053367 secs 2024-07-26T13:21:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:28 smithi171 ceph-mon[37789]: pgmap v9327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:28 smithi044 ceph-mon[32404]: pgmap v9327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:28 smithi116 ceph-mon[37097]: pgmap v9327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:30.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:30 smithi171 ceph-mon[37789]: pgmap v9328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:30 smithi044 ceph-mon[32404]: pgmap v9328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:30 smithi116 ceph-mon[37097]: pgmap v9328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:30.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:30.786+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:21:30.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:30.786+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:21:30.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:30.786+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:32 smithi171 ceph-mon[37789]: pgmap v9329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16836.053521 secs 2024-07-26T13:21:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:32 smithi044 ceph-mon[32404]: pgmap v9329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16836.053521 secs 2024-07-26T13:21:32.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:32 smithi116 ceph-mon[37097]: pgmap v9329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16836.053521 secs 2024-07-26T13:21:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:33 smithi171 ceph-mon[37789]: pgmap v9330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:33 smithi044 ceph-mon[32404]: pgmap v9330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:33 smithi116 ceph-mon[37097]: pgmap v9330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:35 smithi044 ceph-mon[32404]: pgmap v9331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:35.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:35 smithi116 ceph-mon[37097]: pgmap v9331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:35 smithi171 ceph-mon[37789]: pgmap v9331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16841.053686 secs 2024-07-26T13:21:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:37 smithi044 ceph-mon[32404]: pgmap v9332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:37.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16841.053686 secs 2024-07-26T13:21:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:37 smithi116 ceph-mon[37097]: pgmap v9332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16841.053686 secs 2024-07-26T13:21:37.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:37 smithi171 ceph-mon[37789]: pgmap v9332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:39 smithi044 ceph-mon[32404]: pgmap v9333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:39 smithi116 ceph-mon[37097]: pgmap v9333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:39 smithi171 ceph-mon[37789]: pgmap v9333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:41 smithi044 ceph-mon[32404]: pgmap v9334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:41 smithi116 ceph-mon[37097]: pgmap v9334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:41 smithi171 ceph-mon[37789]: pgmap v9334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16846.053846 secs 2024-07-26T13:21:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16846.053846 secs 2024-07-26T13:21:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16846.053846 secs 2024-07-26T13:21:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:43 smithi044 ceph-mon[32404]: pgmap v9335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:43 smithi116 ceph-mon[37097]: pgmap v9335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:43 smithi171 ceph-mon[37789]: pgmap v9335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:45 smithi044 ceph-mon[32404]: pgmap v9336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:45 smithi116 ceph-mon[37097]: pgmap v9336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:45.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:45.789+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:21:45.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:45.789+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:21:45.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:45.789+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:45 smithi171 ceph-mon[37789]: pgmap v9336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:47.721+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:21:47.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:47.721+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:21:47.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16851.054006 secs 2024-07-26T13:21:47.723 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:47 smithi044 ceph-mon[32404]: pgmap v9337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:47.722+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:21:47.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:21:47.722+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:21:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16851.054006 secs 2024-07-26T13:21:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:47 smithi116 ceph-mon[37097]: pgmap v9337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16851.054006 secs 2024-07-26T13:21:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:47 smithi171 ceph-mon[37789]: pgmap v9337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:49 smithi044 ceph-mon[32404]: pgmap v9338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:49 smithi116 ceph-mon[37097]: pgmap v9338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:49 smithi171 ceph-mon[37789]: pgmap v9338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:51 smithi044 ceph-mon[32404]: pgmap v9339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:51 smithi116 ceph-mon[37097]: pgmap v9339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:51 smithi171 ceph-mon[37789]: pgmap v9339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16856.054151 secs 2024-07-26T13:21:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16856.054151 secs 2024-07-26T13:21:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16856.054151 secs 2024-07-26T13:21:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:53 smithi044 ceph-mon[32404]: pgmap v9340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:53.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:53 smithi116 ceph-mon[37097]: pgmap v9340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:53 smithi171 ceph-mon[37789]: pgmap v9340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:21:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:55 smithi044 ceph-mon[32404]: pgmap v9341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:55.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:55 smithi116 ceph-mon[37097]: pgmap v9341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:55 smithi171 ceph-mon[37789]: pgmap v9341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:21:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16861.054303 secs 2024-07-26T13:21:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:57 smithi044 ceph-mon[32404]: pgmap v9342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16861.054303 secs 2024-07-26T13:21:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:57 smithi116 ceph-mon[37097]: pgmap v9342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16861.054303 secs 2024-07-26T13:21:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:57 smithi171 ceph-mon[37789]: pgmap v9342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:21:59 smithi044 ceph-mon[32404]: pgmap v9343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:21:59 smithi116 ceph-mon[37097]: pgmap v9343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:21:59 smithi171 ceph-mon[37789]: pgmap v9343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:00.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:00.790+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:22:00.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:00.790+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:22:00.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:00.790+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:01 smithi044 ceph-mon[32404]: pgmap v9344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:01 smithi116 ceph-mon[37097]: pgmap v9344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:01 smithi171 ceph-mon[37789]: pgmap v9344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16866.054459 secs 2024-07-26T13:22:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16866.054459 secs 2024-07-26T13:22:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16866.054459 secs 2024-07-26T13:22:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:03 smithi116 ceph-mon[37097]: pgmap v9345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:03 smithi171 ceph-mon[37789]: pgmap v9345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:03 smithi044 ceph-mon[32404]: pgmap v9345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:05 smithi171 ceph-mon[37789]: pgmap v9346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:05 smithi044 ceph-mon[32404]: pgmap v9346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:05 smithi116 ceph-mon[37097]: pgmap v9346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:07.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16871.054611 secs 2024-07-26T13:22:07.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:07 smithi116 ceph-mon[37097]: pgmap v9347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16871.054611 secs 2024-07-26T13:22:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:07 smithi171 ceph-mon[37789]: pgmap v9347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16871.054611 secs 2024-07-26T13:22:08.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:07 smithi044 ceph-mon[32404]: pgmap v9347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:09 smithi171 ceph-mon[37789]: pgmap v9348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:09.953 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:09 smithi044 ceph-mon[32404]: pgmap v9348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:09 smithi116 ceph-mon[37097]: pgmap v9348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:11 smithi171 ceph-mon[37789]: pgmap v9349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:11 smithi044 ceph-mon[32404]: pgmap v9349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:11 smithi116 ceph-mon[37097]: pgmap v9349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16876.054751 secs 2024-07-26T13:22:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16876.054751 secs 2024-07-26T13:22:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16876.054751 secs 2024-07-26T13:22:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:14 smithi044 ceph-mon[32404]: pgmap v9350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:14 smithi116 ceph-mon[37097]: pgmap v9350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:14 smithi171 ceph-mon[37789]: pgmap v9350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:15.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:15.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:15.796+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:22:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:15.796+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:22:15.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:15.796+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:16 smithi044 ceph-mon[32404]: pgmap v9351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:22:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:16 smithi116 ceph-mon[37097]: pgmap v9351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:22:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:16 smithi171 ceph-mon[37789]: pgmap v9351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:22:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:17 smithi044 ceph-mon[32404]: pgmap v9352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:22:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:17 smithi044 ceph-mon[32404]: pgmap v9353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16881.054854 secs 2024-07-26T13:22:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:17 smithi116 ceph-mon[37097]: pgmap v9352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:22:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:17 smithi116 ceph-mon[37097]: pgmap v9353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16881.054854 secs 2024-07-26T13:22:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:17 smithi171 ceph-mon[37789]: pgmap v9352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:22:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:17 smithi171 ceph-mon[37789]: pgmap v9353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16881.054854 secs 2024-07-26T13:22:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:19 smithi044 ceph-mon[32404]: pgmap v9354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:19.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:19 smithi116 ceph-mon[37097]: pgmap v9354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:19 smithi171 ceph-mon[37789]: pgmap v9354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:21 smithi044 ceph-mon[32404]: pgmap v9355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:21 smithi116 ceph-mon[37097]: pgmap v9355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:21 smithi171 ceph-mon[37789]: pgmap v9355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16886.055035 secs 2024-07-26T13:22:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16886.055035 secs 2024-07-26T13:22:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16886.055035 secs 2024-07-26T13:22:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:23 smithi044 ceph-mon[32404]: pgmap v9356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:23 smithi116 ceph-mon[37097]: pgmap v9356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:23 smithi171 ceph-mon[37789]: pgmap v9356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:25 smithi044 ceph-mon[32404]: pgmap v9357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:25.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:25 smithi116 ceph-mon[37097]: pgmap v9357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:25 smithi171 ceph-mon[37789]: pgmap v9357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:27 smithi044 ceph-mon[32404]: pgmap v9358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16891.055213 secs 2024-07-26T13:22:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:27 smithi116 ceph-mon[37097]: pgmap v9358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16891.055213 secs 2024-07-26T13:22:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:27 smithi171 ceph-mon[37789]: pgmap v9358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16891.055213 secs 2024-07-26T13:22:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:29 smithi044 ceph-mon[32404]: pgmap v9359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:29 smithi116 ceph-mon[37097]: pgmap v9359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:29 smithi171 ceph-mon[37789]: pgmap v9359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:30.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:30.796+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:22:30.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:30.796+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:22:30.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:30.796+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:31 smithi044 ceph-mon[32404]: pgmap v9360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:31 smithi116 ceph-mon[37097]: pgmap v9360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:31 smithi171 ceph-mon[37789]: pgmap v9360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16896.055386 secs 2024-07-26T13:22:32.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16896.055386 secs 2024-07-26T13:22:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16896.055386 secs 2024-07-26T13:22:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:33 smithi044 ceph-mon[32404]: pgmap v9361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:33 smithi116 ceph-mon[37097]: pgmap v9361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:33 smithi171 ceph-mon[37789]: pgmap v9361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:35 smithi044 ceph-mon[32404]: pgmap v9362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:35 smithi116 ceph-mon[37097]: pgmap v9362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:35 smithi171 ceph-mon[37789]: pgmap v9362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:37 smithi044 ceph-mon[32404]: pgmap v9363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:37.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16901.055526 secs 2024-07-26T13:22:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:37 smithi116 ceph-mon[37097]: pgmap v9363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16901.055526 secs 2024-07-26T13:22:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:37 smithi171 ceph-mon[37789]: pgmap v9363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16901.055526 secs 2024-07-26T13:22:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:39 smithi044 ceph-mon[32404]: pgmap v9364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:39 smithi116 ceph-mon[37097]: pgmap v9364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:39 smithi171 ceph-mon[37789]: pgmap v9364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:41 smithi044 ceph-mon[32404]: pgmap v9365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:41 smithi116 ceph-mon[37097]: pgmap v9365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:41 smithi171 ceph-mon[37789]: pgmap v9365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16906.055669 secs 2024-07-26T13:22:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16906.055669 secs 2024-07-26T13:22:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16906.055669 secs 2024-07-26T13:22:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:43 smithi116 ceph-mon[37097]: pgmap v9366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:43 smithi171 ceph-mon[37789]: pgmap v9366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:43 smithi044 ceph-mon[32404]: pgmap v9366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:45 smithi116 ceph-mon[37097]: pgmap v9367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:45 smithi171 ceph-mon[37789]: pgmap v9367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:45 smithi044 ceph-mon[32404]: pgmap v9367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:45.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:45.801+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:22:45.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:45.802+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:22:45.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:22:45.802+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:47 smithi171 ceph-mon[37789]: pgmap v9368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:47.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16911.055811 secs 2024-07-26T13:22:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:47 smithi044 ceph-mon[32404]: pgmap v9368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:47.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16911.055811 secs 2024-07-26T13:22:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:47 smithi116 ceph-mon[37097]: pgmap v9368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16911.055811 secs 2024-07-26T13:22:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:49 smithi171 ceph-mon[37789]: pgmap v9369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:49 smithi044 ceph-mon[32404]: pgmap v9369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:49 smithi116 ceph-mon[37097]: pgmap v9369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:51 smithi171 ceph-mon[37789]: pgmap v9370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:51 smithi044 ceph-mon[32404]: pgmap v9370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:51.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:51 smithi116 ceph-mon[37097]: pgmap v9370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:22:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16916.055951 secs 2024-07-26T13:22:52.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16916.055951 secs 2024-07-26T13:22:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16916.055951 secs 2024-07-26T13:22:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:53 smithi171 ceph-mon[37789]: pgmap v9371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:53 smithi044 ceph-mon[32404]: pgmap v9371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:53 smithi116 ceph-mon[37097]: pgmap v9371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:55 smithi171 ceph-mon[37789]: pgmap v9372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:55 smithi044 ceph-mon[32404]: pgmap v9372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:55 smithi116 ceph-mon[37097]: pgmap v9372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:57 smithi171 ceph-mon[37789]: pgmap v9373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16921.056107 secs 2024-07-26T13:22:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:57 smithi044 ceph-mon[32404]: pgmap v9373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:57.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16921.056107 secs 2024-07-26T13:22:57.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:57 smithi116 ceph-mon[37097]: pgmap v9373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16921.056107 secs 2024-07-26T13:22:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:22:58 smithi044 ceph-mon[32404]: pgmap v9374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:22:58 smithi116 ceph-mon[37097]: pgmap v9374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:22:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:22:58 smithi171 ceph-mon[37789]: pgmap v9374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:00.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:00.803+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:00.803+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:23:00.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:00.803+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:00 smithi044 ceph-mon[32404]: pgmap v9375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:01.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:00 smithi116 ceph-mon[37097]: pgmap v9375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:00 smithi171 ceph-mon[37789]: pgmap v9375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16926.056273 secs 2024-07-26T13:23:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16926.056273 secs 2024-07-26T13:23:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16926.056273 secs 2024-07-26T13:23:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:02 smithi044 ceph-mon[32404]: pgmap v9376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:02 smithi116 ceph-mon[37097]: pgmap v9376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:02 smithi171 ceph-mon[37789]: pgmap v9376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:04 smithi044 ceph-mon[32404]: pgmap v9377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:05.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:04 smithi116 ceph-mon[37097]: pgmap v9377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:04 smithi171 ceph-mon[37789]: pgmap v9377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:06 smithi044 ceph-mon[32404]: pgmap v9378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16931.056417 secs 2024-07-26T13:23:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:06 smithi116 ceph-mon[37097]: pgmap v9378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16931.056417 secs 2024-07-26T13:23:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:06 smithi171 ceph-mon[37789]: pgmap v9378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16931.056417 secs 2024-07-26T13:23:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:09 smithi044 ceph-mon[32404]: pgmap v9379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:09.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:09 smithi116 ceph-mon[37097]: pgmap v9379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:09 smithi171 ceph-mon[37789]: pgmap v9379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:11 smithi044 ceph-mon[32404]: pgmap v9380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:11 smithi116 ceph-mon[37097]: pgmap v9380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:11 smithi171 ceph-mon[37789]: pgmap v9380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16936.056566 secs 2024-07-26T13:23:12.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16936.056566 secs 2024-07-26T13:23:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16936.056566 secs 2024-07-26T13:23:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:13 smithi044 ceph-mon[32404]: pgmap v9381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:13 smithi116 ceph-mon[37097]: pgmap v9381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:13 smithi171 ceph-mon[37789]: pgmap v9381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:15 smithi044 ceph-mon[32404]: pgmap v9382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:15 smithi116 ceph-mon[37097]: pgmap v9382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:15 smithi171 ceph-mon[37789]: pgmap v9382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:15.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:15.805+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:23:15.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:15.805+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:23:15.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:15.805+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:17 smithi044 ceph-mon[32404]: pgmap v9383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16941.056719 secs 2024-07-26T13:23:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:17.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:17 smithi116 ceph-mon[37097]: pgmap v9383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16941.056719 secs 2024-07-26T13:23:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:17 smithi171 ceph-mon[37789]: pgmap v9383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16941.056719 secs 2024-07-26T13:23:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:17.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:17.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:17.737+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:23:17.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:17.737+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:23:17.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:17.738+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:23:17.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:17.738+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:23:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:18.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:18.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:19 smithi044 ceph-mon[32404]: pgmap v9384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:19 smithi044 ceph-mon[32404]: pgmap v9385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:19 smithi116 ceph-mon[37097]: pgmap v9384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:19 smithi116 ceph-mon[37097]: pgmap v9385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:19 smithi171 ceph-mon[37789]: pgmap v9384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:19.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:19 smithi171 ceph-mon[37789]: pgmap v9385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:21 smithi044 ceph-mon[32404]: pgmap v9386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:21 smithi116 ceph-mon[37097]: pgmap v9386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:21 smithi171 ceph-mon[37789]: pgmap v9386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16946.056886 secs 2024-07-26T13:23:22.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16946.056886 secs 2024-07-26T13:23:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16946.056886 secs 2024-07-26T13:23:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:23 smithi044 ceph-mon[32404]: pgmap v9387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:23:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:23 smithi116 ceph-mon[37097]: pgmap v9387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:23:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:23 smithi171 ceph-mon[37789]: pgmap v9387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:23:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:25 smithi044 ceph-mon[32404]: pgmap v9388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:25.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:25 smithi116 ceph-mon[37097]: pgmap v9388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:25 smithi171 ceph-mon[37789]: pgmap v9388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:27 smithi044 ceph-mon[32404]: pgmap v9389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16951.057031 secs 2024-07-26T13:23:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:27 smithi116 ceph-mon[37097]: pgmap v9389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16951.057031 secs 2024-07-26T13:23:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:27 smithi171 ceph-mon[37789]: pgmap v9389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:23:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16951.057031 secs 2024-07-26T13:23:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:29 smithi044 ceph-mon[32404]: pgmap v9390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:29 smithi116 ceph-mon[37097]: pgmap v9390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:29 smithi171 ceph-mon[37789]: pgmap v9390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:30.805+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:23:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:30.806+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:23:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:30.806+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:31 smithi116 ceph-mon[37097]: pgmap v9391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:31 smithi171 ceph-mon[37789]: pgmap v9391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:31 smithi044 ceph-mon[32404]: pgmap v9391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16956.057189 secs 2024-07-26T13:23:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16956.057189 secs 2024-07-26T13:23:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16956.057189 secs 2024-07-26T13:23:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:33 smithi171 ceph-mon[37789]: pgmap v9392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:33 smithi044 ceph-mon[32404]: pgmap v9392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:33 smithi116 ceph-mon[37097]: pgmap v9392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:35 smithi171 ceph-mon[37789]: pgmap v9393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:35 smithi044 ceph-mon[32404]: pgmap v9393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:35 smithi116 ceph-mon[37097]: pgmap v9393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:37 smithi171 ceph-mon[37789]: pgmap v9394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:37.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16961.057330 secs 2024-07-26T13:23:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:37 smithi044 ceph-mon[32404]: pgmap v9394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:37.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16961.057330 secs 2024-07-26T13:23:37.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:37 smithi116 ceph-mon[37097]: pgmap v9394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16961.057330 secs 2024-07-26T13:23:39.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:39 smithi116 ceph-mon[37097]: pgmap v9395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:39 smithi171 ceph-mon[37789]: pgmap v9395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:39 smithi044 ceph-mon[32404]: pgmap v9395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:41 smithi171 ceph-mon[37789]: pgmap v9396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:41 smithi044 ceph-mon[32404]: pgmap v9396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:41 smithi116 ceph-mon[37097]: pgmap v9396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16966.057467 secs 2024-07-26T13:23:42.625 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16966.057467 secs 2024-07-26T13:23:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16966.057467 secs 2024-07-26T13:23:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:43 smithi171 ceph-mon[37789]: pgmap v9397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:43 smithi044 ceph-mon[32404]: pgmap v9397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:43.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:43 smithi116 ceph-mon[37097]: pgmap v9397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:44 smithi171 ceph-mon[37789]: pgmap v9398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:44 smithi044 ceph-mon[32404]: pgmap v9398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:44.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:44 smithi116 ceph-mon[37097]: pgmap v9398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:45.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:45.812+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:23:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:45.812+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:45.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:23:45.812+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:23:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:46 smithi044 ceph-mon[32404]: pgmap v9399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:46 smithi116 ceph-mon[37097]: pgmap v9399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:46 smithi171 ceph-mon[37789]: pgmap v9399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16971.057553 secs 2024-07-26T13:23:48.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16971.057553 secs 2024-07-26T13:23:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16971.057553 secs 2024-07-26T13:23:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:48 smithi044 ceph-mon[32404]: pgmap v9400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:48 smithi116 ceph-mon[37097]: pgmap v9400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:48 smithi171 ceph-mon[37789]: pgmap v9400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:50 smithi044 ceph-mon[32404]: pgmap v9401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:50 smithi116 ceph-mon[37097]: pgmap v9401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:50 smithi171 ceph-mon[37789]: pgmap v9401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:23:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16976.057719 secs 2024-07-26T13:23:53.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:52 smithi044 ceph-mon[32404]: pgmap v9402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:53.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16976.057719 secs 2024-07-26T13:23:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:52 smithi116 ceph-mon[37097]: pgmap v9402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16976.057719 secs 2024-07-26T13:23:53.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:52 smithi171 ceph-mon[37789]: pgmap v9402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:54 smithi044 ceph-mon[32404]: pgmap v9403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:54 smithi116 ceph-mon[37097]: pgmap v9403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:54 smithi171 ceph-mon[37789]: pgmap v9403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:23:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:56 smithi044 ceph-mon[32404]: pgmap v9404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:57.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:56 smithi116 ceph-mon[37097]: pgmap v9404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:56 smithi171 ceph-mon[37789]: pgmap v9404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:57.985 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16981.057891 secs 2024-07-26T13:23:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16981.057891 secs 2024-07-26T13:23:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16981.057891 secs 2024-07-26T13:23:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:23:58 smithi044 ceph-mon[32404]: pgmap v9405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:23:58 smithi116 ceph-mon[37097]: pgmap v9405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:23:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:23:58 smithi171 ceph-mon[37789]: pgmap v9405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:00.814+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:24:00.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:00.814+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:00.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:00.814+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:24:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:00 smithi044 ceph-mon[32404]: pgmap v9406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:00 smithi116 ceph-mon[37097]: pgmap v9406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:00 smithi171 ceph-mon[37789]: pgmap v9406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16986.058073 secs 2024-07-26T13:24:03.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:02 smithi044 ceph-mon[32404]: pgmap v9407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16986.058073 secs 2024-07-26T13:24:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:02 smithi116 ceph-mon[37097]: pgmap v9407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16986.058073 secs 2024-07-26T13:24:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:02 smithi171 ceph-mon[37789]: pgmap v9407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:04 smithi044 ceph-mon[32404]: pgmap v9408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:05.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:04 smithi116 ceph-mon[37097]: pgmap v9408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:04 smithi171 ceph-mon[37789]: pgmap v9408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:06 smithi044 ceph-mon[32404]: pgmap v9409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:07.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:06 smithi116 ceph-mon[37097]: pgmap v9409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:06 smithi171 ceph-mon[37789]: pgmap v9409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16991.058231 secs 2024-07-26T13:24:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16991.058231 secs 2024-07-26T13:24:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16991.058231 secs 2024-07-26T13:24:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:08 smithi044 ceph-mon[32404]: pgmap v9410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:09.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:08 smithi116 ceph-mon[37097]: pgmap v9410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:08 smithi171 ceph-mon[37789]: pgmap v9410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:10 smithi044 ceph-mon[32404]: pgmap v9411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:10 smithi116 ceph-mon[37097]: pgmap v9411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:10 smithi171 ceph-mon[37789]: pgmap v9411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 16996.058398 secs 2024-07-26T13:24:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:12 smithi044 ceph-mon[32404]: pgmap v9412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 16996.058398 secs 2024-07-26T13:24:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:12 smithi116 ceph-mon[37097]: pgmap v9412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 16996.058398 secs 2024-07-26T13:24:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:12 smithi171 ceph-mon[37789]: pgmap v9412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:14 smithi044 ceph-mon[32404]: pgmap v9413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:15.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:14 smithi116 ceph-mon[37097]: pgmap v9413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:14 smithi171 ceph-mon[37789]: pgmap v9413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:15.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:15.819+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:24:15.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:15.819+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:15.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:15.819+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:24:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:16 smithi044 ceph-mon[32404]: pgmap v9414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17001.058562 secs 2024-07-26T13:24:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:16 smithi116 ceph-mon[37097]: pgmap v9414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17001.058562 secs 2024-07-26T13:24:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:16 smithi171 ceph-mon[37789]: pgmap v9414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17001.058562 secs 2024-07-26T13:24:18.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:18 smithi044 ceph-mon[32404]: pgmap v9415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:18.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:18.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.966 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.967 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.967 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.992 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:18 smithi171 ceph-mon[37789]: pgmap v9415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:18.993 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:18.993 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.993 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.993 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:18.993 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:18 smithi116 ceph-mon[37097]: pgmap v9415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:19.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:19.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:19.026 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:20 smithi116 ceph-mon[37097]: pgmap v9416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:20 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:20 smithi171 ceph-mon[37789]: pgmap v9416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:20 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:20 smithi044 ceph-mon[32404]: pgmap v9416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:20 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17006.058689 secs 2024-07-26T13:24:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:22 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:22 smithi044 ceph-mon[32404]: pgmap v9417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17006.058689 secs 2024-07-26T13:24:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:22 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:22 smithi116 ceph-mon[37097]: pgmap v9417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17006.058689 secs 2024-07-26T13:24:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:22 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:22 smithi171 ceph-mon[37789]: pgmap v9417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:23 smithi044 ceph-mon[32404]: pgmap v9418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:24:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:23 smithi044 ceph-mon[32404]: pgmap v9419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:23 smithi116 ceph-mon[37097]: pgmap v9418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:24:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:23 smithi116 ceph-mon[37097]: pgmap v9419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:23 smithi171 ceph-mon[37789]: pgmap v9418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:24:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:23 smithi171 ceph-mon[37789]: pgmap v9419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:25 smithi044 ceph-mon[32404]: pgmap v9420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:25 smithi116 ceph-mon[37097]: pgmap v9420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:25 smithi171 ceph-mon[37789]: pgmap v9420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:28.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17011.058857 secs 2024-07-26T13:24:28.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:27 smithi044 ceph-mon[32404]: pgmap v9421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17011.058857 secs 2024-07-26T13:24:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:27 smithi116 ceph-mon[37097]: pgmap v9421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17011.058857 secs 2024-07-26T13:24:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:27 smithi171 ceph-mon[37789]: pgmap v9421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:29 smithi044 ceph-mon[32404]: pgmap v9422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:29 smithi116 ceph-mon[37097]: pgmap v9422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:29 smithi171 ceph-mon[37789]: pgmap v9422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:30.821+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:24:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:30.821+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:30.821+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:24:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:31 smithi044 ceph-mon[32404]: pgmap v9423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:31 smithi116 ceph-mon[37097]: pgmap v9423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:31 smithi171 ceph-mon[37789]: pgmap v9423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17016.059014 secs 2024-07-26T13:24:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17016.059014 secs 2024-07-26T13:24:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17016.059014 secs 2024-07-26T13:24:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:33 smithi044 ceph-mon[32404]: pgmap v9424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:33 smithi116 ceph-mon[37097]: pgmap v9424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:33 smithi171 ceph-mon[37789]: pgmap v9424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:35 smithi044 ceph-mon[32404]: pgmap v9425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:35 smithi116 ceph-mon[37097]: pgmap v9425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:35 smithi171 ceph-mon[37789]: pgmap v9425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:37 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17021.059160 secs 2024-07-26T13:24:38.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:37 smithi044 ceph-mon[32404]: pgmap v9426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:37 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17021.059160 secs 2024-07-26T13:24:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:37 smithi116 ceph-mon[37097]: pgmap v9426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:37 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17021.059160 secs 2024-07-26T13:24:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:37 smithi171 ceph-mon[37789]: pgmap v9426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:39 smithi044 ceph-mon[32404]: pgmap v9427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:39 smithi116 ceph-mon[37097]: pgmap v9427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:39 smithi171 ceph-mon[37789]: pgmap v9427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:24:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:41 smithi044 ceph-mon[32404]: pgmap v9428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:42.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:41 smithi116 ceph-mon[37097]: pgmap v9428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:41 smithi171 ceph-mon[37789]: pgmap v9428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:42.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:42 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17026.059346 secs 2024-07-26T13:24:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:42 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17026.059346 secs 2024-07-26T13:24:43.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:42 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17026.059346 secs 2024-07-26T13:24:44.006 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:43 smithi171 ceph-mon[37789]: pgmap v9429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:43 smithi044 ceph-mon[32404]: pgmap v9429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:43 smithi116 ceph-mon[37097]: pgmap v9429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:24:45.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:45.825+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:24:45.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:45.825+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:45.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:24:45.825+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:24:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:45 smithi044 ceph-mon[32404]: pgmap v9430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:45 smithi116 ceph-mon[37097]: pgmap v9430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:45 smithi171 ceph-mon[37789]: pgmap v9430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17031.059526 secs 2024-07-26T13:24:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:47 smithi044 ceph-mon[32404]: pgmap v9431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:24:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17031.059526 secs 2024-07-26T13:24:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:47 smithi116 ceph-mon[37097]: pgmap v9431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:24:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17031.059526 secs 2024-07-26T13:24:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:47 smithi171 ceph-mon[37789]: pgmap v9431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:24:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:49 smithi044 ceph-mon[32404]: pgmap v9432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:24:50.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:49 smithi116 ceph-mon[37097]: pgmap v9432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:24:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:49 smithi171 ceph-mon[37789]: pgmap v9432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:24:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:51 smithi044 ceph-mon[32404]: pgmap v9433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:24:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17036.059683 secs 2024-07-26T13:24:52.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:51 smithi116 ceph-mon[37097]: pgmap v9433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:24:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17036.059683 secs 2024-07-26T13:24:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:51 smithi171 ceph-mon[37789]: pgmap v9433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:24:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17036.059683 secs 2024-07-26T13:24:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:53 smithi044 ceph-mon[32404]: pgmap v9434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:24:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:53 smithi116 ceph-mon[37097]: pgmap v9434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:24:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:53 smithi171 ceph-mon[37789]: pgmap v9434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:24:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:55 smithi044 ceph-mon[32404]: pgmap v9435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:56.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:55 smithi116 ceph-mon[37097]: pgmap v9435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:55 smithi171 ceph-mon[37789]: pgmap v9435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17041.059853 secs 2024-07-26T13:24:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17041.059853 secs 2024-07-26T13:24:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17041.059853 secs 2024-07-26T13:24:58.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:57 smithi044 ceph-mon[32404]: pgmap v9436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:57 smithi116 ceph-mon[37097]: pgmap v9436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:57 smithi171 ceph-mon[37789]: pgmap v9436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:24:59 smithi044 ceph-mon[32404]: pgmap v9437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:24:59 smithi116 ceph-mon[37097]: pgmap v9437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:24:59 smithi171 ceph-mon[37789]: pgmap v9437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:00.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:00.827+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:25:00.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:00.827+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:25:00.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:00.827+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:01 smithi044 ceph-mon[32404]: pgmap v9438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17046.059971 secs 2024-07-26T13:25:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:01 smithi116 ceph-mon[37097]: pgmap v9438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17046.059971 secs 2024-07-26T13:25:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:01 smithi171 ceph-mon[37789]: pgmap v9438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17046.059971 secs 2024-07-26T13:25:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:03 smithi044 ceph-mon[32404]: pgmap v9439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:04.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:03 smithi116 ceph-mon[37097]: pgmap v9439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:03 smithi171 ceph-mon[37789]: pgmap v9439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:05 smithi044 ceph-mon[32404]: pgmap v9440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:05 smithi116 ceph-mon[37097]: pgmap v9440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:05 smithi171 ceph-mon[37789]: pgmap v9440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17051.060119 secs 2024-07-26T13:25:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17051.060119 secs 2024-07-26T13:25:07.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17051.060119 secs 2024-07-26T13:25:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:07 smithi171 ceph-mon[37789]: pgmap v9441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:07 smithi044 ceph-mon[32404]: pgmap v9441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:07 smithi116 ceph-mon[37097]: pgmap v9441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:09 smithi171 ceph-mon[37789]: pgmap v9442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:09 smithi044 ceph-mon[32404]: pgmap v9442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:09 smithi116 ceph-mon[37097]: pgmap v9442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:12.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:11 smithi116 ceph-mon[37097]: pgmap v9443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:12.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17056.060289 secs 2024-07-26T13:25:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:11 smithi171 ceph-mon[37789]: pgmap v9443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17056.060289 secs 2024-07-26T13:25:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:11 smithi044 ceph-mon[32404]: pgmap v9443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17056.060289 secs 2024-07-26T13:25:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:13 smithi171 ceph-mon[37789]: pgmap v9444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:14.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:13 smithi044 ceph-mon[32404]: pgmap v9444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:14.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:13 smithi116 ceph-mon[37097]: pgmap v9444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:15.832+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:25:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:15.832+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:15.832+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:25:16.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:15 smithi044 ceph-mon[32404]: pgmap v9445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:15 smithi171 ceph-mon[37789]: pgmap v9445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:15 smithi116 ceph-mon[37097]: pgmap v9445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17061.060434 secs 2024-07-26T13:25:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17061.060434 secs 2024-07-26T13:25:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17061.060434 secs 2024-07-26T13:25:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:17 smithi171 ceph-mon[37789]: pgmap v9446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:17 smithi044 ceph-mon[32404]: pgmap v9446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:17 smithi116 ceph-mon[37097]: pgmap v9446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:19 smithi171 ceph-mon[37789]: pgmap v9447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:20.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:19 smithi044 ceph-mon[32404]: pgmap v9447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:19 smithi116 ceph-mon[37097]: pgmap v9447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:25:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:21 smithi171 ceph-mon[37789]: pgmap v9448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17066.060600 secs 2024-07-26T13:25:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:21 smithi044 ceph-mon[32404]: pgmap v9448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17066.060600 secs 2024-07-26T13:25:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:21 smithi116 ceph-mon[37097]: pgmap v9448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17066.060600 secs 2024-07-26T13:25:23.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:22 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:22 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:22 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:23 smithi044 ceph-mon[32404]: pgmap v9449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:23 smithi116 ceph-mon[37097]: pgmap v9449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:23 smithi171 ceph-mon[37789]: pgmap v9449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:25.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:25.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:25.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:24 smithi044 ceph-mon[32404]: pgmap v9450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:25.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:24 smithi044 ceph-mon[32404]: pgmap v9451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:25.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:25.277 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:25.277 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:25.277 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:24 smithi116 ceph-mon[37097]: pgmap v9450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:25.278 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:24 smithi116 ceph-mon[37097]: pgmap v9451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:25.278 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:24 smithi171 ceph-mon[37789]: pgmap v9450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:24 smithi171 ceph-mon[37789]: pgmap v9451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:25.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17071.060744 secs 2024-07-26T13:25:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:26 smithi044 ceph-mon[32404]: pgmap v9452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:27.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17071.060744 secs 2024-07-26T13:25:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:26 smithi116 ceph-mon[37097]: pgmap v9452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17071.060744 secs 2024-07-26T13:25:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:26 smithi171 ceph-mon[37789]: pgmap v9452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:28 smithi044 ceph-mon[32404]: pgmap v9453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:25:29.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:28 smithi116 ceph-mon[37097]: pgmap v9453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:25:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:28 smithi171 ceph-mon[37789]: pgmap v9453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:25:30.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:30.833+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:25:30.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:30.833+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:25:30.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:30.833+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:30 smithi044 ceph-mon[32404]: pgmap v9454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:30 smithi116 ceph-mon[37097]: pgmap v9454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:30 smithi171 ceph-mon[37789]: pgmap v9454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:25:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17076.060919 secs 2024-07-26T13:25:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17076.060919 secs 2024-07-26T13:25:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17076.060919 secs 2024-07-26T13:25:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:32 smithi044 ceph-mon[32404]: pgmap v9455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:25:33.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:32 smithi116 ceph-mon[37097]: pgmap v9455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:25:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:32 smithi171 ceph-mon[37789]: pgmap v9455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:25:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:34 smithi044 ceph-mon[32404]: pgmap v9456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:35.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:34 smithi116 ceph-mon[37097]: pgmap v9456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:34 smithi171 ceph-mon[37789]: pgmap v9456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17081.061068 secs 2024-07-26T13:25:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:36 smithi044 ceph-mon[32404]: pgmap v9457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:37.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17081.061068 secs 2024-07-26T13:25:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:36 smithi116 ceph-mon[37097]: pgmap v9457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17081.061068 secs 2024-07-26T13:25:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:36 smithi171 ceph-mon[37789]: pgmap v9457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:38 smithi044 ceph-mon[32404]: pgmap v9458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:38 smithi116 ceph-mon[37097]: pgmap v9458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:38 smithi171 ceph-mon[37789]: pgmap v9458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:40 smithi044 ceph-mon[32404]: pgmap v9459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:25:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:40 smithi116 ceph-mon[37097]: pgmap v9459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:25:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:40 smithi171 ceph-mon[37789]: pgmap v9459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:25:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17086.061224 secs 2024-07-26T13:25:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17086.061224 secs 2024-07-26T13:25:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17086.061224 secs 2024-07-26T13:25:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:42 smithi044 ceph-mon[32404]: pgmap v9460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:42 smithi116 ceph-mon[37097]: pgmap v9460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:42 smithi171 ceph-mon[37789]: pgmap v9460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:44 smithi044 ceph-mon[32404]: pgmap v9461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:25:45.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:44 smithi116 ceph-mon[37097]: pgmap v9461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:25:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:44 smithi171 ceph-mon[37789]: pgmap v9461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:25:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:45.837+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:25:45.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:45.837+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:25:45.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:25:45.837+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17091.061385 secs 2024-07-26T13:25:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:47 smithi044 ceph-mon[32404]: pgmap v9462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:25:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17091.061385 secs 2024-07-26T13:25:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:47 smithi116 ceph-mon[37097]: pgmap v9462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:25:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17091.061385 secs 2024-07-26T13:25:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:47 smithi171 ceph-mon[37789]: pgmap v9462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:25:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:49 smithi044 ceph-mon[32404]: pgmap v9463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:49.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:49 smithi116 ceph-mon[37097]: pgmap v9463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:49 smithi171 ceph-mon[37789]: pgmap v9463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:51 smithi044 ceph-mon[32404]: pgmap v9464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:51 smithi116 ceph-mon[37097]: pgmap v9464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:51 smithi171 ceph-mon[37789]: pgmap v9464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:25:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17096.061516 secs 2024-07-26T13:25:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17096.061516 secs 2024-07-26T13:25:52.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17096.061516 secs 2024-07-26T13:25:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:53 smithi171 ceph-mon[37789]: pgmap v9465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:53.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:53 smithi044 ceph-mon[32404]: pgmap v9465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:53 smithi116 ceph-mon[37097]: pgmap v9465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:55 smithi171 ceph-mon[37789]: pgmap v9466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:55 smithi044 ceph-mon[32404]: pgmap v9466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:55.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:55 smithi116 ceph-mon[37097]: pgmap v9466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:57 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17101.061694 secs 2024-07-26T13:25:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:57 smithi171 ceph-mon[37789]: pgmap v9467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:57 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17101.061694 secs 2024-07-26T13:25:57.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:57 smithi044 ceph-mon[32404]: pgmap v9467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:57.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:57 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17101.061694 secs 2024-07-26T13:25:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:57 smithi116 ceph-mon[37097]: pgmap v9467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:25:59 smithi171 ceph-mon[37789]: pgmap v9468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:25:59 smithi044 ceph-mon[32404]: pgmap v9468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:25:59.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:25:59 smithi116 ceph-mon[37097]: pgmap v9468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:00.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:00.839+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:26:00.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:00.839+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:26:00.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:00.839+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:01 smithi044 ceph-mon[32404]: pgmap v9469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:01 smithi171 ceph-mon[37789]: pgmap v9469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:01 smithi116 ceph-mon[37097]: pgmap v9469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:02 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17106.061846 secs 2024-07-26T13:26:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:02 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17106.061846 secs 2024-07-26T13:26:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:02 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17106.061846 secs 2024-07-26T13:26:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:03 smithi044 ceph-mon[32404]: pgmap v9470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:03 smithi171 ceph-mon[37789]: pgmap v9470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:03 smithi116 ceph-mon[37097]: pgmap v9470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:05.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:05 smithi171 ceph-mon[37789]: pgmap v9471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:05 smithi044 ceph-mon[32404]: pgmap v9471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:05 smithi116 ceph-mon[37097]: pgmap v9471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17111.062005 secs 2024-07-26T13:26:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17111.062005 secs 2024-07-26T13:26:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17111.062005 secs 2024-07-26T13:26:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:07 smithi171 ceph-mon[37789]: pgmap v9472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:07 smithi044 ceph-mon[32404]: pgmap v9472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:07 smithi116 ceph-mon[37097]: pgmap v9472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:09 smithi044 ceph-mon[32404]: pgmap v9473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:09.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:09 smithi116 ceph-mon[37097]: pgmap v9473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:09 smithi171 ceph-mon[37789]: pgmap v9473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:11 smithi044 ceph-mon[32404]: pgmap v9474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17116.062166 secs 2024-07-26T13:26:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:11 smithi116 ceph-mon[37097]: pgmap v9474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17116.062166 secs 2024-07-26T13:26:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:11 smithi171 ceph-mon[37789]: pgmap v9474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17116.062166 secs 2024-07-26T13:26:13.388 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:13 smithi171 ceph-mon[37789]: pgmap v9475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:13 smithi044 ceph-mon[32404]: pgmap v9475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:13.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:13 smithi116 ceph-mon[37097]: pgmap v9475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:14.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:26:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:26:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fcf580318f0 tx=0x7fcf58069040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:26:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fcf580318f0 tx=0x7fcf58069040 comp rx=0 tx=0).stop 2024-07-26T13:26:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T13:26:14.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:26:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:26:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.772+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:26:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.773+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:26:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:14.776+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58050000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:26:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:15 smithi044 ceph-mon[32404]: pgmap v9476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:15 smithi116 ceph-mon[37097]: pgmap v9476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:15 smithi171 ceph-mon[37789]: pgmap v9476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:15.840+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:26:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:15.840+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:26:15.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:15.840+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17121.062291 secs 2024-07-26T13:26:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17121.062291 secs 2024-07-26T13:26:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17121.062291 secs 2024-07-26T13:26:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:17 smithi044 ceph-mon[32404]: pgmap v9477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:17 smithi116 ceph-mon[37097]: pgmap v9477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:17 smithi171 ceph-mon[37789]: pgmap v9477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:17.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:17.770+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:26:17.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:17.770+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:26:17.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:17.771+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:26:17.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:17.771+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:26:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:19 smithi044 ceph-mon[32404]: pgmap v9478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:19 smithi116 ceph-mon[37097]: pgmap v9478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:19 smithi171 ceph-mon[37789]: pgmap v9478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:21 smithi044 ceph-mon[32404]: pgmap v9479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:21.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17126.062450 secs 2024-07-26T13:26:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:21 smithi116 ceph-mon[37097]: pgmap v9479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:21.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17126.062450 secs 2024-07-26T13:26:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:21 smithi171 ceph-mon[37789]: pgmap v9479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17126.062450 secs 2024-07-26T13:26:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:23 smithi044 ceph-mon[32404]: pgmap v9480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:23 smithi116 ceph-mon[37097]: pgmap v9480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:23 smithi171 ceph-mon[37789]: pgmap v9480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:25 smithi044 ceph-mon[32404]: pgmap v9481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:25.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:25.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:25 smithi116 ceph-mon[37097]: pgmap v9481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:25.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:25 smithi171 ceph-mon[37789]: pgmap v9481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:26 smithi044 ceph-mon[32404]: pgmap v9482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:26 smithi044 ceph-mon[32404]: pgmap v9483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17131.062666 secs 2024-07-26T13:26:26.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:26 smithi116 ceph-mon[37097]: pgmap v9482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:26 smithi116 ceph-mon[37097]: pgmap v9483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17131.062666 secs 2024-07-26T13:26:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:26 smithi171 ceph-mon[37789]: pgmap v9482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:26 smithi171 ceph-mon[37789]: pgmap v9483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17131.062666 secs 2024-07-26T13:26:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:28 smithi044 ceph-mon[32404]: pgmap v9484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:28 smithi116 ceph-mon[37097]: pgmap v9484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:28 smithi171 ceph-mon[37789]: pgmap v9484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:30.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:30 smithi044 ceph-mon[32404]: pgmap v9485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:30 smithi116 ceph-mon[37097]: pgmap v9485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:30 smithi171 ceph-mon[37789]: pgmap v9485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:30.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:30.841+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:26:30.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:30.841+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:26:30.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:30.841+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17136.062850 secs 2024-07-26T13:26:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17136.062850 secs 2024-07-26T13:26:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17136.062850 secs 2024-07-26T13:26:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:32 smithi044 ceph-mon[32404]: pgmap v9486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:32 smithi116 ceph-mon[37097]: pgmap v9486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:32 smithi171 ceph-mon[37789]: pgmap v9486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:34 smithi044 ceph-mon[32404]: pgmap v9487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:34 smithi116 ceph-mon[37097]: pgmap v9487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:34 smithi171 ceph-mon[37789]: pgmap v9487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:26:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:36 smithi044 ceph-mon[32404]: pgmap v9488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:26:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17141.062947 secs 2024-07-26T13:26:36.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:36 smithi116 ceph-mon[37097]: pgmap v9488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:26:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17141.062947 secs 2024-07-26T13:26:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:36 smithi171 ceph-mon[37789]: pgmap v9488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:26:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17141.062947 secs 2024-07-26T13:26:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:38 smithi044 ceph-mon[32404]: pgmap v9489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:38.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:38 smithi116 ceph-mon[37097]: pgmap v9489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:38 smithi171 ceph-mon[37789]: pgmap v9489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:40 smithi044 ceph-mon[32404]: pgmap v9490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:40.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:40 smithi116 ceph-mon[37097]: pgmap v9490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:40 smithi171 ceph-mon[37789]: pgmap v9490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17146.063111 secs 2024-07-26T13:26:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17146.063111 secs 2024-07-26T13:26:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17146.063111 secs 2024-07-26T13:26:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:42 smithi044 ceph-mon[32404]: pgmap v9491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:42.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:42 smithi116 ceph-mon[37097]: pgmap v9491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:42 smithi171 ceph-mon[37789]: pgmap v9491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:44 smithi044 ceph-mon[32404]: pgmap v9492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:44 smithi116 ceph-mon[37097]: pgmap v9492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:44 smithi171 ceph-mon[37789]: pgmap v9492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:45.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:45.846+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:26:45.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:45.846+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:26:45.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:26:45.846+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:46 smithi044 ceph-mon[32404]: pgmap v9493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17151.063267 secs 2024-07-26T13:26:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:46 smithi116 ceph-mon[37097]: pgmap v9493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17151.063267 secs 2024-07-26T13:26:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:46 smithi171 ceph-mon[37789]: pgmap v9493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17151.063267 secs 2024-07-26T13:26:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:48 smithi044 ceph-mon[32404]: pgmap v9494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:48 smithi116 ceph-mon[37097]: pgmap v9494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:48 smithi171 ceph-mon[37789]: pgmap v9494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:50 smithi044 ceph-mon[32404]: pgmap v9495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:50 smithi116 ceph-mon[37097]: pgmap v9495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:50 smithi171 ceph-mon[37789]: pgmap v9495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17156.063419 secs 2024-07-26T13:26:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17156.063419 secs 2024-07-26T13:26:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17156.063419 secs 2024-07-26T13:26:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:52 smithi044 ceph-mon[32404]: pgmap v9496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:52 smithi116 ceph-mon[37097]: pgmap v9496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:52 smithi171 ceph-mon[37789]: pgmap v9496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:54.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:54 smithi171 ceph-mon[37789]: pgmap v9497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:54 smithi044 ceph-mon[32404]: pgmap v9497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:54 smithi116 ceph-mon[37097]: pgmap v9497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:56 smithi171 ceph-mon[37789]: pgmap v9498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17161.063516 secs 2024-07-26T13:26:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:56 smithi044 ceph-mon[32404]: pgmap v9498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17161.063516 secs 2024-07-26T13:26:56.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:56 smithi116 ceph-mon[37097]: pgmap v9498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17161.063516 secs 2024-07-26T13:26:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:26:58 smithi171 ceph-mon[37789]: pgmap v9499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:26:58 smithi044 ceph-mon[32404]: pgmap v9499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:26:58.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:26:58 smithi116 ceph-mon[37097]: pgmap v9499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:00 smithi171 ceph-mon[37789]: pgmap v9500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:00 smithi044 ceph-mon[32404]: pgmap v9500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:00 smithi116 ceph-mon[37097]: pgmap v9500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:00.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:00.847+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:27:00.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:00.847+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:27:00.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:00.847+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17166.063682 secs 2024-07-26T13:27:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17166.063682 secs 2024-07-26T13:27:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17166.063682 secs 2024-07-26T13:27:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:02 smithi171 ceph-mon[37789]: pgmap v9501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:02 smithi044 ceph-mon[32404]: pgmap v9501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:02 smithi116 ceph-mon[37097]: pgmap v9501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:04 smithi171 ceph-mon[37789]: pgmap v9502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:04 smithi044 ceph-mon[32404]: pgmap v9502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:04 smithi116 ceph-mon[37097]: pgmap v9502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:06 smithi171 ceph-mon[37789]: pgmap v9503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17171.063809 secs 2024-07-26T13:27:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:06 smithi044 ceph-mon[32404]: pgmap v9503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17171.063809 secs 2024-07-26T13:27:06.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:06 smithi116 ceph-mon[37097]: pgmap v9503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17171.063809 secs 2024-07-26T13:27:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:08 smithi171 ceph-mon[37789]: pgmap v9504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:08 smithi044 ceph-mon[32404]: pgmap v9504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:08 smithi116 ceph-mon[37097]: pgmap v9504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:10 smithi171 ceph-mon[37789]: pgmap v9505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:10 smithi044 ceph-mon[32404]: pgmap v9505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:10 smithi116 ceph-mon[37097]: pgmap v9505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17176.063946 secs 2024-07-26T13:27:11.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17176.063946 secs 2024-07-26T13:27:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17176.063946 secs 2024-07-26T13:27:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:12 smithi044 ceph-mon[32404]: pgmap v9506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:12 smithi116 ceph-mon[37097]: pgmap v9506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:12 smithi171 ceph-mon[37789]: pgmap v9506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:14.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:14 smithi171 ceph-mon[37789]: pgmap v9507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:14 smithi044 ceph-mon[32404]: pgmap v9507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:14 smithi116 ceph-mon[37097]: pgmap v9507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:15.848+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:27:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:15.848+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:27:15.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:15.849+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:16 smithi044 ceph-mon[32404]: pgmap v9508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17181.064080 secs 2024-07-26T13:27:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:16 smithi116 ceph-mon[37097]: pgmap v9508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17181.064080 secs 2024-07-26T13:27:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:16 smithi171 ceph-mon[37789]: pgmap v9508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17181.064080 secs 2024-07-26T13:27:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:18 smithi044 ceph-mon[32404]: pgmap v9509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:18 smithi116 ceph-mon[37097]: pgmap v9509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:18 smithi171 ceph-mon[37789]: pgmap v9509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:20.739 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:20 smithi044 ceph-mon[32404]: pgmap v9510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:20 smithi116 ceph-mon[37097]: pgmap v9510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:20 smithi171 ceph-mon[37789]: pgmap v9510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17186.064236 secs 2024-07-26T13:27:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17186.064236 secs 2024-07-26T13:27:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17186.064236 secs 2024-07-26T13:27:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:22 smithi044 ceph-mon[32404]: pgmap v9511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:22 smithi116 ceph-mon[37097]: pgmap v9511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:22 smithi171 ceph-mon[37789]: pgmap v9511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:23 smithi044 ceph-mon[32404]: pgmap v9512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:23.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:23 smithi116 ceph-mon[37097]: pgmap v9512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:23 smithi171 ceph-mon[37789]: pgmap v9512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:26.468 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:26 smithi044 ceph-mon[32404]: pgmap v9513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:26.469 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:26.469 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17191.064375 secs 2024-07-26T13:27:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:26 smithi116 ceph-mon[37097]: pgmap v9513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17191.064375 secs 2024-07-26T13:27:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:26 smithi171 ceph-mon[37789]: pgmap v9513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17191.064375 secs 2024-07-26T13:27:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:27 smithi044 ceph-mon[32404]: pgmap v9514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:27:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:27 smithi044 ceph-mon[32404]: pgmap v9515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:27:27.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:27 smithi116 ceph-mon[37097]: pgmap v9514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:27:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:27 smithi116 ceph-mon[37097]: pgmap v9515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:27:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:27 smithi171 ceph-mon[37789]: pgmap v9514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:27:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:27 smithi171 ceph-mon[37789]: pgmap v9515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:27:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:28 smithi044 ceph-mon[32404]: pgmap v9516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:28.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:28 smithi116 ceph-mon[37097]: pgmap v9516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:28 smithi171 ceph-mon[37789]: pgmap v9516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:30 smithi044 ceph-mon[32404]: pgmap v9517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:30.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:30 smithi116 ceph-mon[37097]: pgmap v9517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:30 smithi171 ceph-mon[37789]: pgmap v9517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:30.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:30.850+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:27:30.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:30.850+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:27:30.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:30.850+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17196.064521 secs 2024-07-26T13:27:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17196.064521 secs 2024-07-26T13:27:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17196.064521 secs 2024-07-26T13:27:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:32 smithi044 ceph-mon[32404]: pgmap v9518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:32 smithi116 ceph-mon[37097]: pgmap v9518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:32 smithi171 ceph-mon[37789]: pgmap v9518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:34 smithi044 ceph-mon[32404]: pgmap v9519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:34 smithi116 ceph-mon[37097]: pgmap v9519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:34.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:34 smithi171 ceph-mon[37789]: pgmap v9519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17201.064686 secs 2024-07-26T13:27:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17201.064686 secs 2024-07-26T13:27:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17201.064686 secs 2024-07-26T13:27:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:36 smithi171 ceph-mon[37789]: pgmap v9520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:36 smithi044 ceph-mon[32404]: pgmap v9520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:36 smithi116 ceph-mon[37097]: pgmap v9520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:38 smithi044 ceph-mon[32404]: pgmap v9521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:38 smithi116 ceph-mon[37097]: pgmap v9521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:38 smithi171 ceph-mon[37789]: pgmap v9521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:40.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:40 smithi044 ceph-mon[32404]: pgmap v9522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:40.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:40 smithi116 ceph-mon[37097]: pgmap v9522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:40 smithi171 ceph-mon[37789]: pgmap v9522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17206.064824 secs 2024-07-26T13:27:41.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17206.064824 secs 2024-07-26T13:27:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17206.064824 secs 2024-07-26T13:27:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:42 smithi044 ceph-mon[32404]: pgmap v9523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:42.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:42 smithi116 ceph-mon[37097]: pgmap v9523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:42 smithi171 ceph-mon[37789]: pgmap v9523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:44 smithi044 ceph-mon[32404]: pgmap v9524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:44 smithi116 ceph-mon[37097]: pgmap v9524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:44 smithi171 ceph-mon[37789]: pgmap v9524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:45.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:45.856+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:27:45.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:45.856+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:27:45.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:27:45.856+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17211.064955 secs 2024-07-26T13:27:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17211.064955 secs 2024-07-26T13:27:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17211.064955 secs 2024-07-26T13:27:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:46 smithi171 ceph-mon[37789]: pgmap v9525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:46 smithi044 ceph-mon[32404]: pgmap v9525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:46 smithi116 ceph-mon[37097]: pgmap v9525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:48 smithi044 ceph-mon[32404]: pgmap v9526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:48.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:48 smithi116 ceph-mon[37097]: pgmap v9526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:48 smithi171 ceph-mon[37789]: pgmap v9526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:50.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:50 smithi044 ceph-mon[32404]: pgmap v9527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:50 smithi116 ceph-mon[37097]: pgmap v9527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:50 smithi171 ceph-mon[37789]: pgmap v9527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17216.065051 secs 2024-07-26T13:27:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17216.065051 secs 2024-07-26T13:27:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17216.065051 secs 2024-07-26T13:27:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:52 smithi044 ceph-mon[32404]: pgmap v9528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:27:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:52 smithi116 ceph-mon[37097]: pgmap v9528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:27:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:52 smithi171 ceph-mon[37789]: pgmap v9528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:27:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:54 smithi044 ceph-mon[32404]: pgmap v9529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:54 smithi116 ceph-mon[37097]: pgmap v9529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:54 smithi171 ceph-mon[37789]: pgmap v9529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:27:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17221.065210 secs 2024-07-26T13:27:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17221.065210 secs 2024-07-26T13:27:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17221.065210 secs 2024-07-26T13:27:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:56 smithi171 ceph-mon[37789]: pgmap v9530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:56 smithi044 ceph-mon[32404]: pgmap v9530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:56 smithi116 ceph-mon[37097]: pgmap v9530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:58.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:27:58 smithi044 ceph-mon[32404]: pgmap v9531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:27:58 smithi116 ceph-mon[37097]: pgmap v9531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:27:58 smithi171 ceph-mon[37789]: pgmap v9531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:00 smithi044 ceph-mon[32404]: pgmap v9532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:00 smithi116 ceph-mon[37097]: pgmap v9532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:00 smithi171 ceph-mon[37789]: pgmap v9532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:00.856+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:28:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:00.857+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:00.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:00.857+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:28:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17226.065360 secs 2024-07-26T13:28:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17226.065360 secs 2024-07-26T13:28:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17226.065360 secs 2024-07-26T13:28:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:02 smithi044 ceph-mon[32404]: pgmap v9533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:02 smithi116 ceph-mon[37097]: pgmap v9533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:02 smithi171 ceph-mon[37789]: pgmap v9533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:04 smithi044 ceph-mon[32404]: pgmap v9534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:04.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:04 smithi116 ceph-mon[37097]: pgmap v9534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:04 smithi171 ceph-mon[37789]: pgmap v9534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17231.065502 secs 2024-07-26T13:28:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17231.065502 secs 2024-07-26T13:28:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17231.065502 secs 2024-07-26T13:28:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:06 smithi171 ceph-mon[37789]: pgmap v9535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:06 smithi044 ceph-mon[32404]: pgmap v9535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:06 smithi116 ceph-mon[37097]: pgmap v9535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:08 smithi044 ceph-mon[32404]: pgmap v9536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:08 smithi116 ceph-mon[37097]: pgmap v9536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:08 smithi171 ceph-mon[37789]: pgmap v9536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:10 smithi044 ceph-mon[32404]: pgmap v9537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:10 smithi116 ceph-mon[37097]: pgmap v9537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:10 smithi171 ceph-mon[37789]: pgmap v9537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17236.065672 secs 2024-07-26T13:28:11.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17236.065672 secs 2024-07-26T13:28:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17236.065672 secs 2024-07-26T13:28:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:12 smithi044 ceph-mon[32404]: pgmap v9538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:12 smithi116 ceph-mon[37097]: pgmap v9538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:12 smithi171 ceph-mon[37789]: pgmap v9538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:14.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:14 smithi044 ceph-mon[32404]: pgmap v9539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:14 smithi116 ceph-mon[37097]: pgmap v9539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:14 smithi171 ceph-mon[37789]: pgmap v9539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:15.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:15.862+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:28:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:15.862+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:28:15.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:15.862+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17241.065832 secs 2024-07-26T13:28:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17241.065832 secs 2024-07-26T13:28:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17241.065832 secs 2024-07-26T13:28:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:16 smithi171 ceph-mon[37789]: pgmap v9540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:16 smithi044 ceph-mon[32404]: pgmap v9540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:16 smithi116 ceph-mon[37097]: pgmap v9540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:18.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:18 smithi044 ceph-mon[32404]: pgmap v9541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:18 smithi116 ceph-mon[37097]: pgmap v9541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:18 smithi171 ceph-mon[37789]: pgmap v9541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:20 smithi044 ceph-mon[32404]: pgmap v9542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:20.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:20 smithi116 ceph-mon[37097]: pgmap v9542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:20 smithi171 ceph-mon[37789]: pgmap v9542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17246.066000 secs 2024-07-26T13:28:21.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17246.066000 secs 2024-07-26T13:28:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17246.066000 secs 2024-07-26T13:28:22.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:22 smithi044 ceph-mon[32404]: pgmap v9543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:28:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:22 smithi116 ceph-mon[37097]: pgmap v9543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:28:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:22 smithi171 ceph-mon[37789]: pgmap v9543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:28:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:24 smithi044 ceph-mon[32404]: pgmap v9544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:28:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:24 smithi116 ceph-mon[37097]: pgmap v9544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:28:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:24 smithi171 ceph-mon[37789]: pgmap v9544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:28:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17251.066165 secs 2024-07-26T13:28:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17251.066165 secs 2024-07-26T13:28:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17251.066165 secs 2024-07-26T13:28:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:26 smithi171 ceph-mon[37789]: pgmap v9545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:28:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:26 smithi044 ceph-mon[32404]: pgmap v9545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:28:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:27.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:26 smithi116 ceph-mon[37097]: pgmap v9545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:28:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:28:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:28:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:28:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:28 smithi171 ceph-mon[37789]: pgmap v9546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:28 smithi171 ceph-mon[37789]: pgmap v9547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:28:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:28 smithi044 ceph-mon[32404]: pgmap v9546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:28 smithi044 ceph-mon[32404]: pgmap v9547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:28:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:28 smithi116 ceph-mon[37097]: pgmap v9546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:28 smithi116 ceph-mon[37097]: pgmap v9547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:28:30.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:30.863+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:28:30.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:30.863+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:28:30.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:30.863+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:30 smithi171 ceph-mon[37789]: pgmap v9548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:31.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17256.066336 secs 2024-07-26T13:28:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:30 smithi044 ceph-mon[32404]: pgmap v9548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:31.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17256.066336 secs 2024-07-26T13:28:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:30 smithi116 ceph-mon[37097]: pgmap v9548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17256.066336 secs 2024-07-26T13:28:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:32 smithi171 ceph-mon[37789]: pgmap v9549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:28:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:32 smithi044 ceph-mon[32404]: pgmap v9549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:28:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:32 smithi116 ceph-mon[37097]: pgmap v9549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:28:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:34 smithi044 ceph-mon[32404]: pgmap v9550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:34 smithi116 ceph-mon[37097]: pgmap v9550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:34 smithi171 ceph-mon[37789]: pgmap v9550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17261.066477 secs 2024-07-26T13:28:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17261.066477 secs 2024-07-26T13:28:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17261.066477 secs 2024-07-26T13:28:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:36 smithi044 ceph-mon[32404]: pgmap v9551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:37.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:36 smithi116 ceph-mon[37097]: pgmap v9551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:36 smithi171 ceph-mon[37789]: pgmap v9551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:28:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:38 smithi044 ceph-mon[32404]: pgmap v9552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:38 smithi116 ceph-mon[37097]: pgmap v9552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:38 smithi171 ceph-mon[37789]: pgmap v9552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:40 smithi044 ceph-mon[32404]: pgmap v9553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:41.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17266.066687 secs 2024-07-26T13:28:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:40 smithi116 ceph-mon[37097]: pgmap v9553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17266.066687 secs 2024-07-26T13:28:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:40 smithi171 ceph-mon[37789]: pgmap v9553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:41.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17266.066687 secs 2024-07-26T13:28:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:42 smithi044 ceph-mon[32404]: pgmap v9554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:42 smithi116 ceph-mon[37097]: pgmap v9554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:42 smithi171 ceph-mon[37789]: pgmap v9554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:45.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:44 smithi171 ceph-mon[37789]: pgmap v9555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:44 smithi044 ceph-mon[32404]: pgmap v9555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:44 smithi116 ceph-mon[37097]: pgmap v9555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:45.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:45.869+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:28:45.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:45.869+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:28:45.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:28:45.869+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17271.066833 secs 2024-07-26T13:28:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17271.066833 secs 2024-07-26T13:28:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17271.066833 secs 2024-07-26T13:28:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:46 smithi044 ceph-mon[32404]: pgmap v9556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:46 smithi116 ceph-mon[37097]: pgmap v9556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:46 smithi171 ceph-mon[37789]: pgmap v9556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:48 smithi044 ceph-mon[32404]: pgmap v9557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:48 smithi116 ceph-mon[37097]: pgmap v9557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:48 smithi171 ceph-mon[37789]: pgmap v9557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:50 smithi044 ceph-mon[32404]: pgmap v9558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17276.066931 secs 2024-07-26T13:28:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:50 smithi116 ceph-mon[37097]: pgmap v9558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17276.066931 secs 2024-07-26T13:28:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:50 smithi171 ceph-mon[37789]: pgmap v9558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:28:51.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17276.066931 secs 2024-07-26T13:28:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:52 smithi044 ceph-mon[32404]: pgmap v9559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:53.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:52 smithi116 ceph-mon[37097]: pgmap v9559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:52 smithi171 ceph-mon[37789]: pgmap v9559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:54 smithi044 ceph-mon[32404]: pgmap v9560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:54 smithi116 ceph-mon[37097]: pgmap v9560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:54 smithi171 ceph-mon[37789]: pgmap v9560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:56.169 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17281.067092 secs 2024-07-26T13:28:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17281.067092 secs 2024-07-26T13:28:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17281.067092 secs 2024-07-26T13:28:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:56 smithi044 ceph-mon[32404]: pgmap v9561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:56 smithi116 ceph-mon[37097]: pgmap v9561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:56 smithi171 ceph-mon[37789]: pgmap v9561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:28:58 smithi044 ceph-mon[32404]: pgmap v9562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:28:58 smithi116 ceph-mon[37097]: pgmap v9562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:28:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:28:58 smithi171 ceph-mon[37789]: pgmap v9562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:00.869+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:29:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:00.869+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:29:00.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:00.869+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:00 smithi044 ceph-mon[32404]: pgmap v9563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17286.067257 secs 2024-07-26T13:29:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:00 smithi116 ceph-mon[37097]: pgmap v9563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17286.067257 secs 2024-07-26T13:29:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:00 smithi171 ceph-mon[37789]: pgmap v9563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17286.067257 secs 2024-07-26T13:29:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:02 smithi044 ceph-mon[32404]: pgmap v9564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:02 smithi116 ceph-mon[37097]: pgmap v9564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:02 smithi171 ceph-mon[37789]: pgmap v9564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:04 smithi044 ceph-mon[32404]: pgmap v9565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:04 smithi116 ceph-mon[37097]: pgmap v9565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:04 smithi171 ceph-mon[37789]: pgmap v9565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17291.067402 secs 2024-07-26T13:29:06.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17291.067402 secs 2024-07-26T13:29:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17291.067402 secs 2024-07-26T13:29:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:06 smithi044 ceph-mon[32404]: pgmap v9566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:06 smithi116 ceph-mon[37097]: pgmap v9566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:06 smithi171 ceph-mon[37789]: pgmap v9566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:08 smithi044 ceph-mon[32404]: pgmap v9567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:08 smithi116 ceph-mon[37097]: pgmap v9567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:08 smithi171 ceph-mon[37789]: pgmap v9567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:10 smithi044 ceph-mon[32404]: pgmap v9568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17296.067547 secs 2024-07-26T13:29:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:10 smithi116 ceph-mon[37097]: pgmap v9568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17296.067547 secs 2024-07-26T13:29:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:10 smithi171 ceph-mon[37789]: pgmap v9568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17296.067547 secs 2024-07-26T13:29:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:13 smithi044 ceph-mon[32404]: pgmap v9569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:13 smithi116 ceph-mon[37097]: pgmap v9569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:13 smithi171 ceph-mon[37789]: pgmap v9569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:15.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:15 smithi044 ceph-mon[32404]: pgmap v9570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:15 smithi116 ceph-mon[37097]: pgmap v9570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:15 smithi171 ceph-mon[37789]: pgmap v9570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:15.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:15.874+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:29:15.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:15.874+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:29:15.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:15.874+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17301.067705 secs 2024-07-26T13:29:16.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17301.067705 secs 2024-07-26T13:29:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17301.067705 secs 2024-07-26T13:29:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:17 smithi116 ceph-mon[37097]: pgmap v9571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:17 smithi171 ceph-mon[37789]: pgmap v9571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:17 smithi044 ceph-mon[32404]: pgmap v9571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:17.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:17.804+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:29:17.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:17.804+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:29:17.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:17.804+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:29:17.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:17.805+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:29:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:19 smithi171 ceph-mon[37789]: pgmap v9572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:19.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:19 smithi044 ceph-mon[32404]: pgmap v9572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:19 smithi116 ceph-mon[37097]: pgmap v9572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:21 smithi171 ceph-mon[37789]: pgmap v9573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17306.067876 secs 2024-07-26T13:29:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:21 smithi044 ceph-mon[32404]: pgmap v9573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17306.067876 secs 2024-07-26T13:29:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:21 smithi116 ceph-mon[37097]: pgmap v9573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17306.067876 secs 2024-07-26T13:29:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:23 smithi171 ceph-mon[37789]: pgmap v9574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:23 smithi044 ceph-mon[32404]: pgmap v9574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:23 smithi116 ceph-mon[37097]: pgmap v9574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:25 smithi171 ceph-mon[37789]: pgmap v9575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:25 smithi044 ceph-mon[32404]: pgmap v9575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:25 smithi116 ceph-mon[37097]: pgmap v9575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17311.068035 secs 2024-07-26T13:29:26.421 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17311.068035 secs 2024-07-26T13:29:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17311.068035 secs 2024-07-26T13:29:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:27 smithi171 ceph-mon[37789]: pgmap v9576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:29:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:27 smithi044 ceph-mon[32404]: pgmap v9576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:29:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:27 smithi116 ceph-mon[37097]: pgmap v9576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:29:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:29 smithi171 ceph-mon[37789]: pgmap v9577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:29.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:29 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:29:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:29 smithi044 ceph-mon[32404]: pgmap v9577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:29.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:29 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:29:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:29 smithi116 ceph-mon[37097]: pgmap v9577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:29 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:29:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:30 smithi171 ceph-mon[37789]: pgmap v9578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:30 smithi044 ceph-mon[32404]: pgmap v9578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:30.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:30 smithi116 ceph-mon[37097]: pgmap v9578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:30.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:30.875+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:29:30.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:30.875+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:29:30.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:30.875+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17316.068186 secs 2024-07-26T13:29:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17316.068186 secs 2024-07-26T13:29:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17316.068186 secs 2024-07-26T13:29:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:32 smithi044 ceph-mon[32404]: pgmap v9579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:32.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:32 smithi116 ceph-mon[37097]: pgmap v9579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:32 smithi171 ceph-mon[37789]: pgmap v9579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:34 smithi044 ceph-mon[32404]: pgmap v9580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:34 smithi116 ceph-mon[37097]: pgmap v9580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:34 smithi171 ceph-mon[37789]: pgmap v9580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:36 smithi044 ceph-mon[32404]: pgmap v9581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17321.068346 secs 2024-07-26T13:29:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:36 smithi116 ceph-mon[37097]: pgmap v9581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17321.068346 secs 2024-07-26T13:29:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:36 smithi171 ceph-mon[37789]: pgmap v9581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17321.068346 secs 2024-07-26T13:29:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:38 smithi044 ceph-mon[32404]: pgmap v9582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:38 smithi116 ceph-mon[37097]: pgmap v9582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:38 smithi171 ceph-mon[37789]: pgmap v9582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:40 smithi044 ceph-mon[32404]: pgmap v9583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:40 smithi116 ceph-mon[37097]: pgmap v9583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:40 smithi171 ceph-mon[37789]: pgmap v9583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17326.068494 secs 2024-07-26T13:29:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17326.068494 secs 2024-07-26T13:29:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17326.068494 secs 2024-07-26T13:29:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:42 smithi044 ceph-mon[32404]: pgmap v9584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:29:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:42 smithi116 ceph-mon[37097]: pgmap v9584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:29:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:42 smithi171 ceph-mon[37789]: pgmap v9584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:29:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:44 smithi044 ceph-mon[32404]: pgmap v9585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:29:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:44 smithi116 ceph-mon[37097]: pgmap v9585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:29:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:44 smithi171 ceph-mon[37789]: pgmap v9585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:29:45.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:45.881+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:29:45.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:45.881+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:29:45.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:29:45.881+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:46.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:46 smithi171 ceph-mon[37789]: pgmap v9586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:29:46.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17331.068588 secs 2024-07-26T13:29:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:46 smithi044 ceph-mon[32404]: pgmap v9586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:29:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17331.068588 secs 2024-07-26T13:29:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:46 smithi116 ceph-mon[37097]: pgmap v9586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:29:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17331.068588 secs 2024-07-26T13:29:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:48 smithi044 ceph-mon[32404]: pgmap v9587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:29:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:48 smithi116 ceph-mon[37097]: pgmap v9587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:29:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:48 smithi171 ceph-mon[37789]: pgmap v9587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:29:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:50 smithi044 ceph-mon[32404]: pgmap v9588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:50.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:50 smithi116 ceph-mon[37097]: pgmap v9588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:50 smithi171 ceph-mon[37789]: pgmap v9588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17336.068736 secs 2024-07-26T13:29:51.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17336.068736 secs 2024-07-26T13:29:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17336.068736 secs 2024-07-26T13:29:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:52 smithi044 ceph-mon[32404]: pgmap v9589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:52 smithi116 ceph-mon[37097]: pgmap v9589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:52 smithi171 ceph-mon[37789]: pgmap v9589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:54 smithi044 ceph-mon[32404]: pgmap v9590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:54 smithi116 ceph-mon[37097]: pgmap v9590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:54 smithi171 ceph-mon[37789]: pgmap v9590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:56 smithi044 ceph-mon[32404]: pgmap v9591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17341.068900 secs 2024-07-26T13:29:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:56 smithi116 ceph-mon[37097]: pgmap v9591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17341.068900 secs 2024-07-26T13:29:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:56 smithi171 ceph-mon[37789]: pgmap v9591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:29:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17341.068900 secs 2024-07-26T13:29:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:29:58 smithi044 ceph-mon[32404]: pgmap v9592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:58.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:29:58 smithi116 ceph-mon[37097]: pgmap v9592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:29:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:29:58 smithi171 ceph-mon[37789]: pgmap v9592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:00 smithi044 ceph-mon[32404]: pgmap v9593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:00.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:00 smithi116 ceph-mon[37097]: pgmap v9593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:00 smithi171 ceph-mon[37789]: pgmap v9593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:00.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:00.882+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:30:00.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:00.882+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:30:00.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:00.882+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17346.069036 secs 2024-07-26T13:30:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:01 smithi044 ceph-mon[32404]: pgmap v9594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17346.069036 secs 2024-07-26T13:30:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:01 smithi116 ceph-mon[37097]: pgmap v9594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17346.069036 secs 2024-07-26T13:30:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:01 smithi171 ceph-mon[37789]: pgmap v9594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:02.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:02.811+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:30:02.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:02.811+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:30:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:02.812+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:30:02.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:02.812+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:30:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:04 smithi044 ceph-mon[32404]: pgmap v9595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:04 smithi116 ceph-mon[37097]: pgmap v9595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:04 smithi171 ceph-mon[37789]: pgmap v9595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:06 smithi044 ceph-mon[32404]: pgmap v9596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17351.069199 secs 2024-07-26T13:30:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:06 smithi116 ceph-mon[37097]: pgmap v9596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17351.069199 secs 2024-07-26T13:30:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:06 smithi171 ceph-mon[37789]: pgmap v9596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17351.069199 secs 2024-07-26T13:30:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:08 smithi044 ceph-mon[32404]: pgmap v9597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:08.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:08 smithi116 ceph-mon[37097]: pgmap v9597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:08 smithi171 ceph-mon[37789]: pgmap v9597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:10 smithi116 ceph-mon[37097]: pgmap v9598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:10 smithi171 ceph-mon[37789]: pgmap v9598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:10 smithi044 ceph-mon[32404]: pgmap v9598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17356.069364 secs 2024-07-26T13:30:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17356.069364 secs 2024-07-26T13:30:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17356.069364 secs 2024-07-26T13:30:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:12 smithi171 ceph-mon[37789]: pgmap v9599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:12 smithi044 ceph-mon[32404]: pgmap v9599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:12 smithi116 ceph-mon[37097]: pgmap v9599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:14 smithi171 ceph-mon[37789]: pgmap v9600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:14 smithi044 ceph-mon[32404]: pgmap v9600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:14.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:14 smithi116 ceph-mon[37097]: pgmap v9600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:15.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:15.887+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:30:15.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:15.887+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:30:15.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:15.887+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:16 smithi044 ceph-mon[32404]: pgmap v9601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17361.069529 secs 2024-07-26T13:30:16.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:16 smithi116 ceph-mon[37097]: pgmap v9601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:16.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17361.069529 secs 2024-07-26T13:30:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:16 smithi171 ceph-mon[37789]: pgmap v9601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17361.069529 secs 2024-07-26T13:30:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:18 smithi171 ceph-mon[37789]: pgmap v9602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:18 smithi044 ceph-mon[32404]: pgmap v9602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:18 smithi116 ceph-mon[37097]: pgmap v9602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:20 smithi171 ceph-mon[37789]: pgmap v9603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:20 smithi044 ceph-mon[32404]: pgmap v9603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:20 smithi116 ceph-mon[37097]: pgmap v9603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17366.069673 secs 2024-07-26T13:30:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17366.069673 secs 2024-07-26T13:30:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17366.069673 secs 2024-07-26T13:30:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:22 smithi171 ceph-mon[37789]: pgmap v9604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:22 smithi044 ceph-mon[32404]: pgmap v9604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:22 smithi116 ceph-mon[37097]: pgmap v9604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:24 smithi171 ceph-mon[37789]: pgmap v9605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:24 smithi044 ceph-mon[32404]: pgmap v9605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:24 smithi116 ceph-mon[37097]: pgmap v9605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:26 smithi171 ceph-mon[37789]: pgmap v9606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17371.069829 secs 2024-07-26T13:30:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:26 smithi044 ceph-mon[32404]: pgmap v9606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17371.069829 secs 2024-07-26T13:30:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:26 smithi116 ceph-mon[37097]: pgmap v9606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17371.069829 secs 2024-07-26T13:30:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:28 smithi044 ceph-mon[32404]: pgmap v9607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:28 smithi116 ceph-mon[37097]: pgmap v9607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:28 smithi171 ceph-mon[37789]: pgmap v9607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:29.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: pgmap v9608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:30.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:30:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: pgmap v9608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:30:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: pgmap v9608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:30:30.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:30.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:30:30.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:30.888+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:30:30.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:30.888+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:30.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:30.888+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:30:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:31 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:30:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:31 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:30:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:31 smithi044 ceph-mon[32404]: pgmap v9609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:30:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:31 smithi044 ceph-mon[32404]: pgmap v9610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17376.070002 secs 2024-07-26T13:30:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:31 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:30:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:31 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:30:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:31 smithi116 ceph-mon[37097]: pgmap v9609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:30:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:31 smithi116 ceph-mon[37097]: pgmap v9610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17376.070002 secs 2024-07-26T13:30:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:31 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:30:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:31 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:30:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:31 smithi171 ceph-mon[37789]: pgmap v9609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:30:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:31 smithi171 ceph-mon[37789]: pgmap v9610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17376.070002 secs 2024-07-26T13:30:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:33 smithi044 ceph-mon[32404]: pgmap v9611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:30:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:33 smithi116 ceph-mon[37097]: pgmap v9611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:30:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:33 smithi171 ceph-mon[37789]: pgmap v9611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:30:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:35 smithi044 ceph-mon[32404]: pgmap v9612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:35 smithi116 ceph-mon[37097]: pgmap v9612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:35 smithi171 ceph-mon[37789]: pgmap v9612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17381.070146 secs 2024-07-26T13:30:36.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17381.070146 secs 2024-07-26T13:30:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17381.070146 secs 2024-07-26T13:30:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:37 smithi044 ceph-mon[32404]: pgmap v9613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:30:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:37 smithi116 ceph-mon[37097]: pgmap v9613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:30:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:37 smithi171 ceph-mon[37789]: pgmap v9613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:30:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:38 smithi044 ceph-mon[32404]: pgmap v9614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:38 smithi116 ceph-mon[37097]: pgmap v9614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:38 smithi171 ceph-mon[37789]: pgmap v9614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:30:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:41 smithi171 ceph-mon[37789]: pgmap v9615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:30:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17386.070298 secs 2024-07-26T13:30:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:41 smithi044 ceph-mon[32404]: pgmap v9615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:30:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17386.070298 secs 2024-07-26T13:30:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:41 smithi116 ceph-mon[37097]: pgmap v9615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:30:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17386.070298 secs 2024-07-26T13:30:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:43 smithi171 ceph-mon[37789]: pgmap v9616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:43 smithi044 ceph-mon[32404]: pgmap v9616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:43 smithi116 ceph-mon[37097]: pgmap v9616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:45 smithi171 ceph-mon[37789]: pgmap v9617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:45 smithi044 ceph-mon[32404]: pgmap v9617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:45 smithi116 ceph-mon[37097]: pgmap v9617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:45.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:45.894+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:30:45.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:45.894+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:45.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:30:45.894+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:30:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17391.070430 secs 2024-07-26T13:30:46.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17391.070430 secs 2024-07-26T13:30:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17391.070430 secs 2024-07-26T13:30:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:47 smithi171 ceph-mon[37789]: pgmap v9618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:47.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:47 smithi116 ceph-mon[37097]: pgmap v9618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:47 smithi044 ceph-mon[32404]: pgmap v9618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:49 smithi171 ceph-mon[37789]: pgmap v9619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:49 smithi044 ceph-mon[32404]: pgmap v9619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:49 smithi116 ceph-mon[37097]: pgmap v9619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:51 smithi171 ceph-mon[37789]: pgmap v9620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17396.070583 secs 2024-07-26T13:30:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:51 smithi044 ceph-mon[32404]: pgmap v9620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17396.070583 secs 2024-07-26T13:30:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:51 smithi116 ceph-mon[37097]: pgmap v9620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17396.070583 secs 2024-07-26T13:30:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:53 smithi171 ceph-mon[37789]: pgmap v9621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:53 smithi044 ceph-mon[32404]: pgmap v9621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:53 smithi116 ceph-mon[37097]: pgmap v9621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:30:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:55 smithi044 ceph-mon[32404]: pgmap v9622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:55 smithi116 ceph-mon[37097]: pgmap v9622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:55 smithi171 ceph-mon[37789]: pgmap v9622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:30:56.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17401.070714 secs 2024-07-26T13:30:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17401.070714 secs 2024-07-26T13:30:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17401.070714 secs 2024-07-26T13:30:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:57 smithi044 ceph-mon[32404]: pgmap v9623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:30:57.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:57 smithi116 ceph-mon[37097]: pgmap v9623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:30:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:57 smithi171 ceph-mon[37789]: pgmap v9623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:30:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:30:59 smithi044 ceph-mon[32404]: pgmap v9624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:30:59 smithi116 ceph-mon[37097]: pgmap v9624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:30:59 smithi171 ceph-mon[37789]: pgmap v9624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:00.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:00.894+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:31:00.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:00.894+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:31:00.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:00.894+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:01 smithi044 ceph-mon[32404]: pgmap v9625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17406.070851 secs 2024-07-26T13:31:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:01 smithi116 ceph-mon[37097]: pgmap v9625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17406.070851 secs 2024-07-26T13:31:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:01 smithi171 ceph-mon[37789]: pgmap v9625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17406.070851 secs 2024-07-26T13:31:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:02 smithi044 ceph-mon[32404]: pgmap v9626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:02 smithi116 ceph-mon[37097]: pgmap v9626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:02 smithi171 ceph-mon[37789]: pgmap v9626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:05 smithi171 ceph-mon[37789]: pgmap v9627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:05 smithi044 ceph-mon[32404]: pgmap v9627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:05 smithi116 ceph-mon[37097]: pgmap v9627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17411.070982 secs 2024-07-26T13:31:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17411.070982 secs 2024-07-26T13:31:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17411.070982 secs 2024-07-26T13:31:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:07 smithi171 ceph-mon[37789]: pgmap v9628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:07.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:07 smithi044 ceph-mon[32404]: pgmap v9628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:07 smithi116 ceph-mon[37097]: pgmap v9628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:09 smithi171 ceph-mon[37789]: pgmap v9629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:09.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:09 smithi044 ceph-mon[32404]: pgmap v9629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:09 smithi116 ceph-mon[37097]: pgmap v9629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:11 smithi171 ceph-mon[37789]: pgmap v9630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17416.071124 secs 2024-07-26T13:31:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:11 smithi044 ceph-mon[32404]: pgmap v9630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17416.071124 secs 2024-07-26T13:31:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:11 smithi116 ceph-mon[37097]: pgmap v9630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17416.071124 secs 2024-07-26T13:31:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:13 smithi171 ceph-mon[37789]: pgmap v9631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:13.697 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:13 smithi044 ceph-mon[32404]: pgmap v9631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:13 smithi116 ceph-mon[37097]: pgmap v9631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:15 smithi171 ceph-mon[37789]: pgmap v9632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:15 smithi044 ceph-mon[32404]: pgmap v9632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:15 smithi116 ceph-mon[37097]: pgmap v9632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:15.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:15.900+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:31:15.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:15.900+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:31:15.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:15.900+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:16.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17421.071275 secs 2024-07-26T13:31:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17421.071275 secs 2024-07-26T13:31:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17421.071275 secs 2024-07-26T13:31:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:17 smithi044 ceph-mon[32404]: pgmap v9633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:17 smithi116 ceph-mon[37097]: pgmap v9633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:17 smithi171 ceph-mon[37789]: pgmap v9633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:19 smithi044 ceph-mon[32404]: pgmap v9634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:19.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:19 smithi116 ceph-mon[37097]: pgmap v9634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:19 smithi171 ceph-mon[37789]: pgmap v9634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:21 smithi044 ceph-mon[32404]: pgmap v9635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17426.071422 secs 2024-07-26T13:31:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:21 smithi116 ceph-mon[37097]: pgmap v9635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17426.071422 secs 2024-07-26T13:31:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:21 smithi171 ceph-mon[37789]: pgmap v9635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17426.071422 secs 2024-07-26T13:31:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:23 smithi044 ceph-mon[32404]: pgmap v9636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:23 smithi116 ceph-mon[37097]: pgmap v9636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:23 smithi171 ceph-mon[37789]: pgmap v9636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:25 smithi044 ceph-mon[32404]: pgmap v9637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:25 smithi116 ceph-mon[37097]: pgmap v9637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:25 smithi171 ceph-mon[37789]: pgmap v9637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:26.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17431.071570 secs 2024-07-26T13:31:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17431.071570 secs 2024-07-26T13:31:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17431.071570 secs 2024-07-26T13:31:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:27 smithi044 ceph-mon[32404]: pgmap v9638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:27 smithi116 ceph-mon[37097]: pgmap v9638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:27 smithi171 ceph-mon[37789]: pgmap v9638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:29 smithi044 ceph-mon[32404]: pgmap v9639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:29 smithi116 ceph-mon[37097]: pgmap v9639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:29 smithi171 ceph-mon[37789]: pgmap v9639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:30.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:30.902+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:31:30.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:30.902+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:31:30.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:30.902+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: pgmap v9640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17436.071738 secs 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:31 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: pgmap v9640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17436.071738 secs 2024-07-26T13:31:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:31 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: pgmap v9640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17436.071738 secs 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:31.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:31 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:31:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:31:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:31:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:31:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:31:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:31:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:31:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:31:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:31:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:31:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:31:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:33 smithi044 ceph-mon[32404]: pgmap v9641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:33.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:33 smithi044 ceph-mon[32404]: pgmap v9642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:31:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:33 smithi116 ceph-mon[37097]: pgmap v9641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:33 smithi116 ceph-mon[37097]: pgmap v9642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:31:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:33 smithi171 ceph-mon[37789]: pgmap v9641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:33 smithi171 ceph-mon[37789]: pgmap v9642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:31:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:35 smithi044 ceph-mon[32404]: pgmap v9643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:35 smithi116 ceph-mon[37097]: pgmap v9643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:35 smithi171 ceph-mon[37789]: pgmap v9643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17441.071867 secs 2024-07-26T13:31:36.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17441.071867 secs 2024-07-26T13:31:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17441.071867 secs 2024-07-26T13:31:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:37 smithi044 ceph-mon[32404]: pgmap v9644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:37 smithi116 ceph-mon[37097]: pgmap v9644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:37 smithi171 ceph-mon[37789]: pgmap v9644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:38 smithi044 ceph-mon[32404]: pgmap v9645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:31:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:38 smithi116 ceph-mon[37097]: pgmap v9645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:31:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:38 smithi171 ceph-mon[37789]: pgmap v9645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:31:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:40 smithi171 ceph-mon[37789]: pgmap v9646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:40 smithi044 ceph-mon[32404]: pgmap v9646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:40 smithi116 ceph-mon[37097]: pgmap v9646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17446.072009 secs 2024-07-26T13:31:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17446.072009 secs 2024-07-26T13:31:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17446.072009 secs 2024-07-26T13:31:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:42 smithi044 ceph-mon[32404]: pgmap v9647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:43.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:42 smithi116 ceph-mon[37097]: pgmap v9647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:42 smithi171 ceph-mon[37789]: pgmap v9647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:44.929 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:44 smithi044 ceph-mon[32404]: pgmap v9648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:44 smithi116 ceph-mon[37097]: pgmap v9648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:44 smithi171 ceph-mon[37789]: pgmap v9648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:45.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:45.908+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:31:45.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:45.908+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:45.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:31:45.908+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:31:46.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:46 smithi171 ceph-mon[37789]: pgmap v9649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:46.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17451.072158 secs 2024-07-26T13:31:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:46 smithi044 ceph-mon[32404]: pgmap v9649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17451.072158 secs 2024-07-26T13:31:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:46 smithi116 ceph-mon[37097]: pgmap v9649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17451.072158 secs 2024-07-26T13:31:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:48 smithi044 ceph-mon[32404]: pgmap v9650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:49.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:48 smithi116 ceph-mon[37097]: pgmap v9650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:48 smithi171 ceph-mon[37789]: pgmap v9650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:50 smithi044 ceph-mon[32404]: pgmap v9651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:51.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:50 smithi116 ceph-mon[37097]: pgmap v9651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:50 smithi171 ceph-mon[37789]: pgmap v9651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17456.072319 secs 2024-07-26T13:31:52.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17456.072319 secs 2024-07-26T13:31:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17456.072319 secs 2024-07-26T13:31:53.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:52 smithi044 ceph-mon[32404]: pgmap v9652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:52 smithi116 ceph-mon[37097]: pgmap v9652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:52 smithi171 ceph-mon[37789]: pgmap v9652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:54 smithi044 ceph-mon[32404]: pgmap v9653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:54 smithi116 ceph-mon[37097]: pgmap v9653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:54 smithi171 ceph-mon[37789]: pgmap v9653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:56 smithi044 ceph-mon[32404]: pgmap v9654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17461.072477 secs 2024-07-26T13:31:57.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:56 smithi116 ceph-mon[37097]: pgmap v9654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17461.072477 secs 2024-07-26T13:31:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:56 smithi171 ceph-mon[37789]: pgmap v9654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17461.072477 secs 2024-07-26T13:31:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:31:58 smithi044 ceph-mon[32404]: pgmap v9655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:31:58 smithi116 ceph-mon[37097]: pgmap v9655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:31:58 smithi171 ceph-mon[37789]: pgmap v9655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:00.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:00.910+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:32:00.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:00.910+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:32:00.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:00.910+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:00.912 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:00 smithi044 ceph-mon[32404]: pgmap v9656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:00 smithi116 ceph-mon[37097]: pgmap v9656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:00 smithi171 ceph-mon[37789]: pgmap v9656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17466.072678 secs 2024-07-26T13:32:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17466.072678 secs 2024-07-26T13:32:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17466.072678 secs 2024-07-26T13:32:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:02 smithi044 ceph-mon[32404]: pgmap v9657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:03.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:02 smithi116 ceph-mon[37097]: pgmap v9657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:02 smithi171 ceph-mon[37789]: pgmap v9657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:04 smithi044 ceph-mon[32404]: pgmap v9658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:05.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:04 smithi116 ceph-mon[37097]: pgmap v9658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:04 smithi171 ceph-mon[37789]: pgmap v9658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:06 smithi044 ceph-mon[32404]: pgmap v9659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17471.072815 secs 2024-07-26T13:32:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:06 smithi116 ceph-mon[37097]: pgmap v9659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17471.072815 secs 2024-07-26T13:32:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:06 smithi171 ceph-mon[37789]: pgmap v9659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17471.072815 secs 2024-07-26T13:32:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:08 smithi044 ceph-mon[32404]: pgmap v9660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:08 smithi116 ceph-mon[37097]: pgmap v9660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:08 smithi171 ceph-mon[37789]: pgmap v9660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:10 smithi044 ceph-mon[32404]: pgmap v9661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:10 smithi116 ceph-mon[37097]: pgmap v9661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:10 smithi171 ceph-mon[37789]: pgmap v9661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17476.072954 secs 2024-07-26T13:32:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17476.072954 secs 2024-07-26T13:32:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17476.072954 secs 2024-07-26T13:32:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:12 smithi044 ceph-mon[32404]: pgmap v9662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:13.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:12 smithi116 ceph-mon[37097]: pgmap v9662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:12 smithi171 ceph-mon[37789]: pgmap v9662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:14 smithi044 ceph-mon[32404]: pgmap v9663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:14 smithi116 ceph-mon[37097]: pgmap v9663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:14 smithi171 ceph-mon[37789]: pgmap v9663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:15.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:15.916+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:32:15.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:15.916+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:32:15.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:15.916+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:16 smithi044 ceph-mon[32404]: pgmap v9664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17481.073123 secs 2024-07-26T13:32:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:16 smithi116 ceph-mon[37097]: pgmap v9664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17481.073123 secs 2024-07-26T13:32:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:16 smithi171 ceph-mon[37789]: pgmap v9664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17481.073123 secs 2024-07-26T13:32:17.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:17.837+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:32:17.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:17.837+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:32:17.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:17.838+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:32:17.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:17.838+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:32:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:18 smithi044 ceph-mon[32404]: pgmap v9665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:18 smithi116 ceph-mon[37097]: pgmap v9665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:18 smithi171 ceph-mon[37789]: pgmap v9665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:20 smithi044 ceph-mon[32404]: pgmap v9666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:20 smithi116 ceph-mon[37097]: pgmap v9666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:20 smithi171 ceph-mon[37789]: pgmap v9666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17486.073216 secs 2024-07-26T13:32:22.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17486.073216 secs 2024-07-26T13:32:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17486.073216 secs 2024-07-26T13:32:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:22 smithi044 ceph-mon[32404]: pgmap v9667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:22 smithi116 ceph-mon[37097]: pgmap v9667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:22 smithi171 ceph-mon[37789]: pgmap v9667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:24 smithi044 ceph-mon[32404]: pgmap v9668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:24 smithi116 ceph-mon[37097]: pgmap v9668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:24 smithi171 ceph-mon[37789]: pgmap v9668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:26 smithi044 ceph-mon[32404]: pgmap v9669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17491.073375 secs 2024-07-26T13:32:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:26 smithi116 ceph-mon[37097]: pgmap v9669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17491.073375 secs 2024-07-26T13:32:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:26 smithi171 ceph-mon[37789]: pgmap v9669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17491.073375 secs 2024-07-26T13:32:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:28 smithi044 ceph-mon[32404]: pgmap v9670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:29.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:28 smithi116 ceph-mon[37097]: pgmap v9670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:28 smithi171 ceph-mon[37789]: pgmap v9670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:30.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:30.918+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:32:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:30.918+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:32:30.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:30.918+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:30 smithi044 ceph-mon[32404]: pgmap v9671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:31.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:30 smithi116 ceph-mon[37097]: pgmap v9671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:30 smithi171 ceph-mon[37789]: pgmap v9671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17496.073544 secs 2024-07-26T13:32:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17496.073544 secs 2024-07-26T13:32:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17496.073544 secs 2024-07-26T13:32:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:32 smithi171 ceph-mon[37789]: pgmap v9672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:33.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:32:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:32 smithi044 ceph-mon[32404]: pgmap v9672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:33.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:32:33.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:32 smithi116 ceph-mon[37097]: pgmap v9672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:32:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:33 smithi171 ceph-mon[37789]: pgmap v9673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:32:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:33 smithi044 ceph-mon[32404]: pgmap v9673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:32:34.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:33 smithi116 ceph-mon[37097]: pgmap v9673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:32:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:35 smithi171 ceph-mon[37789]: pgmap v9674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:35 smithi044 ceph-mon[32404]: pgmap v9674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:35 smithi116 ceph-mon[37097]: pgmap v9674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17501.073711 secs 2024-07-26T13:32:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17501.073711 secs 2024-07-26T13:32:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17501.073711 secs 2024-07-26T13:32:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:37 smithi171 ceph-mon[37789]: pgmap v9675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:32:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:37 smithi044 ceph-mon[32404]: pgmap v9675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:32:38.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:37 smithi116 ceph-mon[37097]: pgmap v9675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:32:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:39 smithi171 ceph-mon[37789]: pgmap v9676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:32:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:39 smithi044 ceph-mon[32404]: pgmap v9676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:32:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:39 smithi116 ceph-mon[37097]: pgmap v9676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:32:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17506.073877 secs 2024-07-26T13:32:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17506.073877 secs 2024-07-26T13:32:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17506.073877 secs 2024-07-26T13:32:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:41 smithi044 ceph-mon[32404]: pgmap v9677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:41 smithi116 ceph-mon[37097]: pgmap v9677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:41 smithi171 ceph-mon[37789]: pgmap v9677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:43 smithi044 ceph-mon[32404]: pgmap v9678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:32:44.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:43 smithi116 ceph-mon[37097]: pgmap v9678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:32:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:43 smithi171 ceph-mon[37789]: pgmap v9678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:32:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:45.923+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:32:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:45.923+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:45.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:32:45.923+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:32:46.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:45 smithi171 ceph-mon[37789]: pgmap v9679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:46.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17511.074017 secs 2024-07-26T13:32:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:45 smithi044 ceph-mon[32404]: pgmap v9679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17511.074017 secs 2024-07-26T13:32:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:45 smithi116 ceph-mon[37097]: pgmap v9679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17511.074017 secs 2024-07-26T13:32:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:47 smithi044 ceph-mon[32404]: pgmap v9680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:47 smithi116 ceph-mon[37097]: pgmap v9680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:47 smithi171 ceph-mon[37789]: pgmap v9680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:49 smithi044 ceph-mon[32404]: pgmap v9681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:49 smithi116 ceph-mon[37097]: pgmap v9681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:49 smithi171 ceph-mon[37789]: pgmap v9681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17516.074155 secs 2024-07-26T13:32:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17516.074155 secs 2024-07-26T13:32:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17516.074155 secs 2024-07-26T13:32:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:51 smithi044 ceph-mon[32404]: pgmap v9682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:51 smithi116 ceph-mon[37097]: pgmap v9682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:51 smithi171 ceph-mon[37789]: pgmap v9682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:53 smithi044 ceph-mon[32404]: pgmap v9683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:53 smithi116 ceph-mon[37097]: pgmap v9683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:53 smithi171 ceph-mon[37789]: pgmap v9683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:55 smithi044 ceph-mon[32404]: pgmap v9684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17521.074261 secs 2024-07-26T13:32:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:55 smithi116 ceph-mon[37097]: pgmap v9684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17521.074261 secs 2024-07-26T13:32:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:55 smithi171 ceph-mon[37789]: pgmap v9684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:56.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17521.074261 secs 2024-07-26T13:32:58.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:57 smithi044 ceph-mon[32404]: pgmap v9685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:57 smithi116 ceph-mon[37097]: pgmap v9685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:57 smithi171 ceph-mon[37789]: pgmap v9685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:32:59 smithi044 ceph-mon[32404]: pgmap v9686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:32:59 smithi116 ceph-mon[37097]: pgmap v9686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:32:59 smithi171 ceph-mon[37789]: pgmap v9686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:00.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:00.925+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:33:00.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:00.925+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:33:00.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:00.925+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17526.074406 secs 2024-07-26T13:33:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17526.074406 secs 2024-07-26T13:33:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17526.074406 secs 2024-07-26T13:33:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:01 smithi044 ceph-mon[32404]: pgmap v9687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:01 smithi116 ceph-mon[37097]: pgmap v9687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:01 smithi171 ceph-mon[37789]: pgmap v9687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:03 smithi044 ceph-mon[32404]: pgmap v9688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:03 smithi116 ceph-mon[37097]: pgmap v9688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:03 smithi171 ceph-mon[37789]: pgmap v9688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:05 smithi044 ceph-mon[32404]: pgmap v9689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17531.074567 secs 2024-07-26T13:33:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:05 smithi116 ceph-mon[37097]: pgmap v9689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17531.074567 secs 2024-07-26T13:33:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:05 smithi171 ceph-mon[37789]: pgmap v9689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17531.074567 secs 2024-07-26T13:33:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:07 smithi044 ceph-mon[32404]: pgmap v9690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:07 smithi116 ceph-mon[37097]: pgmap v9690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:07 smithi171 ceph-mon[37789]: pgmap v9690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:10.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:09 smithi044 ceph-mon[32404]: pgmap v9691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:09 smithi116 ceph-mon[37097]: pgmap v9691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:09 smithi171 ceph-mon[37789]: pgmap v9691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17536.074734 secs 2024-07-26T13:33:11.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17536.074734 secs 2024-07-26T13:33:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17536.074734 secs 2024-07-26T13:33:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:11 smithi044 ceph-mon[32404]: pgmap v9692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:33:12.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:11 smithi116 ceph-mon[37097]: pgmap v9692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:33:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:11 smithi171 ceph-mon[37789]: pgmap v9692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:33:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:13 smithi044 ceph-mon[32404]: pgmap v9693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:14.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:13 smithi116 ceph-mon[37097]: pgmap v9693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:13 smithi171 ceph-mon[37789]: pgmap v9693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:15.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:15.930+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:33:15.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:15.930+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:15.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:15.930+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:33:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:15 smithi044 ceph-mon[32404]: pgmap v9694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:33:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17541.074897 secs 2024-07-26T13:33:16.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:15 smithi116 ceph-mon[37097]: pgmap v9694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:33:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17541.074897 secs 2024-07-26T13:33:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:15 smithi171 ceph-mon[37789]: pgmap v9694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:33:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17541.074897 secs 2024-07-26T13:33:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:17 smithi044 ceph-mon[32404]: pgmap v9695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:17 smithi116 ceph-mon[37097]: pgmap v9695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:17 smithi171 ceph-mon[37789]: pgmap v9695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:33:20.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:19 smithi044 ceph-mon[32404]: pgmap v9696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:19 smithi116 ceph-mon[37097]: pgmap v9696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:19 smithi171 ceph-mon[37789]: pgmap v9696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17546.075064 secs 2024-07-26T13:33:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17546.075064 secs 2024-07-26T13:33:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17546.075064 secs 2024-07-26T13:33:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:21 smithi044 ceph-mon[32404]: pgmap v9697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:22.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:21 smithi116 ceph-mon[37097]: pgmap v9697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:21 smithi171 ceph-mon[37789]: pgmap v9697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:24 smithi044 ceph-mon[32404]: pgmap v9698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:24 smithi116 ceph-mon[37097]: pgmap v9698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:24 smithi171 ceph-mon[37789]: pgmap v9698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:26 smithi044 ceph-mon[32404]: pgmap v9699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17551.075226 secs 2024-07-26T13:33:26.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:26 smithi116 ceph-mon[37097]: pgmap v9699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17551.075226 secs 2024-07-26T13:33:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:26 smithi171 ceph-mon[37789]: pgmap v9699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17551.075226 secs 2024-07-26T13:33:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:28 smithi044 ceph-mon[32404]: pgmap v9700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:28 smithi116 ceph-mon[37097]: pgmap v9700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:28 smithi171 ceph-mon[37789]: pgmap v9700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:30 smithi116 ceph-mon[37097]: pgmap v9701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:30 smithi171 ceph-mon[37789]: pgmap v9701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:30 smithi044 ceph-mon[32404]: pgmap v9701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:30.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:30.932+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:33:30.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:30.932+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:33:30.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:30.932+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17556.075370 secs 2024-07-26T13:33:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17556.075370 secs 2024-07-26T13:33:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17556.075370 secs 2024-07-26T13:33:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:32 smithi171 ceph-mon[37789]: pgmap v9702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:32 smithi044 ceph-mon[32404]: pgmap v9702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:32 smithi116 ceph-mon[37097]: pgmap v9702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:34 smithi171 ceph-mon[37789]: pgmap v9703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:34 smithi044 ceph-mon[32404]: pgmap v9703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:34.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:34 smithi116 ceph-mon[37097]: pgmap v9703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:36 smithi171 ceph-mon[37789]: pgmap v9704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:36 smithi171 ceph-mon[37789]: pgmap v9705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17561.075532 secs 2024-07-26T13:33:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:36 smithi044 ceph-mon[32404]: pgmap v9704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:36 smithi044 ceph-mon[32404]: pgmap v9705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17561.075532 secs 2024-07-26T13:33:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:36 smithi116 ceph-mon[37097]: pgmap v9704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:36 smithi116 ceph-mon[37097]: pgmap v9705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17561.075532 secs 2024-07-26T13:33:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:38 smithi171 ceph-mon[37789]: pgmap v9706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:38 smithi044 ceph-mon[32404]: pgmap v9706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:38 smithi116 ceph-mon[37097]: pgmap v9706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:40 smithi171 ceph-mon[37789]: pgmap v9707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:40 smithi044 ceph-mon[32404]: pgmap v9707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:40 smithi116 ceph-mon[37097]: pgmap v9707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17566.075690 secs 2024-07-26T13:33:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17566.075690 secs 2024-07-26T13:33:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17566.075690 secs 2024-07-26T13:33:42.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:42 smithi044 ceph-mon[32404]: pgmap v9708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:42.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:42 smithi116 ceph-mon[37097]: pgmap v9708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:42 smithi171 ceph-mon[37789]: pgmap v9708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:44 smithi044 ceph-mon[32404]: pgmap v9709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:44 smithi116 ceph-mon[37097]: pgmap v9709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:44 smithi171 ceph-mon[37789]: pgmap v9709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:33:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:45.937+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:33:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:45.937+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:45.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:33:45.937+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:33:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:46 smithi044 ceph-mon[32404]: pgmap v9710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17571.075848 secs 2024-07-26T13:33:46.388 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:46 smithi171 ceph-mon[37789]: pgmap v9710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:46.388 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17571.075848 secs 2024-07-26T13:33:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:46 smithi116 ceph-mon[37097]: pgmap v9710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17571.075848 secs 2024-07-26T13:33:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:48 smithi044 ceph-mon[32404]: pgmap v9711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:48 smithi116 ceph-mon[37097]: pgmap v9711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:48 smithi171 ceph-mon[37789]: pgmap v9711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:50 smithi044 ceph-mon[32404]: pgmap v9712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:50 smithi116 ceph-mon[37097]: pgmap v9712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:50.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:50 smithi171 ceph-mon[37789]: pgmap v9712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17576.075983 secs 2024-07-26T13:33:51.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17576.075983 secs 2024-07-26T13:33:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17576.075983 secs 2024-07-26T13:33:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:52 smithi044 ceph-mon[32404]: pgmap v9713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:52 smithi116 ceph-mon[37097]: pgmap v9713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:52 smithi171 ceph-mon[37789]: pgmap v9713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:54.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:54 smithi044 ceph-mon[32404]: pgmap v9714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:54.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:54 smithi116 ceph-mon[37097]: pgmap v9714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:54 smithi171 ceph-mon[37789]: pgmap v9714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:56 smithi044 ceph-mon[32404]: pgmap v9715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17581.076125 secs 2024-07-26T13:33:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:56 smithi116 ceph-mon[37097]: pgmap v9715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17581.076125 secs 2024-07-26T13:33:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:56 smithi171 ceph-mon[37789]: pgmap v9715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:33:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17581.076125 secs 2024-07-26T13:33:58.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:33:58 smithi044 ceph-mon[32404]: pgmap v9716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:58.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:33:58 smithi116 ceph-mon[37097]: pgmap v9716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:33:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:33:58 smithi171 ceph-mon[37789]: pgmap v9716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:00.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:00 smithi044 ceph-mon[32404]: pgmap v9717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:00.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:00 smithi116 ceph-mon[37097]: pgmap v9717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:00 smithi171 ceph-mon[37789]: pgmap v9717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:00.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:00.939+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:34:00.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:00.939+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:00.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:00.939+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:34:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17586.076221 secs 2024-07-26T13:34:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17586.076221 secs 2024-07-26T13:34:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17586.076221 secs 2024-07-26T13:34:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:02 smithi044 ceph-mon[32404]: pgmap v9718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:02 smithi116 ceph-mon[37097]: pgmap v9718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:02 smithi171 ceph-mon[37789]: pgmap v9718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:04 smithi044 ceph-mon[32404]: pgmap v9719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:04 smithi116 ceph-mon[37097]: pgmap v9719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:04 smithi171 ceph-mon[37789]: pgmap v9719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:06 smithi044 ceph-mon[32404]: pgmap v9720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17591.076363 secs 2024-07-26T13:34:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:06 smithi116 ceph-mon[37097]: pgmap v9720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17591.076363 secs 2024-07-26T13:34:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:06 smithi171 ceph-mon[37789]: pgmap v9720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17591.076363 secs 2024-07-26T13:34:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:08 smithi044 ceph-mon[32404]: pgmap v9721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:08.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:08 smithi116 ceph-mon[37097]: pgmap v9721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:08 smithi171 ceph-mon[37789]: pgmap v9721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:10 smithi044 ceph-mon[32404]: pgmap v9722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:10 smithi116 ceph-mon[37097]: pgmap v9722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:10 smithi171 ceph-mon[37789]: pgmap v9722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17596.076512 secs 2024-07-26T13:34:11.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17596.076512 secs 2024-07-26T13:34:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17596.076512 secs 2024-07-26T13:34:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:12 smithi044 ceph-mon[32404]: pgmap v9723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:12.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:12 smithi116 ceph-mon[37097]: pgmap v9723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:12 smithi171 ceph-mon[37789]: pgmap v9723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:14 smithi044 ceph-mon[32404]: pgmap v9724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:14.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:14 smithi116 ceph-mon[37097]: pgmap v9724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:14 smithi171 ceph-mon[37789]: pgmap v9724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:15.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:15.944+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:34:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:15.944+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:15.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:15.944+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:34:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:16 smithi044 ceph-mon[32404]: pgmap v9725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17601.076673 secs 2024-07-26T13:34:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:16 smithi116 ceph-mon[37097]: pgmap v9725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17601.076673 secs 2024-07-26T13:34:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:16 smithi171 ceph-mon[37789]: pgmap v9725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17601.076673 secs 2024-07-26T13:34:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:18 smithi044 ceph-mon[32404]: pgmap v9726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:18 smithi116 ceph-mon[37097]: pgmap v9726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:18 smithi171 ceph-mon[37789]: pgmap v9726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:20 smithi044 ceph-mon[32404]: pgmap v9727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:20 smithi116 ceph-mon[37097]: pgmap v9727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:20 smithi171 ceph-mon[37789]: pgmap v9727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17606.076834 secs 2024-07-26T13:34:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17606.076834 secs 2024-07-26T13:34:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17606.076834 secs 2024-07-26T13:34:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:22 smithi044 ceph-mon[32404]: pgmap v9728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:22.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:22 smithi116 ceph-mon[37097]: pgmap v9728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:22 smithi171 ceph-mon[37789]: pgmap v9728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:24 smithi044 ceph-mon[32404]: pgmap v9729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:24.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:24 smithi116 ceph-mon[37097]: pgmap v9729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:24 smithi171 ceph-mon[37789]: pgmap v9729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:26 smithi044 ceph-mon[32404]: pgmap v9730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17611.076974 secs 2024-07-26T13:34:26.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:26 smithi116 ceph-mon[37097]: pgmap v9730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17611.076974 secs 2024-07-26T13:34:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:26 smithi171 ceph-mon[37789]: pgmap v9730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17611.076974 secs 2024-07-26T13:34:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:28 smithi044 ceph-mon[32404]: pgmap v9731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:28 smithi116 ceph-mon[37097]: pgmap v9731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:28 smithi171 ceph-mon[37789]: pgmap v9731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:30 smithi044 ceph-mon[32404]: pgmap v9732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:30.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:30 smithi116 ceph-mon[37097]: pgmap v9732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:30 smithi171 ceph-mon[37789]: pgmap v9732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:30.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:30.946+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:34:30.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:30.946+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:30.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:30.946+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:34:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17616.077128 secs 2024-07-26T13:34:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17616.077128 secs 2024-07-26T13:34:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17616.077128 secs 2024-07-26T13:34:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:32 smithi171 ceph-mon[37789]: pgmap v9733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:32 smithi044 ceph-mon[32404]: pgmap v9733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:32 smithi116 ceph-mon[37097]: pgmap v9733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:34 smithi171 ceph-mon[37789]: pgmap v9734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:34 smithi044 ceph-mon[32404]: pgmap v9734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:34.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:34 smithi116 ceph-mon[37097]: pgmap v9734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:35.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:35.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:36 smithi171 ceph-mon[37789]: pgmap v9735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17621.077216 secs 2024-07-26T13:34:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:36 smithi044 ceph-mon[32404]: pgmap v9735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17621.077216 secs 2024-07-26T13:34:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:36 smithi116 ceph-mon[37097]: pgmap v9735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17621.077216 secs 2024-07-26T13:34:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: pgmap v9736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:38.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:37 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: pgmap v9736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:37 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: pgmap v9736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:37 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:39 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:39 smithi044 ceph-mon[32404]: pgmap v9737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:39 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:39 smithi116 ceph-mon[37097]: pgmap v9737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:39 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:40.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:39 smithi171 ceph-mon[37789]: pgmap v9737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: pgmap v9738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: pgmap v9739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17626.077370 secs 2024-07-26T13:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: pgmap v9738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:34:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: pgmap v9739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17626.077370 secs 2024-07-26T13:34:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: pgmap v9738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:34:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: pgmap v9739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:41.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17626.077370 secs 2024-07-26T13:34:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:43 smithi044 ceph-mon[32404]: pgmap v9740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:43 smithi116 ceph-mon[37097]: pgmap v9740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:43 smithi171 ceph-mon[37789]: pgmap v9740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:45.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:45 smithi044 ceph-mon[32404]: pgmap v9741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:45 smithi116 ceph-mon[37097]: pgmap v9741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:45 smithi171 ceph-mon[37789]: pgmap v9741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:45.952+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:34:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:45.952+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:45.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:34:45.952+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:34:46.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17631.077516 secs 2024-07-26T13:34:46.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17631.077516 secs 2024-07-26T13:34:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17631.077516 secs 2024-07-26T13:34:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:47 smithi044 ceph-mon[32404]: pgmap v9742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:34:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:47 smithi116 ceph-mon[37097]: pgmap v9742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:34:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:47 smithi171 ceph-mon[37789]: pgmap v9742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:34:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:49 smithi044 ceph-mon[32404]: pgmap v9743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:49.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:49 smithi116 ceph-mon[37097]: pgmap v9743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:49 smithi171 ceph-mon[37789]: pgmap v9743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:51 smithi044 ceph-mon[32404]: pgmap v9744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17636.077669 secs 2024-07-26T13:34:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:51 smithi116 ceph-mon[37097]: pgmap v9744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17636.077669 secs 2024-07-26T13:34:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:51 smithi171 ceph-mon[37789]: pgmap v9744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17636.077669 secs 2024-07-26T13:34:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:53 smithi044 ceph-mon[32404]: pgmap v9745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:53 smithi116 ceph-mon[37097]: pgmap v9745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:53 smithi171 ceph-mon[37789]: pgmap v9745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:34:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:54 smithi044 ceph-mon[32404]: pgmap v9746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:54 smithi116 ceph-mon[37097]: pgmap v9746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:54 smithi171 ceph-mon[37789]: pgmap v9746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:34:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17641.077815 secs 2024-07-26T13:34:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17641.077815 secs 2024-07-26T13:34:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17641.077815 secs 2024-07-26T13:34:57.001 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:56 smithi044 ceph-mon[32404]: pgmap v9747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:56 smithi116 ceph-mon[37097]: pgmap v9747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:56 smithi171 ceph-mon[37789]: pgmap v9747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:34:59 smithi044 ceph-mon[32404]: pgmap v9748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:34:59 smithi116 ceph-mon[37097]: pgmap v9748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:34:59 smithi171 ceph-mon[37789]: pgmap v9748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:00.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:00.953+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:35:00.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:00.953+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:00.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:00.953+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:35:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:01 smithi044 ceph-mon[32404]: pgmap v9749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17646.077922 secs 2024-07-26T13:35:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:01 smithi116 ceph-mon[37097]: pgmap v9749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17646.077922 secs 2024-07-26T13:35:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:01 smithi171 ceph-mon[37789]: pgmap v9749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17646.077922 secs 2024-07-26T13:35:03.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:03 smithi044 ceph-mon[32404]: pgmap v9750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:35:03.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:03 smithi116 ceph-mon[37097]: pgmap v9750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:35:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:03 smithi171 ceph-mon[37789]: pgmap v9750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:35:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:05 smithi044 ceph-mon[32404]: pgmap v9751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:05.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:05 smithi116 ceph-mon[37097]: pgmap v9751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:05 smithi171 ceph-mon[37789]: pgmap v9751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17651.078080 secs 2024-07-26T13:35:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17651.078080 secs 2024-07-26T13:35:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17651.078080 secs 2024-07-26T13:35:07.681 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:07 smithi044 ceph-mon[32404]: pgmap v9752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:35:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:07 smithi116 ceph-mon[37097]: pgmap v9752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:35:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:07 smithi171 ceph-mon[37789]: pgmap v9752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:35:09.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:09 smithi044 ceph-mon[32404]: pgmap v9753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:35:09.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:09 smithi116 ceph-mon[37097]: pgmap v9753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:35:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:09 smithi171 ceph-mon[37789]: pgmap v9753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:35:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:11 smithi044 ceph-mon[32404]: pgmap v9754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17656.078207 secs 2024-07-26T13:35:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:11 smithi116 ceph-mon[37097]: pgmap v9754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17656.078207 secs 2024-07-26T13:35:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:11 smithi171 ceph-mon[37789]: pgmap v9754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17656.078207 secs 2024-07-26T13:35:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:13 smithi044 ceph-mon[32404]: pgmap v9755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:35:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:13 smithi116 ceph-mon[37097]: pgmap v9755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:35:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:13 smithi171 ceph-mon[37789]: pgmap v9755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:35:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:15 smithi044 ceph-mon[32404]: pgmap v9756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:15 smithi116 ceph-mon[37097]: pgmap v9756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:15 smithi171 ceph-mon[37789]: pgmap v9756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:15.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:15.959+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:35:15.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:15.959+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:15.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:15.959+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:35:16.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17661.078346 secs 2024-07-26T13:35:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17661.078346 secs 2024-07-26T13:35:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17661.078346 secs 2024-07-26T13:35:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:17 smithi044 ceph-mon[32404]: pgmap v9757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:17 smithi116 ceph-mon[37097]: pgmap v9757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:17 smithi171 ceph-mon[37789]: pgmap v9757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:17.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:17.869+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:35:17.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:17.869+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:35:17.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:17.870+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:35:17.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:17.870+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:35:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:19 smithi044 ceph-mon[32404]: pgmap v9758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:19.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:19 smithi116 ceph-mon[37097]: pgmap v9758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:19 smithi171 ceph-mon[37789]: pgmap v9758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:21 smithi044 ceph-mon[32404]: pgmap v9759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17666.078493 secs 2024-07-26T13:35:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:21 smithi116 ceph-mon[37097]: pgmap v9759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17666.078493 secs 2024-07-26T13:35:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:21 smithi171 ceph-mon[37789]: pgmap v9759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17666.078493 secs 2024-07-26T13:35:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:23 smithi044 ceph-mon[32404]: pgmap v9760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:35:23.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:23 smithi116 ceph-mon[37097]: pgmap v9760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:35:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:23 smithi171 ceph-mon[37789]: pgmap v9760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:35:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:25 smithi044 ceph-mon[32404]: pgmap v9761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:25 smithi116 ceph-mon[37097]: pgmap v9761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:25 smithi171 ceph-mon[37789]: pgmap v9761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17671.078655 secs 2024-07-26T13:35:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17671.078655 secs 2024-07-26T13:35:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17671.078655 secs 2024-07-26T13:35:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:27 smithi044 ceph-mon[32404]: pgmap v9762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:27.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:27 smithi116 ceph-mon[37097]: pgmap v9762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:27 smithi171 ceph-mon[37789]: pgmap v9762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:28 smithi044 ceph-mon[32404]: pgmap v9763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:35:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:28 smithi116 ceph-mon[37097]: pgmap v9763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:35:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:28 smithi171 ceph-mon[37789]: pgmap v9763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:35:30.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:30.960+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:35:30.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:30.960+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:35:30.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:30.960+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:31 smithi044 ceph-mon[32404]: pgmap v9764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17676.078816 secs 2024-07-26T13:35:31.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:31 smithi116 ceph-mon[37097]: pgmap v9764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17676.078816 secs 2024-07-26T13:35:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:31 smithi171 ceph-mon[37789]: pgmap v9764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17676.078816 secs 2024-07-26T13:35:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:33 smithi044 ceph-mon[32404]: pgmap v9765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:33 smithi116 ceph-mon[37097]: pgmap v9765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:33 smithi171 ceph-mon[37789]: pgmap v9765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:35 smithi044 ceph-mon[32404]: pgmap v9766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:35:35.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:35 smithi116 ceph-mon[37097]: pgmap v9766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:35:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:35 smithi171 ceph-mon[37789]: pgmap v9766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:35:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17681.078977 secs 2024-07-26T13:35:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17681.078977 secs 2024-07-26T13:35:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17681.078977 secs 2024-07-26T13:35:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:37 smithi044 ceph-mon[32404]: pgmap v9767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:37 smithi116 ceph-mon[37097]: pgmap v9767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:37 smithi171 ceph-mon[37789]: pgmap v9767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:39 smithi044 ceph-mon[32404]: pgmap v9768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:35:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:39 smithi116 ceph-mon[37097]: pgmap v9768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:35:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:39 smithi171 ceph-mon[37789]: pgmap v9768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:35:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:41 smithi044 ceph-mon[32404]: pgmap v9769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17686.079069 secs 2024-07-26T13:35:41.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:41 smithi116 ceph-mon[37097]: pgmap v9769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17686.079069 secs 2024-07-26T13:35:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:41 smithi171 ceph-mon[37789]: pgmap v9769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:41.860 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:41.860 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17686.079069 secs 2024-07-26T13:35:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:42 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:42 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:42.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:42.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:42.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:42 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:43 smithi044 ceph-mon[32404]: pgmap v9770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:43 smithi044 ceph-mon[32404]: pgmap v9771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:35:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:43 smithi116 ceph-mon[37097]: pgmap v9770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:43 smithi116 ceph-mon[37097]: pgmap v9771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:35:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:43 smithi171 ceph-mon[37789]: pgmap v9770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:43.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:43 smithi171 ceph-mon[37789]: pgmap v9771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:35:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:45 smithi044 ceph-mon[32404]: pgmap v9772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:45 smithi116 ceph-mon[37097]: pgmap v9772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:45 smithi171 ceph-mon[37789]: pgmap v9772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:45.965+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:35:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:45.965+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:35:45.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:35:45.965+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:46.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17691.079217 secs 2024-07-26T13:35:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17691.079217 secs 2024-07-26T13:35:46.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17691.079217 secs 2024-07-26T13:35:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:47 smithi044 ceph-mon[32404]: pgmap v9773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:47.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:47 smithi116 ceph-mon[37097]: pgmap v9773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:47 smithi171 ceph-mon[37789]: pgmap v9773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:49 smithi044 ceph-mon[32404]: pgmap v9774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:49 smithi116 ceph-mon[37097]: pgmap v9774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:49 smithi171 ceph-mon[37789]: pgmap v9774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:51 smithi044 ceph-mon[32404]: pgmap v9775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17696.079387 secs 2024-07-26T13:35:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:51 smithi116 ceph-mon[37097]: pgmap v9775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17696.079387 secs 2024-07-26T13:35:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:51 smithi171 ceph-mon[37789]: pgmap v9775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:35:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17696.079387 secs 2024-07-26T13:35:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:53 smithi044 ceph-mon[32404]: pgmap v9776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:53 smithi116 ceph-mon[37097]: pgmap v9776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:53 smithi171 ceph-mon[37789]: pgmap v9776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:54 smithi044 ceph-mon[32404]: pgmap v9777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:54.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:54 smithi116 ceph-mon[37097]: pgmap v9777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:54 smithi171 ceph-mon[37789]: pgmap v9777: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:56 smithi044 ceph-mon[32404]: pgmap v9778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17701.079545 secs 2024-07-26T13:35:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:56 smithi116 ceph-mon[37097]: pgmap v9778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17701.079545 secs 2024-07-26T13:35:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:56 smithi171 ceph-mon[37789]: pgmap v9778: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17701.079545 secs 2024-07-26T13:35:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:35:58 smithi044 ceph-mon[32404]: pgmap v9779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:35:58 smithi116 ceph-mon[37097]: pgmap v9779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:35:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:35:58 smithi171 ceph-mon[37789]: pgmap v9779: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:00.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:00.966+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:36:00.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:00.966+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:36:00.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:00.966+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:00.969 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:00 smithi044 ceph-mon[32404]: pgmap v9780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:00 smithi116 ceph-mon[37097]: pgmap v9780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:00 smithi171 ceph-mon[37789]: pgmap v9780: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17706.079731 secs 2024-07-26T13:36:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17706.079731 secs 2024-07-26T13:36:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17706.079731 secs 2024-07-26T13:36:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:02 smithi044 ceph-mon[32404]: pgmap v9781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:02 smithi116 ceph-mon[37097]: pgmap v9781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:02 smithi171 ceph-mon[37789]: pgmap v9781: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:04 smithi044 ceph-mon[32404]: pgmap v9782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:04 smithi116 ceph-mon[37097]: pgmap v9782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:04 smithi171 ceph-mon[37789]: pgmap v9782: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:06 smithi044 ceph-mon[32404]: pgmap v9783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17711.079847 secs 2024-07-26T13:36:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:06 smithi116 ceph-mon[37097]: pgmap v9783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17711.079847 secs 2024-07-26T13:36:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:06 smithi171 ceph-mon[37789]: pgmap v9783: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17711.079847 secs 2024-07-26T13:36:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:08 smithi044 ceph-mon[32404]: pgmap v9784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:08 smithi116 ceph-mon[37097]: pgmap v9784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:08 smithi171 ceph-mon[37789]: pgmap v9784: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:10 smithi044 ceph-mon[32404]: pgmap v9785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:10 smithi116 ceph-mon[37097]: pgmap v9785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:10 smithi171 ceph-mon[37789]: pgmap v9785: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17716.079999 secs 2024-07-26T13:36:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17716.079999 secs 2024-07-26T13:36:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17716.079999 secs 2024-07-26T13:36:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:12 smithi044 ceph-mon[32404]: pgmap v9786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:12 smithi116 ceph-mon[37097]: pgmap v9786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:12 smithi171 ceph-mon[37789]: pgmap v9786: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:14 smithi044 ceph-mon[32404]: pgmap v9787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:14 smithi116 ceph-mon[37097]: pgmap v9787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:14 smithi171 ceph-mon[37789]: pgmap v9787: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:15.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:15.972+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:36:15.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:15.972+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:36:15.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:15.972+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:16 smithi116 ceph-mon[37097]: pgmap v9788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17721.080138 secs 2024-07-26T13:36:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:16 smithi171 ceph-mon[37789]: pgmap v9788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17721.080138 secs 2024-07-26T13:36:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:16 smithi044 ceph-mon[32404]: pgmap v9788: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17721.080138 secs 2024-07-26T13:36:19.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:18 smithi171 ceph-mon[37789]: pgmap v9789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:19.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:18 smithi044 ceph-mon[32404]: pgmap v9789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:18 smithi116 ceph-mon[37097]: pgmap v9789: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:20 smithi171 ceph-mon[37789]: pgmap v9790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:20 smithi044 ceph-mon[32404]: pgmap v9790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:21.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:20 smithi116 ceph-mon[37097]: pgmap v9790: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17726.080287 secs 2024-07-26T13:36:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17726.080287 secs 2024-07-26T13:36:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17726.080287 secs 2024-07-26T13:36:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:22 smithi171 ceph-mon[37789]: pgmap v9791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:22 smithi044 ceph-mon[32404]: pgmap v9791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:23.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:22 smithi116 ceph-mon[37097]: pgmap v9791: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:24 smithi171 ceph-mon[37789]: pgmap v9792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:24 smithi044 ceph-mon[32404]: pgmap v9792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:24 smithi116 ceph-mon[37097]: pgmap v9792: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:27.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:26 smithi044 ceph-mon[32404]: pgmap v9793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:27.069 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17731.080444 secs 2024-07-26T13:36:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:26 smithi171 ceph-mon[37789]: pgmap v9793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17731.080444 secs 2024-07-26T13:36:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:26 smithi116 ceph-mon[37097]: pgmap v9793: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17731.080444 secs 2024-07-26T13:36:29.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:28 smithi171 ceph-mon[37789]: pgmap v9794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:28 smithi044 ceph-mon[32404]: pgmap v9794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:28 smithi116 ceph-mon[37097]: pgmap v9794: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:30.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:30.972+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:36:30.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:30.972+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:36:30.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:30.972+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:31.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:30 smithi171 ceph-mon[37789]: pgmap v9795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:30 smithi044 ceph-mon[32404]: pgmap v9795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:31.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:30 smithi116 ceph-mon[37097]: pgmap v9795: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17736.080665 secs 2024-07-26T13:36:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17736.080665 secs 2024-07-26T13:36:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17736.080665 secs 2024-07-26T13:36:33.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:32 smithi171 ceph-mon[37789]: pgmap v9796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:32 smithi044 ceph-mon[32404]: pgmap v9796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:32 smithi116 ceph-mon[37097]: pgmap v9796: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:34 smithi171 ceph-mon[37789]: pgmap v9797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:34 smithi044 ceph-mon[32404]: pgmap v9797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:34 smithi116 ceph-mon[37097]: pgmap v9797: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:36 smithi044 ceph-mon[32404]: pgmap v9798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17741.080747 secs 2024-07-26T13:36:37.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:36 smithi116 ceph-mon[37097]: pgmap v9798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17741.080747 secs 2024-07-26T13:36:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:36 smithi171 ceph-mon[37789]: pgmap v9798: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:37.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17741.080747 secs 2024-07-26T13:36:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:38 smithi044 ceph-mon[32404]: pgmap v9799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:39.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:38 smithi116 ceph-mon[37097]: pgmap v9799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:38 smithi171 ceph-mon[37789]: pgmap v9799: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:40 smithi044 ceph-mon[32404]: pgmap v9800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:40 smithi116 ceph-mon[37097]: pgmap v9800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:40 smithi171 ceph-mon[37789]: pgmap v9800: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17746.080887 secs 2024-07-26T13:36:42.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17746.080887 secs 2024-07-26T13:36:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17746.080887 secs 2024-07-26T13:36:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:42 smithi044 ceph-mon[32404]: pgmap v9801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:42 smithi116 ceph-mon[37097]: pgmap v9801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:42 smithi171 ceph-mon[37789]: pgmap v9801: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:43.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:43 smithi044 ceph-mon[32404]: pgmap v9802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:44.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:43 smithi044 ceph-mon[32404]: pgmap v9803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:44.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:43 smithi116 ceph-mon[37097]: pgmap v9802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:43 smithi116 ceph-mon[37097]: pgmap v9803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:43 smithi171 ceph-mon[37789]: pgmap v9802: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:43 smithi171 ceph-mon[37789]: pgmap v9803: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:44.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:45.976+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:36:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:45.976+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:36:45.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:36:45.976+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:45 smithi044 ceph-mon[32404]: pgmap v9804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17751.080992 secs 2024-07-26T13:36:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:45 smithi116 ceph-mon[37097]: pgmap v9804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17751.080992 secs 2024-07-26T13:36:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:45 smithi171 ceph-mon[37789]: pgmap v9804: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17751.080992 secs 2024-07-26T13:36:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:47 smithi044 ceph-mon[32404]: pgmap v9805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:47 smithi116 ceph-mon[37097]: pgmap v9805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:47 smithi171 ceph-mon[37789]: pgmap v9805: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:50.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:49 smithi044 ceph-mon[32404]: pgmap v9806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:49 smithi116 ceph-mon[37097]: pgmap v9806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:49 smithi171 ceph-mon[37789]: pgmap v9806: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:36:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:50 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17756.081162 secs 2024-07-26T13:36:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:50 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17756.081162 secs 2024-07-26T13:36:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:50 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17756.081162 secs 2024-07-26T13:36:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:51 smithi044 ceph-mon[32404]: pgmap v9807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:51 smithi116 ceph-mon[37097]: pgmap v9807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:51 smithi171 ceph-mon[37789]: pgmap v9807: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:53 smithi044 ceph-mon[32404]: pgmap v9808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:36:54.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:53 smithi116 ceph-mon[37097]: pgmap v9808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:36:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:53 smithi171 ceph-mon[37789]: pgmap v9808: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:36:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:55 smithi044 ceph-mon[32404]: pgmap v9809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17761.081305 secs 2024-07-26T13:36:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:55 smithi116 ceph-mon[37097]: pgmap v9809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17761.081305 secs 2024-07-26T13:36:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:55 smithi171 ceph-mon[37789]: pgmap v9809: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:36:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17761.081305 secs 2024-07-26T13:36:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:36:58 smithi044 ceph-mon[32404]: pgmap v9810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:36:58 smithi116 ceph-mon[37097]: pgmap v9810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:36:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:36:58 smithi171 ceph-mon[37789]: pgmap v9810: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:00 smithi044 ceph-mon[32404]: pgmap v9811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:00 smithi116 ceph-mon[37097]: pgmap v9811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:00 smithi171 ceph-mon[37789]: pgmap v9811: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:00.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:00.977+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:37:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:00.977+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:00.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:00.977+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:37:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17766.081474 secs 2024-07-26T13:37:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17766.081474 secs 2024-07-26T13:37:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17766.081474 secs 2024-07-26T13:37:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:02 smithi171 ceph-mon[37789]: pgmap v9812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:02 smithi044 ceph-mon[32404]: pgmap v9812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:02.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:02 smithi116 ceph-mon[37097]: pgmap v9812: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:04 smithi171 ceph-mon[37789]: pgmap v9813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:04 smithi044 ceph-mon[32404]: pgmap v9813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:04 smithi116 ceph-mon[37097]: pgmap v9813: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:06 smithi171 ceph-mon[37789]: pgmap v9814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17771.081672 secs 2024-07-26T13:37:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:06 smithi044 ceph-mon[32404]: pgmap v9814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17771.081672 secs 2024-07-26T13:37:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:06 smithi116 ceph-mon[37097]: pgmap v9814: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17771.081672 secs 2024-07-26T13:37:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:08 smithi171 ceph-mon[37789]: pgmap v9815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:08 smithi044 ceph-mon[32404]: pgmap v9815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:08.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:08 smithi116 ceph-mon[37097]: pgmap v9815: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:10 smithi171 ceph-mon[37789]: pgmap v9816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:10 smithi044 ceph-mon[32404]: pgmap v9816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:10 smithi116 ceph-mon[37097]: pgmap v9816: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17776.081736 secs 2024-07-26T13:37:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17776.081736 secs 2024-07-26T13:37:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17776.081736 secs 2024-07-26T13:37:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:12 smithi171 ceph-mon[37789]: pgmap v9817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:12 smithi044 ceph-mon[32404]: pgmap v9817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:12 smithi116 ceph-mon[37097]: pgmap v9817: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:14 smithi171 ceph-mon[37789]: pgmap v9818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:14 smithi044 ceph-mon[32404]: pgmap v9818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:14 smithi116 ceph-mon[37097]: pgmap v9818: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:15.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:15.983+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:37:15.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:15.983+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:37:15.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:15.983+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:16 smithi044 ceph-mon[32404]: pgmap v9819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:16.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17781.081885 secs 2024-07-26T13:37:16.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:16 smithi116 ceph-mon[37097]: pgmap v9819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:16.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17781.081885 secs 2024-07-26T13:37:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:16 smithi171 ceph-mon[37789]: pgmap v9819: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17781.081885 secs 2024-07-26T13:37:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:18 smithi171 ceph-mon[37789]: pgmap v9820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:18 smithi044 ceph-mon[32404]: pgmap v9820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:18 smithi116 ceph-mon[37097]: pgmap v9820: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:20 smithi171 ceph-mon[37789]: pgmap v9821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:20 smithi044 ceph-mon[32404]: pgmap v9821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:20 smithi116 ceph-mon[37097]: pgmap v9821: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17786.082050 secs 2024-07-26T13:37:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17786.082050 secs 2024-07-26T13:37:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17786.082050 secs 2024-07-26T13:37:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:22 smithi044 ceph-mon[32404]: pgmap v9822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:22 smithi116 ceph-mon[37097]: pgmap v9822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:22 smithi171 ceph-mon[37789]: pgmap v9822: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:24 smithi044 ceph-mon[32404]: pgmap v9823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:24 smithi116 ceph-mon[37097]: pgmap v9823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:24 smithi171 ceph-mon[37789]: pgmap v9823: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:26 smithi044 ceph-mon[32404]: pgmap v9824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17791.082218 secs 2024-07-26T13:37:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:26 smithi116 ceph-mon[37097]: pgmap v9824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17791.082218 secs 2024-07-26T13:37:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:26 smithi171 ceph-mon[37789]: pgmap v9824: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17791.082218 secs 2024-07-26T13:37:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:28 smithi044 ceph-mon[32404]: pgmap v9825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:28 smithi116 ceph-mon[37097]: pgmap v9825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:28 smithi171 ceph-mon[37789]: pgmap v9825: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:30 smithi044 ceph-mon[32404]: pgmap v9826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:30 smithi116 ceph-mon[37097]: pgmap v9826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:30 smithi171 ceph-mon[37789]: pgmap v9826: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:30.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:30.983+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:37:30.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:30.984+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:37:30.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:30.984+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17796.082315 secs 2024-07-26T13:37:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17796.082315 secs 2024-07-26T13:37:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17796.082315 secs 2024-07-26T13:37:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:32 smithi044 ceph-mon[32404]: pgmap v9827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:32 smithi116 ceph-mon[37097]: pgmap v9827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:32 smithi171 ceph-mon[37789]: pgmap v9827: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:34 smithi044 ceph-mon[32404]: pgmap v9828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:34 smithi116 ceph-mon[37097]: pgmap v9828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:34 smithi171 ceph-mon[37789]: pgmap v9828: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:36 smithi044 ceph-mon[32404]: pgmap v9829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17801.082445 secs 2024-07-26T13:37:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:36 smithi116 ceph-mon[37097]: pgmap v9829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17801.082445 secs 2024-07-26T13:37:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:36 smithi171 ceph-mon[37789]: pgmap v9829: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:37:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17801.082445 secs 2024-07-26T13:37:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:38 smithi044 ceph-mon[32404]: pgmap v9830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:38 smithi116 ceph-mon[37097]: pgmap v9830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:38 smithi171 ceph-mon[37789]: pgmap v9830: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:40.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:40 smithi044 ceph-mon[32404]: pgmap v9831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:40 smithi116 ceph-mon[37097]: pgmap v9831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:40 smithi171 ceph-mon[37789]: pgmap v9831: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17806.082562 secs 2024-07-26T13:37:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17806.082562 secs 2024-07-26T13:37:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17806.082562 secs 2024-07-26T13:37:42.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:42 smithi044 ceph-mon[32404]: pgmap v9832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:42 smithi116 ceph-mon[37097]: pgmap v9832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:42 smithi171 ceph-mon[37789]: pgmap v9832: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:43.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:44.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:44 smithi044 ceph-mon[32404]: pgmap v9833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:44.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:37:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:44 smithi116 ceph-mon[37097]: pgmap v9833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:44.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:37:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:44 smithi171 ceph-mon[37789]: pgmap v9833: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:37:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:44.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:37:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:45 smithi044 ceph-mon[32404]: pgmap v9834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:37:45.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:45 smithi044 ceph-mon[32404]: pgmap v9835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:45 smithi116 ceph-mon[37097]: pgmap v9834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:37:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:45 smithi116 ceph-mon[37097]: pgmap v9835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:45 smithi171 ceph-mon[37789]: pgmap v9834: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:37:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:45 smithi171 ceph-mon[37789]: pgmap v9835: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:45.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:45.989+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:37:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:45.989+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:45.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:37:45.989+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:37:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17811.082702 secs 2024-07-26T13:37:46.487 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17811.082702 secs 2024-07-26T13:37:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17811.082702 secs 2024-07-26T13:37:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:47 smithi044 ceph-mon[32404]: pgmap v9836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:47 smithi116 ceph-mon[37097]: pgmap v9836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:47 smithi171 ceph-mon[37789]: pgmap v9836: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:49 smithi044 ceph-mon[32404]: pgmap v9837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:49 smithi116 ceph-mon[37097]: pgmap v9837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:49 smithi171 ceph-mon[37789]: pgmap v9837: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:51 smithi044 ceph-mon[32404]: pgmap v9838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17816.082865 secs 2024-07-26T13:37:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:51 smithi116 ceph-mon[37097]: pgmap v9838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17816.082865 secs 2024-07-26T13:37:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:51 smithi171 ceph-mon[37789]: pgmap v9838: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17816.082865 secs 2024-07-26T13:37:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:53 smithi044 ceph-mon[32404]: pgmap v9839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:53 smithi116 ceph-mon[37097]: pgmap v9839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:53 smithi171 ceph-mon[37789]: pgmap v9839: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:37:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:55 smithi044 ceph-mon[32404]: pgmap v9840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:37:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:55 smithi116 ceph-mon[37097]: pgmap v9840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:37:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:55 smithi171 ceph-mon[37789]: pgmap v9840: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:37:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17821.083024 secs 2024-07-26T13:37:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17821.083024 secs 2024-07-26T13:37:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17821.083024 secs 2024-07-26T13:37:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:57 smithi044 ceph-mon[32404]: pgmap v9841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:57 smithi116 ceph-mon[37097]: pgmap v9841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:57 smithi171 ceph-mon[37789]: pgmap v9841: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:37:59 smithi044 ceph-mon[32404]: pgmap v9842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:37:59 smithi116 ceph-mon[37097]: pgmap v9842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:37:59 smithi171 ceph-mon[37789]: pgmap v9842: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:00.990+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:38:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:00.990+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:00.990+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:38:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:01 smithi044 ceph-mon[32404]: pgmap v9843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17826.083176 secs 2024-07-26T13:38:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:01 smithi116 ceph-mon[37097]: pgmap v9843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17826.083176 secs 2024-07-26T13:38:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:01 smithi171 ceph-mon[37789]: pgmap v9843: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:01.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17826.083176 secs 2024-07-26T13:38:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:03 smithi044 ceph-mon[32404]: pgmap v9844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:03 smithi116 ceph-mon[37097]: pgmap v9844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:03 smithi171 ceph-mon[37789]: pgmap v9844: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:05 smithi116 ceph-mon[37097]: pgmap v9845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:05 smithi171 ceph-mon[37789]: pgmap v9845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:05 smithi044 ceph-mon[32404]: pgmap v9845: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17831.083314 secs 2024-07-26T13:38:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17831.083314 secs 2024-07-26T13:38:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17831.083314 secs 2024-07-26T13:38:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:07 smithi171 ceph-mon[37789]: pgmap v9846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:07.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:07 smithi044 ceph-mon[32404]: pgmap v9846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:07 smithi116 ceph-mon[37097]: pgmap v9846: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:09 smithi171 ceph-mon[37789]: pgmap v9847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:09.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:09 smithi044 ceph-mon[32404]: pgmap v9847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:09 smithi116 ceph-mon[37097]: pgmap v9847: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:11 smithi171 ceph-mon[37789]: pgmap v9848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17836.083463 secs 2024-07-26T13:38:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:11 smithi044 ceph-mon[32404]: pgmap v9848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17836.083463 secs 2024-07-26T13:38:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:11 smithi116 ceph-mon[37097]: pgmap v9848: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17836.083463 secs 2024-07-26T13:38:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:13 smithi171 ceph-mon[37789]: pgmap v9849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:13 smithi044 ceph-mon[32404]: pgmap v9849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:13 smithi116 ceph-mon[37097]: pgmap v9849: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:15 smithi171 ceph-mon[37789]: pgmap v9850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:15 smithi044 ceph-mon[32404]: pgmap v9850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:15 smithi116 ceph-mon[37097]: pgmap v9850: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:15.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:15.995+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:38:15.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:15.995+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:38:15.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:15.995+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17841.083603 secs 2024-07-26T13:38:16.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17841.083603 secs 2024-07-26T13:38:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17841.083603 secs 2024-07-26T13:38:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:17 smithi171 ceph-mon[37789]: pgmap v9851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:17 smithi044 ceph-mon[32404]: pgmap v9851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:17 smithi116 ceph-mon[37097]: pgmap v9851: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:17.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.904+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:38:17.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.904+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:38:17.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.904+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7fcf40003690 con 0x7fcf6c1057b0 2024-07-26T13:38:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.904+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:38:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.904+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7fcf3c003690 con 0x55c94889a070 2024-07-26T13:38:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.904+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:38:17.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.905+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:38:17.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:17.906+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:38:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:19 smithi171 ceph-mon[37789]: pgmap v9852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:19 smithi044 ceph-mon[32404]: pgmap v9852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:19 smithi116 ceph-mon[37097]: pgmap v9852: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:21 smithi171 ceph-mon[37789]: pgmap v9853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17846.083727 secs 2024-07-26T13:38:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:21 smithi044 ceph-mon[32404]: pgmap v9853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17846.083727 secs 2024-07-26T13:38:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:21 smithi116 ceph-mon[37097]: pgmap v9853: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17846.083727 secs 2024-07-26T13:38:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:23 smithi044 ceph-mon[32404]: pgmap v9854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:23 smithi116 ceph-mon[37097]: pgmap v9854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:23 smithi171 ceph-mon[37789]: pgmap v9854: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:25 smithi044 ceph-mon[32404]: pgmap v9855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:25 smithi116 ceph-mon[37097]: pgmap v9855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:25 smithi171 ceph-mon[37789]: pgmap v9855: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17851.083881 secs 2024-07-26T13:38:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17851.083881 secs 2024-07-26T13:38:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17851.083881 secs 2024-07-26T13:38:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:27 smithi044 ceph-mon[32404]: pgmap v9856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:27 smithi116 ceph-mon[37097]: pgmap v9856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:27 smithi171 ceph-mon[37789]: pgmap v9856: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:29.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:29 smithi044 ceph-mon[32404]: pgmap v9857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:29 smithi116 ceph-mon[37097]: pgmap v9857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:29 smithi171 ceph-mon[37789]: pgmap v9857: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:30.995+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:38:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:30.995+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:38:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:30.995+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:31 smithi044 ceph-mon[32404]: pgmap v9858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17856.084025 secs 2024-07-26T13:38:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:31 smithi116 ceph-mon[37097]: pgmap v9858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17856.084025 secs 2024-07-26T13:38:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:31 smithi171 ceph-mon[37789]: pgmap v9858: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17856.084025 secs 2024-07-26T13:38:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:33 smithi044 ceph-mon[32404]: pgmap v9859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:33.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:33 smithi116 ceph-mon[37097]: pgmap v9859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:33 smithi171 ceph-mon[37789]: pgmap v9859: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:35 smithi044 ceph-mon[32404]: pgmap v9860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:35 smithi116 ceph-mon[37097]: pgmap v9860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:35 smithi171 ceph-mon[37789]: pgmap v9860: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17861.084177 secs 2024-07-26T13:38:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17861.084177 secs 2024-07-26T13:38:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17861.084177 secs 2024-07-26T13:38:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:37 smithi044 ceph-mon[32404]: pgmap v9861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:37 smithi116 ceph-mon[37097]: pgmap v9861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:37 smithi171 ceph-mon[37789]: pgmap v9861: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:39 smithi044 ceph-mon[32404]: pgmap v9862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:39 smithi116 ceph-mon[37097]: pgmap v9862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:39 smithi171 ceph-mon[37789]: pgmap v9862: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:41 smithi044 ceph-mon[32404]: pgmap v9863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17866.084344 secs 2024-07-26T13:38:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:41 smithi116 ceph-mon[37097]: pgmap v9863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17866.084344 secs 2024-07-26T13:38:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:41 smithi171 ceph-mon[37789]: pgmap v9863: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17866.084344 secs 2024-07-26T13:38:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:43 smithi044 ceph-mon[32404]: pgmap v9864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:43 smithi116 ceph-mon[37097]: pgmap v9864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:43 smithi171 ceph-mon[37789]: pgmap v9864: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:44 smithi044 ceph-mon[32404]: pgmap v9865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:45.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:45.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:44 smithi116 ceph-mon[37097]: pgmap v9865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:44 smithi171 ceph-mon[37789]: pgmap v9865: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:38:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:46.001+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:38:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:46.001+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:38:46.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:38:46.001+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:46.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:46.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:45 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:46.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:45 smithi044 ceph-mon[32404]: pgmap v9866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:38:46.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:45 smithi044 ceph-mon[32404]: pgmap v9867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:46.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:45 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:38:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:45 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:45 smithi116 ceph-mon[37097]: pgmap v9866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:38:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:45 smithi116 ceph-mon[37097]: pgmap v9867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:45 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:38:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:45 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:45 smithi171 ceph-mon[37789]: pgmap v9866: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:38:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:45 smithi171 ceph-mon[37789]: pgmap v9867: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:45 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:38:46.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17871.084482 secs 2024-07-26T13:38:46.987 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17871.084482 secs 2024-07-26T13:38:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17871.084482 secs 2024-07-26T13:38:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:47 smithi044 ceph-mon[32404]: pgmap v9868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:47 smithi116 ceph-mon[37097]: pgmap v9868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:47 smithi171 ceph-mon[37789]: pgmap v9868: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:50 smithi044 ceph-mon[32404]: pgmap v9869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:50 smithi116 ceph-mon[37097]: pgmap v9869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:50 smithi171 ceph-mon[37789]: pgmap v9869: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17876.084665 secs 2024-07-26T13:38:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17876.084665 secs 2024-07-26T13:38:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17876.084665 secs 2024-07-26T13:38:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:52 smithi044 ceph-mon[32404]: pgmap v9870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:52 smithi116 ceph-mon[37097]: pgmap v9870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:52 smithi171 ceph-mon[37789]: pgmap v9870: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:54.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:54 smithi044 ceph-mon[32404]: pgmap v9871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:54 smithi116 ceph-mon[37097]: pgmap v9871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:54 smithi171 ceph-mon[37789]: pgmap v9871: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:38:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:56 smithi044 ceph-mon[32404]: pgmap v9872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17881.084811 secs 2024-07-26T13:38:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:56 smithi116 ceph-mon[37097]: pgmap v9872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17881.084811 secs 2024-07-26T13:38:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:56 smithi171 ceph-mon[37789]: pgmap v9872: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17881.084811 secs 2024-07-26T13:38:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:38:58 smithi116 ceph-mon[37097]: pgmap v9873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:38:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:38:58 smithi171 ceph-mon[37789]: pgmap v9873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:38:59.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:38:58 smithi044 ceph-mon[32404]: pgmap v9873: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:00 smithi171 ceph-mon[37789]: pgmap v9874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:00.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:00 smithi044 ceph-mon[32404]: pgmap v9874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:01.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:01.003+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:39:01.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:01.003+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:01.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:01.003+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:39:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:00 smithi116 ceph-mon[37097]: pgmap v9874: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17886.084972 secs 2024-07-26T13:39:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:01 smithi171 ceph-mon[37789]: pgmap v9875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:02.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17886.084972 secs 2024-07-26T13:39:02.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:01 smithi044 ceph-mon[32404]: pgmap v9875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17886.084972 secs 2024-07-26T13:39:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:01 smithi116 ceph-mon[37097]: pgmap v9875: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:04 smithi044 ceph-mon[32404]: pgmap v9876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:04.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:04 smithi116 ceph-mon[37097]: pgmap v9876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:04 smithi171 ceph-mon[37789]: pgmap v9876: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:06 smithi044 ceph-mon[32404]: pgmap v9877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17891.085127 secs 2024-07-26T13:39:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:06 smithi116 ceph-mon[37097]: pgmap v9877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17891.085127 secs 2024-07-26T13:39:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:06 smithi171 ceph-mon[37789]: pgmap v9877: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17891.085127 secs 2024-07-26T13:39:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:08 smithi044 ceph-mon[32404]: pgmap v9878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:08 smithi116 ceph-mon[37097]: pgmap v9878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:08 smithi171 ceph-mon[37789]: pgmap v9878: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:10 smithi044 ceph-mon[32404]: pgmap v9879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:10.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:10 smithi116 ceph-mon[37097]: pgmap v9879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:10 smithi171 ceph-mon[37789]: pgmap v9879: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17896.085296 secs 2024-07-26T13:39:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17896.085296 secs 2024-07-26T13:39:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17896.085296 secs 2024-07-26T13:39:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:12 smithi171 ceph-mon[37789]: pgmap v9880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:12 smithi044 ceph-mon[32404]: pgmap v9880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:12 smithi116 ceph-mon[37097]: pgmap v9880: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:13 smithi171 ceph-mon[37789]: pgmap v9881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:13 smithi044 ceph-mon[32404]: pgmap v9881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:13 smithi116 ceph-mon[37097]: pgmap v9881: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:16.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:16.009+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:39:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:16.009+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:39:16.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:16.009+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:16 smithi044 ceph-mon[32404]: pgmap v9882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17901.085441 secs 2024-07-26T13:39:16.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:16 smithi116 ceph-mon[37097]: pgmap v9882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17901.085441 secs 2024-07-26T13:39:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:16 smithi171 ceph-mon[37789]: pgmap v9882: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17901.085441 secs 2024-07-26T13:39:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:18 smithi044 ceph-mon[32404]: pgmap v9883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:18 smithi116 ceph-mon[37097]: pgmap v9883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:18 smithi171 ceph-mon[37789]: pgmap v9883: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:20 smithi044 ceph-mon[32404]: pgmap v9884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:20 smithi116 ceph-mon[37097]: pgmap v9884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:20 smithi171 ceph-mon[37789]: pgmap v9884: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17906.085585 secs 2024-07-26T13:39:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17906.085585 secs 2024-07-26T13:39:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17906.085585 secs 2024-07-26T13:39:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:22 smithi171 ceph-mon[37789]: pgmap v9885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:22 smithi044 ceph-mon[32404]: pgmap v9885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:22 smithi116 ceph-mon[37097]: pgmap v9885: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:23 smithi171 ceph-mon[37789]: pgmap v9886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:23 smithi044 ceph-mon[32404]: pgmap v9886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:24.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:23 smithi116 ceph-mon[37097]: pgmap v9886: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:26 smithi044 ceph-mon[32404]: pgmap v9887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17911.085743 secs 2024-07-26T13:39:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:26 smithi116 ceph-mon[37097]: pgmap v9887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17911.085743 secs 2024-07-26T13:39:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:26 smithi171 ceph-mon[37789]: pgmap v9887: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17911.085743 secs 2024-07-26T13:39:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:28 smithi044 ceph-mon[32404]: pgmap v9888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:28 smithi116 ceph-mon[37097]: pgmap v9888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:28 smithi171 ceph-mon[37789]: pgmap v9888: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:30 smithi116 ceph-mon[37097]: pgmap v9889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:30 smithi171 ceph-mon[37789]: pgmap v9889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:30.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:30 smithi044 ceph-mon[32404]: pgmap v9889: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:31.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:31.010+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:39:31.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:31.010+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:39:31.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:31.010+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17916.085906 secs 2024-07-26T13:39:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17916.085906 secs 2024-07-26T13:39:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17916.085906 secs 2024-07-26T13:39:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:32 smithi171 ceph-mon[37789]: pgmap v9890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:32 smithi044 ceph-mon[32404]: pgmap v9890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:32 smithi116 ceph-mon[37097]: pgmap v9890: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:33 smithi171 ceph-mon[37789]: pgmap v9891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:33 smithi044 ceph-mon[32404]: pgmap v9891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:33 smithi116 ceph-mon[37097]: pgmap v9891: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:36 smithi044 ceph-mon[32404]: pgmap v9892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17921.085997 secs 2024-07-26T13:39:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:36 smithi116 ceph-mon[37097]: pgmap v9892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17921.085997 secs 2024-07-26T13:39:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:36 smithi171 ceph-mon[37789]: pgmap v9892: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17921.085997 secs 2024-07-26T13:39:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:38 smithi116 ceph-mon[37097]: pgmap v9893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:38 smithi171 ceph-mon[37789]: pgmap v9893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:38 smithi044 ceph-mon[32404]: pgmap v9893: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:39:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:40 smithi171 ceph-mon[37789]: pgmap v9894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:40 smithi044 ceph-mon[32404]: pgmap v9894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:40 smithi116 ceph-mon[37097]: pgmap v9894: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:39:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17926.086159 secs 2024-07-26T13:39:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17926.086159 secs 2024-07-26T13:39:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17926.086159 secs 2024-07-26T13:39:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:42 smithi171 ceph-mon[37789]: pgmap v9895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:42 smithi044 ceph-mon[32404]: pgmap v9895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:42 smithi116 ceph-mon[37097]: pgmap v9895: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:43 smithi171 ceph-mon[37789]: pgmap v9896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:43 smithi044 ceph-mon[32404]: pgmap v9896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:43 smithi116 ceph-mon[37097]: pgmap v9896: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:46.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:46.010+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:39:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:46.010+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:39:46.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:39:46.010+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:46 smithi116 ceph-mon[37097]: pgmap v9897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:46 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17931.086281 secs 2024-07-26T13:39:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:46 smithi171 ceph-mon[37789]: pgmap v9897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:46 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17931.086281 secs 2024-07-26T13:39:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:46 smithi044 ceph-mon[32404]: pgmap v9897: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:46 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17931.086281 secs 2024-07-26T13:39:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:47 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:47 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:47.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:47 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:39:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:47 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:47 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:47 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:39:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:47 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:47 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:47 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:39:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:48 smithi171 ceph-mon[37789]: pgmap v9898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:48.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:48 smithi171 ceph-mon[37789]: pgmap v9899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:48 smithi044 ceph-mon[32404]: pgmap v9898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:48 smithi044 ceph-mon[32404]: pgmap v9899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:48 smithi116 ceph-mon[37097]: pgmap v9898: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:48 smithi116 ceph-mon[37097]: pgmap v9899: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:49 smithi171 ceph-mon[37789]: pgmap v9900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:49 smithi044 ceph-mon[32404]: pgmap v9900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:49 smithi116 ceph-mon[37097]: pgmap v9900: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:51 smithi171 ceph-mon[37789]: pgmap v9901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17936.086428 secs 2024-07-26T13:39:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:51 smithi044 ceph-mon[32404]: pgmap v9901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17936.086428 secs 2024-07-26T13:39:52.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:51 smithi116 ceph-mon[37097]: pgmap v9901: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17936.086428 secs 2024-07-26T13:39:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:53 smithi171 ceph-mon[37789]: pgmap v9902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:39:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:53 smithi044 ceph-mon[32404]: pgmap v9902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:39:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:53 smithi116 ceph-mon[37097]: pgmap v9902: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:39:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:55 smithi171 ceph-mon[37789]: pgmap v9903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:55 smithi044 ceph-mon[32404]: pgmap v9903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:56.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:55 smithi116 ceph-mon[37097]: pgmap v9903: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17941.086569 secs 2024-07-26T13:39:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17941.086569 secs 2024-07-26T13:39:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17941.086569 secs 2024-07-26T13:39:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:57 smithi171 ceph-mon[37789]: pgmap v9904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:39:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:57 smithi044 ceph-mon[32404]: pgmap v9904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:39:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:57 smithi116 ceph-mon[37097]: pgmap v9904: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:00.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:39:59 smithi044 ceph-mon[32404]: pgmap v9905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:39:59 smithi171 ceph-mon[37789]: pgmap v9905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:39:59 smithi116 ceph-mon[37097]: pgmap v9905: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:01.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:01.012+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:40:01.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:01.012+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:40:01.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:01.012+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:02.081 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:01 smithi044 ceph-mon[32404]: pgmap v9906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:02.082 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17946.086730 secs 2024-07-26T13:40:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:01 smithi171 ceph-mon[37789]: pgmap v9906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17946.086730 secs 2024-07-26T13:40:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:01 smithi116 ceph-mon[37097]: pgmap v9906: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17946.086730 secs 2024-07-26T13:40:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:03 smithi171 ceph-mon[37789]: pgmap v9907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:03 smithi044 ceph-mon[32404]: pgmap v9907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:03 smithi116 ceph-mon[37097]: pgmap v9907: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:06.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:05 smithi171 ceph-mon[37789]: pgmap v9908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:05 smithi044 ceph-mon[32404]: pgmap v9908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:05 smithi116 ceph-mon[37097]: pgmap v9908: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17951.086893 secs 2024-07-26T13:40:07.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17951.086893 secs 2024-07-26T13:40:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17951.086893 secs 2024-07-26T13:40:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:07 smithi044 ceph-mon[32404]: pgmap v9909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:07 smithi116 ceph-mon[37097]: pgmap v9909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:08.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:07 smithi171 ceph-mon[37789]: pgmap v9909: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:09 smithi044 ceph-mon[32404]: pgmap v9910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:09 smithi116 ceph-mon[37097]: pgmap v9910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:09 smithi171 ceph-mon[37789]: pgmap v9910: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:11 smithi044 ceph-mon[32404]: pgmap v9911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17956.087053 secs 2024-07-26T13:40:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:11 smithi116 ceph-mon[37097]: pgmap v9911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17956.087053 secs 2024-07-26T13:40:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:11 smithi171 ceph-mon[37789]: pgmap v9911: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:12.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17956.087053 secs 2024-07-26T13:40:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:13 smithi044 ceph-mon[32404]: pgmap v9912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:13 smithi116 ceph-mon[37097]: pgmap v9912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:13 smithi171 ceph-mon[37789]: pgmap v9912: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:16.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:16.018+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:40:16.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:16.018+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:40:16.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:16.018+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:16.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:15 smithi171 ceph-mon[37789]: pgmap v9913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:15 smithi044 ceph-mon[32404]: pgmap v9913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:16.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:15 smithi116 ceph-mon[37097]: pgmap v9913: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17961.087208 secs 2024-07-26T13:40:17.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17961.087208 secs 2024-07-26T13:40:17.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17961.087208 secs 2024-07-26T13:40:18.168 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:17 smithi044 ceph-mon[32404]: pgmap v9914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:17 smithi116 ceph-mon[37097]: pgmap v9914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:17 smithi171 ceph-mon[37789]: pgmap v9914: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:20.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:19 smithi044 ceph-mon[32404]: pgmap v9915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:19 smithi116 ceph-mon[37097]: pgmap v9915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:19 smithi171 ceph-mon[37789]: pgmap v9915: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:20 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17966.087303 secs 2024-07-26T13:40:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:20 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17966.087303 secs 2024-07-26T13:40:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:20 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17966.087303 secs 2024-07-26T13:40:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:21 smithi044 ceph-mon[32404]: pgmap v9916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:21 smithi116 ceph-mon[37097]: pgmap v9916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:22.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:21 smithi171 ceph-mon[37789]: pgmap v9916: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:23 smithi044 ceph-mon[32404]: pgmap v9917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:23 smithi116 ceph-mon[37097]: pgmap v9917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:23 smithi171 ceph-mon[37789]: pgmap v9917: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:25 smithi044 ceph-mon[32404]: pgmap v9918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:26.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:25 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17971.087460 secs 2024-07-26T13:40:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:25 smithi116 ceph-mon[37097]: pgmap v9918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:25 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17971.087460 secs 2024-07-26T13:40:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:25 smithi171 ceph-mon[37789]: pgmap v9918: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:26.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:25 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17971.087460 secs 2024-07-26T13:40:28.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:27 smithi044 ceph-mon[32404]: pgmap v9919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:27 smithi116 ceph-mon[37097]: pgmap v9919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:27 smithi171 ceph-mon[37789]: pgmap v9919: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:29 smithi044 ceph-mon[32404]: pgmap v9920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:29 smithi116 ceph-mon[37097]: pgmap v9920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:29 smithi171 ceph-mon[37789]: pgmap v9920: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:31.019+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:40:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:31.019+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:40:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:31.019+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:30 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17976.087658 secs 2024-07-26T13:40:31.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:30 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17976.087658 secs 2024-07-26T13:40:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:30 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17976.087658 secs 2024-07-26T13:40:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:31 smithi044 ceph-mon[32404]: pgmap v9921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:31 smithi116 ceph-mon[37097]: pgmap v9921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:31 smithi171 ceph-mon[37789]: pgmap v9921: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:33 smithi044 ceph-mon[32404]: pgmap v9922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:33 smithi116 ceph-mon[37097]: pgmap v9922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:33 smithi171 ceph-mon[37789]: pgmap v9922: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:35 smithi044 ceph-mon[32404]: pgmap v9923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:35 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17981.087772 secs 2024-07-26T13:40:36.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:35 smithi116 ceph-mon[37097]: pgmap v9923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:35 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17981.087772 secs 2024-07-26T13:40:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:35 smithi171 ceph-mon[37789]: pgmap v9923: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:40:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:35 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17981.087772 secs 2024-07-26T13:40:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:37 smithi044 ceph-mon[32404]: pgmap v9924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:37 smithi116 ceph-mon[37097]: pgmap v9924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:37 smithi171 ceph-mon[37789]: pgmap v9924: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:39 smithi044 ceph-mon[32404]: pgmap v9925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:39 smithi116 ceph-mon[37097]: pgmap v9925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:39 smithi171 ceph-mon[37789]: pgmap v9925: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:40 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17986.087918 secs 2024-07-26T13:40:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:40 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17986.087918 secs 2024-07-26T13:40:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:40 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17986.087918 secs 2024-07-26T13:40:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:41 smithi044 ceph-mon[32404]: pgmap v9926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:40:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:41 smithi116 ceph-mon[37097]: pgmap v9926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:40:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:41 smithi171 ceph-mon[37789]: pgmap v9926: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:40:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:43 smithi044 ceph-mon[32404]: pgmap v9927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:43 smithi116 ceph-mon[37097]: pgmap v9927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:43 smithi171 ceph-mon[37789]: pgmap v9927: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:46.025+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:40:46.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:46.025+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:40:46.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:40:46.025+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:45 smithi044 ceph-mon[32404]: pgmap v9928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:40:46.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:45 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17991.088058 secs 2024-07-26T13:40:46.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:45 smithi116 ceph-mon[37097]: pgmap v9928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:40:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:45 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17991.088058 secs 2024-07-26T13:40:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:45 smithi171 ceph-mon[37789]: pgmap v9928: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:40:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:45 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17991.088058 secs 2024-07-26T13:40:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:47 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:47 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:47 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:48.130 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:48 smithi116 ceph-mon[37097]: pgmap v9929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:48 smithi044 ceph-mon[32404]: pgmap v9929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:48 smithi171 ceph-mon[37789]: pgmap v9929: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:40:50.218 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:50 smithi116 ceph-mon[37097]: pgmap v9930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:50 smithi044 ceph-mon[32404]: pgmap v9930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:50 smithi171 ceph-mon[37789]: pgmap v9930: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:51 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 17996.088164 secs 2024-07-26T13:40:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:51 smithi171 ceph-mon[37789]: pgmap v9931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:40:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:51 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 17996.088164 secs 2024-07-26T13:40:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:51 smithi044 ceph-mon[32404]: pgmap v9931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:40:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:51 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 17996.088164 secs 2024-07-26T13:40:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:51 smithi116 ceph-mon[37097]: pgmap v9931: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:40:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:53 smithi171 ceph-mon[37789]: pgmap v9932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:53 smithi044 ceph-mon[32404]: pgmap v9932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:53 smithi116 ceph-mon[37097]: pgmap v9932: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:55.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:55.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:55.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: pgmap v9933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:55.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: pgmap v9934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:40:55.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:54 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: pgmap v9933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: pgmap v9934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:40:55.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:54 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:40:55.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:40:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:40:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: pgmap v9933: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: pgmap v9934: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:40:55.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:54 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18001.088296 secs 2024-07-26T13:40:56.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18001.088296 secs 2024-07-26T13:40:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18001.088296 secs 2024-07-26T13:40:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:56 smithi044 ceph-mon[32404]: pgmap v9935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:57.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:56 smithi116 ceph-mon[37097]: pgmap v9935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:56 smithi171 ceph-mon[37789]: pgmap v9935: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:40:59 smithi171 ceph-mon[37789]: pgmap v9936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:40:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:40:59 smithi044 ceph-mon[32404]: pgmap v9936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:40:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:40:59 smithi116 ceph-mon[37097]: pgmap v9936: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:41:01.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:01.027+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:41:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:01.027+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:41:01.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:01.027+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:01.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:01 smithi044 ceph-mon[32404]: pgmap v9937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:01.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18006.088455 secs 2024-07-26T13:41:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:01 smithi171 ceph-mon[37789]: pgmap v9937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18006.088455 secs 2024-07-26T13:41:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:01 smithi116 ceph-mon[37097]: pgmap v9937: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18006.088455 secs 2024-07-26T13:41:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:03 smithi171 ceph-mon[37789]: pgmap v9938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:03 smithi044 ceph-mon[32404]: pgmap v9938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:03 smithi116 ceph-mon[37097]: pgmap v9938: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:05.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:05 smithi171 ceph-mon[37789]: pgmap v9939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:41:05.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:05 smithi044 ceph-mon[32404]: pgmap v9939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:41:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:05 smithi116 ceph-mon[37097]: pgmap v9939: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:41:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18011.088610 secs 2024-07-26T13:41:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18011.088610 secs 2024-07-26T13:41:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18011.088610 secs 2024-07-26T13:41:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:07 smithi044 ceph-mon[32404]: pgmap v9940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:41:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:07 smithi116 ceph-mon[37097]: pgmap v9940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:41:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:07 smithi171 ceph-mon[37789]: pgmap v9940: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:41:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:09 smithi044 ceph-mon[32404]: pgmap v9941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:41:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:09 smithi116 ceph-mon[37097]: pgmap v9941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:41:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:09 smithi171 ceph-mon[37789]: pgmap v9941: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:41:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:11 smithi044 ceph-mon[32404]: pgmap v9942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18016.088769 secs 2024-07-26T13:41:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:11 smithi116 ceph-mon[37097]: pgmap v9942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18016.088769 secs 2024-07-26T13:41:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:11 smithi171 ceph-mon[37789]: pgmap v9942: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18016.088769 secs 2024-07-26T13:41:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:13 smithi044 ceph-mon[32404]: pgmap v9943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:13 smithi116 ceph-mon[37097]: pgmap v9943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:13 smithi171 ceph-mon[37789]: pgmap v9943: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:14.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58050000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58050000 comp rx=0 tx=0).stop 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 msgr2=0x7fcf540c4810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:41:14.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.776+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:41:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.777+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:41:14.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:14.779+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf580318f0 tx=0x7fcf58087040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:41:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:15 smithi044 ceph-mon[32404]: pgmap v9944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:15 smithi116 ceph-mon[37097]: pgmap v9944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:15 smithi171 ceph-mon[37789]: pgmap v9944: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:16.028+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:41:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:16.029+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:41:16.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:16.029+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:16.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18021.088868 secs 2024-07-26T13:41:16.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18021.088868 secs 2024-07-26T13:41:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18021.088868 secs 2024-07-26T13:41:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:17 smithi044 ceph-mon[32404]: pgmap v9945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:17 smithi116 ceph-mon[37097]: pgmap v9945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:17 smithi171 ceph-mon[37789]: pgmap v9945: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:17.938+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:41:17.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:17.938+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:41:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:17.938+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:41:17.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:17.939+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:41:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:19 smithi044 ceph-mon[32404]: pgmap v9946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:19 smithi116 ceph-mon[37097]: pgmap v9946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:19 smithi171 ceph-mon[37789]: pgmap v9946: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:21 smithi044 ceph-mon[32404]: pgmap v9947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18026.089035 secs 2024-07-26T13:41:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:21 smithi116 ceph-mon[37097]: pgmap v9947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18026.089035 secs 2024-07-26T13:41:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:21 smithi171 ceph-mon[37789]: pgmap v9947: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18026.089035 secs 2024-07-26T13:41:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:23 smithi044 ceph-mon[32404]: pgmap v9948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:23 smithi116 ceph-mon[37097]: pgmap v9948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:23 smithi171 ceph-mon[37789]: pgmap v9948: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:25 smithi044 ceph-mon[32404]: pgmap v9949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:25 smithi116 ceph-mon[37097]: pgmap v9949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:25.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:25 smithi171 ceph-mon[37789]: pgmap v9949: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18031.089189 secs 2024-07-26T13:41:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18031.089189 secs 2024-07-26T13:41:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18031.089189 secs 2024-07-26T13:41:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:27 smithi044 ceph-mon[32404]: pgmap v9950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:27.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:27 smithi116 ceph-mon[37097]: pgmap v9950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:27 smithi171 ceph-mon[37789]: pgmap v9950: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:29 smithi044 ceph-mon[32404]: pgmap v9951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:29 smithi116 ceph-mon[37097]: pgmap v9951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:29 smithi171 ceph-mon[37789]: pgmap v9951: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:31.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:31.030+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:41:31.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:31.030+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:31.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:31.030+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:41:31.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:31 smithi044 ceph-mon[32404]: pgmap v9952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:31.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18036.089337 secs 2024-07-26T13:41:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:31 smithi116 ceph-mon[37097]: pgmap v9952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18036.089337 secs 2024-07-26T13:41:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:31 smithi171 ceph-mon[37789]: pgmap v9952: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18036.089337 secs 2024-07-26T13:41:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:33 smithi044 ceph-mon[32404]: pgmap v9953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:33 smithi116 ceph-mon[37097]: pgmap v9953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:33 smithi171 ceph-mon[37789]: pgmap v9953: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:35.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:35 smithi044 ceph-mon[32404]: pgmap v9954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:35 smithi116 ceph-mon[37097]: pgmap v9954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:35 smithi171 ceph-mon[37789]: pgmap v9954: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18041.089492 secs 2024-07-26T13:41:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18041.089492 secs 2024-07-26T13:41:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18041.089492 secs 2024-07-26T13:41:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:37 smithi044 ceph-mon[32404]: pgmap v9955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:37 smithi116 ceph-mon[37097]: pgmap v9955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:37 smithi171 ceph-mon[37789]: pgmap v9955: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:39 smithi044 ceph-mon[32404]: pgmap v9956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:39 smithi116 ceph-mon[37097]: pgmap v9956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:39 smithi171 ceph-mon[37789]: pgmap v9956: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:41 smithi044 ceph-mon[32404]: pgmap v9957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18046.089676 secs 2024-07-26T13:41:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:41 smithi116 ceph-mon[37097]: pgmap v9957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18046.089676 secs 2024-07-26T13:41:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:41 smithi171 ceph-mon[37789]: pgmap v9957: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18046.089676 secs 2024-07-26T13:41:43.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:43 smithi044 ceph-mon[32404]: pgmap v9958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:43 smithi116 ceph-mon[37097]: pgmap v9958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:43 smithi171 ceph-mon[37789]: pgmap v9958: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:45 smithi044 ceph-mon[32404]: pgmap v9959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:45 smithi116 ceph-mon[37097]: pgmap v9959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:45 smithi171 ceph-mon[37789]: pgmap v9959: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:46.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:46.035+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:41:46.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:46.035+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:41:46.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:41:46.035+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:46.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18051.089814 secs 2024-07-26T13:41:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18051.089814 secs 2024-07-26T13:41:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18051.089814 secs 2024-07-26T13:41:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:47 smithi044 ceph-mon[32404]: pgmap v9960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:47.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:47 smithi116 ceph-mon[37097]: pgmap v9960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:47 smithi171 ceph-mon[37789]: pgmap v9960: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:49 smithi044 ceph-mon[32404]: pgmap v9961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:49 smithi116 ceph-mon[37097]: pgmap v9961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:49 smithi171 ceph-mon[37789]: pgmap v9961: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:51 smithi044 ceph-mon[32404]: pgmap v9962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18056.089976 secs 2024-07-26T13:41:51.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:51 smithi116 ceph-mon[37097]: pgmap v9962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18056.089976 secs 2024-07-26T13:41:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:51 smithi171 ceph-mon[37789]: pgmap v9962: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18056.089976 secs 2024-07-26T13:41:53.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:53 smithi044 ceph-mon[32404]: pgmap v9963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:53 smithi116 ceph-mon[37097]: pgmap v9963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:53 smithi171 ceph-mon[37789]: pgmap v9963: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:54 smithi044 ceph-mon[32404]: pgmap v9964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:54 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:54 smithi116 ceph-mon[37097]: pgmap v9964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:54 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:54 smithi171 ceph-mon[37789]: pgmap v9964: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:54 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:41:55.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:55 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:41:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:41:56.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:55 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:41:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:41:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:55 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:41:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18061.090118 secs 2024-07-26T13:41:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:56 smithi171 ceph-mon[37789]: pgmap v9965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18061.090118 secs 2024-07-26T13:41:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:56 smithi044 ceph-mon[32404]: pgmap v9965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18061.090118 secs 2024-07-26T13:41:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:56 smithi116 ceph-mon[37097]: pgmap v9965: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:41:59.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:59.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:59.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: pgmap v9966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:59.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: pgmap v9967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:41:59.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:41:58 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: pgmap v9966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: pgmap v9967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:41:59.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:41:58 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:59.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:59.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: pgmap v9966: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:41:59.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: pgmap v9967: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:41:59.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:41:58 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:42:01.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:01.036+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:42:01.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:01.036+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:42:01.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:01.036+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:00 smithi044 ceph-mon[32404]: pgmap v9968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:01.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:00 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18066.090293 secs 2024-07-26T13:42:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:00 smithi116 ceph-mon[37097]: pgmap v9968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:00 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18066.090293 secs 2024-07-26T13:42:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:00 smithi171 ceph-mon[37789]: pgmap v9968: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:01.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:00 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18066.090293 secs 2024-07-26T13:42:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:02 smithi044 ceph-mon[32404]: pgmap v9969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:02 smithi116 ceph-mon[37097]: pgmap v9969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:02 smithi171 ceph-mon[37789]: pgmap v9969: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:04 smithi044 ceph-mon[32404]: pgmap v9970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:04 smithi116 ceph-mon[37097]: pgmap v9970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:04 smithi171 ceph-mon[37789]: pgmap v9970: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:05 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18071.090425 secs 2024-07-26T13:42:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:05 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18071.090425 secs 2024-07-26T13:42:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:05 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18071.090425 secs 2024-07-26T13:42:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:06 smithi044 ceph-mon[32404]: pgmap v9971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:06 smithi116 ceph-mon[37097]: pgmap v9971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:06 smithi171 ceph-mon[37789]: pgmap v9971: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:42:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:08 smithi044 ceph-mon[32404]: pgmap v9972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:08 smithi116 ceph-mon[37097]: pgmap v9972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:08 smithi171 ceph-mon[37789]: pgmap v9972: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:10 smithi044 ceph-mon[32404]: pgmap v9973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:10 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18076.090575 secs 2024-07-26T13:42:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:10 smithi116 ceph-mon[37097]: pgmap v9973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:10 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18076.090575 secs 2024-07-26T13:42:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:10 smithi171 ceph-mon[37789]: pgmap v9973: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:10 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18076.090575 secs 2024-07-26T13:42:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:12 smithi044 ceph-mon[32404]: pgmap v9974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:13.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:12 smithi116 ceph-mon[37097]: pgmap v9974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:12 smithi171 ceph-mon[37789]: pgmap v9974: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:15.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:14 smithi044 ceph-mon[32404]: pgmap v9975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:14 smithi116 ceph-mon[37097]: pgmap v9975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:14 smithi171 ceph-mon[37789]: pgmap v9975: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:16.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:16.042+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:42:16.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:16.042+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:42:16.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:16.042+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:15 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18081.090712 secs 2024-07-26T13:42:16.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:15 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18081.090712 secs 2024-07-26T13:42:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:15 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18081.090712 secs 2024-07-26T13:42:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:16 smithi044 ceph-mon[32404]: pgmap v9976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:16 smithi116 ceph-mon[37097]: pgmap v9976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:16 smithi171 ceph-mon[37789]: pgmap v9976: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:19 smithi044 ceph-mon[32404]: pgmap v9977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:19 smithi116 ceph-mon[37097]: pgmap v9977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:19 smithi171 ceph-mon[37789]: pgmap v9977: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:21 smithi044 ceph-mon[32404]: pgmap v9978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:21.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18086.090876 secs 2024-07-26T13:42:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:21 smithi116 ceph-mon[37097]: pgmap v9978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18086.090876 secs 2024-07-26T13:42:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:21 smithi171 ceph-mon[37789]: pgmap v9978: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:21.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18086.090876 secs 2024-07-26T13:42:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:23 smithi044 ceph-mon[32404]: pgmap v9979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:23 smithi116 ceph-mon[37097]: pgmap v9979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:23 smithi171 ceph-mon[37789]: pgmap v9979: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:25 smithi171 ceph-mon[37789]: pgmap v9980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:25 smithi044 ceph-mon[32404]: pgmap v9980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:25 smithi116 ceph-mon[37097]: pgmap v9980: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18091.091026 secs 2024-07-26T13:42:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18091.091026 secs 2024-07-26T13:42:26.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18091.091026 secs 2024-07-26T13:42:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:27 smithi171 ceph-mon[37789]: pgmap v9981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:27 smithi044 ceph-mon[32404]: pgmap v9981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:27 smithi116 ceph-mon[37097]: pgmap v9981: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:29 smithi171 ceph-mon[37789]: pgmap v9982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:29 smithi044 ceph-mon[32404]: pgmap v9982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:29 smithi116 ceph-mon[37097]: pgmap v9982: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:31.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:31.043+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:42:31.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:31.044+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:31.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:31.044+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:42:31.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:31 smithi044 ceph-mon[32404]: pgmap v9983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:31.319 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18096.091140 secs 2024-07-26T13:42:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:31 smithi171 ceph-mon[37789]: pgmap v9983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:31.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18096.091140 secs 2024-07-26T13:42:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:31 smithi116 ceph-mon[37097]: pgmap v9983: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18096.091140 secs 2024-07-26T13:42:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:33 smithi171 ceph-mon[37789]: pgmap v9984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:33 smithi044 ceph-mon[32404]: pgmap v9984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:33 smithi116 ceph-mon[37097]: pgmap v9984: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:35 smithi171 ceph-mon[37789]: pgmap v9985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:35 smithi044 ceph-mon[32404]: pgmap v9985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:35 smithi116 ceph-mon[37097]: pgmap v9985: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18101.091270 secs 2024-07-26T13:42:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18101.091270 secs 2024-07-26T13:42:36.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18101.091270 secs 2024-07-26T13:42:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:37 smithi171 ceph-mon[37789]: pgmap v9986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:37 smithi044 ceph-mon[32404]: pgmap v9986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:37.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:37 smithi116 ceph-mon[37097]: pgmap v9986: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:39 smithi171 ceph-mon[37789]: pgmap v9987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:39 smithi044 ceph-mon[32404]: pgmap v9987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:39.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:39 smithi116 ceph-mon[37097]: pgmap v9987: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:41 smithi044 ceph-mon[32404]: pgmap v9988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:41.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18106.091414 secs 2024-07-26T13:42:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:41 smithi116 ceph-mon[37097]: pgmap v9988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18106.091414 secs 2024-07-26T13:42:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:41 smithi171 ceph-mon[37789]: pgmap v9988: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18106.091414 secs 2024-07-26T13:42:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:43 smithi044 ceph-mon[32404]: pgmap v9989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:43.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:43 smithi116 ceph-mon[37097]: pgmap v9989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:43 smithi171 ceph-mon[37789]: pgmap v9989: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:45 smithi044 ceph-mon[32404]: pgmap v9990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:45 smithi116 ceph-mon[37097]: pgmap v9990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:45 smithi171 ceph-mon[37789]: pgmap v9990: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:46.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:46.045+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:42:46.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:46.045+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:42:46.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:42:46.045+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:46.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18111.091575 secs 2024-07-26T13:42:46.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18111.091575 secs 2024-07-26T13:42:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18111.091575 secs 2024-07-26T13:42:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:47 smithi044 ceph-mon[32404]: pgmap v9991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:47.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:47 smithi116 ceph-mon[37097]: pgmap v9991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:47 smithi171 ceph-mon[37789]: pgmap v9991: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:49 smithi044 ceph-mon[32404]: pgmap v9992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:49.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:49 smithi116 ceph-mon[37097]: pgmap v9992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:49 smithi171 ceph-mon[37789]: pgmap v9992: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:51 smithi044 ceph-mon[32404]: pgmap v9993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:51.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18116.091724 secs 2024-07-26T13:42:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:51 smithi116 ceph-mon[37097]: pgmap v9993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18116.091724 secs 2024-07-26T13:42:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:51 smithi171 ceph-mon[37789]: pgmap v9993: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:42:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18116.091724 secs 2024-07-26T13:42:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:53 smithi044 ceph-mon[32404]: pgmap v9994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:53 smithi116 ceph-mon[37097]: pgmap v9994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:53 smithi171 ceph-mon[37789]: pgmap v9994: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:55.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:54 smithi044 ceph-mon[32404]: pgmap v9995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:54 smithi116 ceph-mon[37097]: pgmap v9995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:54 smithi171 ceph-mon[37789]: pgmap v9995: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:56.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:55 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18121.091821 secs 2024-07-26T13:42:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:55 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18121.091821 secs 2024-07-26T13:42:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:55 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18121.091821 secs 2024-07-26T13:42:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:56 smithi044 ceph-mon[32404]: pgmap v9996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:56 smithi116 ceph-mon[37097]: pgmap v9996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:56 smithi171 ceph-mon[37789]: pgmap v9996: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:58 smithi044 ceph-mon[32404]: pgmap v9997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:58 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:58 smithi116 ceph-mon[37097]: pgmap v9997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:58 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:58 smithi171 ceph-mon[37789]: pgmap v9997: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:42:59.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:58 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:59 smithi044 ceph-mon[32404]: pgmap v9998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:43:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:59 smithi044 ceph-mon[32404]: pgmap v9999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:42:59 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:59 smithi116 ceph-mon[37097]: pgmap v9998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:43:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:59 smithi116 ceph-mon[37097]: pgmap v9999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:42:59 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:59 smithi171 ceph-mon[37789]: pgmap v9998: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:43:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:59 smithi171 ceph-mon[37789]: pgmap v9999: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:42:59 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:01.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:01.046+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:43:01.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:01.046+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:43:01.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:01.046+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18126.091957 secs 2024-07-26T13:43:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18126.091957 secs 2024-07-26T13:43:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18126.091957 secs 2024-07-26T13:43:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:02 smithi116 ceph-mon[37097]: pgmap v10000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:43:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:02 smithi171 ceph-mon[37789]: pgmap v10000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:43:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:02 smithi044 ceph-mon[32404]: pgmap v10000: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:43:04.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:04 smithi044 ceph-mon[32404]: pgmap v10001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:04.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:04 smithi116 ceph-mon[37097]: pgmap v10001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:04.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:04 smithi171 ceph-mon[37789]: pgmap v10001: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:06 smithi044 ceph-mon[32404]: pgmap v10002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18131.092114 secs 2024-07-26T13:43:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:06 smithi116 ceph-mon[37097]: pgmap v10002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18131.092114 secs 2024-07-26T13:43:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:06 smithi171 ceph-mon[37789]: pgmap v10002: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18131.092114 secs 2024-07-26T13:43:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:08 smithi044 ceph-mon[32404]: pgmap v10003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:08 smithi116 ceph-mon[37097]: pgmap v10003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:08 smithi171 ceph-mon[37789]: pgmap v10003: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:43:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:10 smithi044 ceph-mon[32404]: pgmap v10004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:43:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:10 smithi116 ceph-mon[37097]: pgmap v10004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:43:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:10 smithi171 ceph-mon[37789]: pgmap v10004: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:43:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18136.092273 secs 2024-07-26T13:43:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18136.092273 secs 2024-07-26T13:43:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18136.092273 secs 2024-07-26T13:43:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:12 smithi044 ceph-mon[32404]: pgmap v10005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:12 smithi116 ceph-mon[37097]: pgmap v10005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:12 smithi171 ceph-mon[37789]: pgmap v10005: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:14 smithi044 ceph-mon[32404]: pgmap v10006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:14 smithi116 ceph-mon[37097]: pgmap v10006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:14 smithi171 ceph-mon[37789]: pgmap v10006: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:16.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:16.052+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:43:16.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:16.052+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:43:16.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:16.052+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:16 smithi044 ceph-mon[32404]: pgmap v10007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18141.092381 secs 2024-07-26T13:43:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:16 smithi171 ceph-mon[37789]: pgmap v10007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18141.092381 secs 2024-07-26T13:43:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:16 smithi116 ceph-mon[37097]: pgmap v10007: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18141.092381 secs 2024-07-26T13:43:18.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:18 smithi171 ceph-mon[37789]: pgmap v10008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:18 smithi044 ceph-mon[32404]: pgmap v10008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:18 smithi116 ceph-mon[37097]: pgmap v10008: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:20 smithi171 ceph-mon[37789]: pgmap v10009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:20 smithi044 ceph-mon[32404]: pgmap v10009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:20 smithi116 ceph-mon[37097]: pgmap v10009: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18146.092548 secs 2024-07-26T13:43:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18146.092548 secs 2024-07-26T13:43:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18146.092548 secs 2024-07-26T13:43:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:22 smithi171 ceph-mon[37789]: pgmap v10010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:22 smithi044 ceph-mon[32404]: pgmap v10010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:22.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:22 smithi116 ceph-mon[37097]: pgmap v10010: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:24 smithi171 ceph-mon[37789]: pgmap v10011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:24 smithi044 ceph-mon[32404]: pgmap v10011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:24 smithi116 ceph-mon[37097]: pgmap v10011: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:26 smithi171 ceph-mon[37789]: pgmap v10012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18151.092713 secs 2024-07-26T13:43:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:26 smithi044 ceph-mon[32404]: pgmap v10012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18151.092713 secs 2024-07-26T13:43:26.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:26 smithi116 ceph-mon[37097]: pgmap v10012: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18151.092713 secs 2024-07-26T13:43:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:28 smithi171 ceph-mon[37789]: pgmap v10013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:28 smithi044 ceph-mon[32404]: pgmap v10013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:28.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:28 smithi116 ceph-mon[37097]: pgmap v10013: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:30 smithi171 ceph-mon[37789]: pgmap v10014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:30 smithi044 ceph-mon[32404]: pgmap v10014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:30 smithi116 ceph-mon[37097]: pgmap v10014: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:31.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:31.053+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:43:31.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:31.053+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:43:31.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:31.053+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18156.092809 secs 2024-07-26T13:43:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18156.092809 secs 2024-07-26T13:43:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18156.092809 secs 2024-07-26T13:43:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:32 smithi171 ceph-mon[37789]: pgmap v10015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:32 smithi044 ceph-mon[32404]: pgmap v10015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:32 smithi116 ceph-mon[37097]: pgmap v10015: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:34 smithi044 ceph-mon[32404]: pgmap v10016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:34 smithi116 ceph-mon[37097]: pgmap v10016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:34 smithi171 ceph-mon[37789]: pgmap v10016: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:36 smithi044 ceph-mon[32404]: pgmap v10017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18161.092905 secs 2024-07-26T13:43:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:36 smithi116 ceph-mon[37097]: pgmap v10017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18161.092905 secs 2024-07-26T13:43:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:36 smithi171 ceph-mon[37789]: pgmap v10017: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18161.092905 secs 2024-07-26T13:43:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:38 smithi044 ceph-mon[32404]: pgmap v10018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:38.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:38 smithi116 ceph-mon[37097]: pgmap v10018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:38 smithi171 ceph-mon[37789]: pgmap v10018: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:40.705 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:40 smithi044 ceph-mon[32404]: pgmap v10019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:40 smithi116 ceph-mon[37097]: pgmap v10019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:40 smithi171 ceph-mon[37789]: pgmap v10019: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18166.093058 secs 2024-07-26T13:43:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18166.093058 secs 2024-07-26T13:43:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18166.093058 secs 2024-07-26T13:43:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:42 smithi044 ceph-mon[32404]: pgmap v10020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:42 smithi116 ceph-mon[37097]: pgmap v10020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:42 smithi171 ceph-mon[37789]: pgmap v10020: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:44 smithi044 ceph-mon[32404]: pgmap v10021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:44 smithi116 ceph-mon[37097]: pgmap v10021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:44 smithi171 ceph-mon[37789]: pgmap v10021: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:46.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:46.057+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:43:46.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:46.057+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:43:46.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:43:46.057+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:46.638 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:46 smithi171 ceph-mon[37789]: pgmap v10022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:46.638 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18171.093213 secs 2024-07-26T13:43:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:46 smithi044 ceph-mon[32404]: pgmap v10022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18171.093213 secs 2024-07-26T13:43:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:46 smithi116 ceph-mon[37097]: pgmap v10022: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18171.093213 secs 2024-07-26T13:43:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:48 smithi044 ceph-mon[32404]: pgmap v10023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:48 smithi116 ceph-mon[37097]: pgmap v10023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:48 smithi171 ceph-mon[37789]: pgmap v10023: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:50 smithi044 ceph-mon[32404]: pgmap v10024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:50.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:50 smithi116 ceph-mon[37097]: pgmap v10024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:50 smithi171 ceph-mon[37789]: pgmap v10024: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18176.093356 secs 2024-07-26T13:43:51.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18176.093356 secs 2024-07-26T13:43:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18176.093356 secs 2024-07-26T13:43:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:52 smithi044 ceph-mon[32404]: pgmap v10025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:52 smithi116 ceph-mon[37097]: pgmap v10025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:52 smithi171 ceph-mon[37789]: pgmap v10025: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:54 smithi044 ceph-mon[32404]: pgmap v10026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:54 smithi116 ceph-mon[37097]: pgmap v10026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:54 smithi171 ceph-mon[37789]: pgmap v10026: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:56 smithi044 ceph-mon[32404]: pgmap v10027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18181.093495 secs 2024-07-26T13:43:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:56 smithi116 ceph-mon[37097]: pgmap v10027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18181.093495 secs 2024-07-26T13:43:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:56 smithi171 ceph-mon[37789]: pgmap v10027: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18181.093495 secs 2024-07-26T13:43:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:58 smithi044 ceph-mon[32404]: pgmap v10028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:58 smithi116 ceph-mon[37097]: pgmap v10028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:58 smithi171 ceph-mon[37789]: pgmap v10028: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:43:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:43:59 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:43:59 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:43:59 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:00 smithi044 ceph-mon[32404]: pgmap v10029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:00 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:00 smithi116 ceph-mon[37097]: pgmap v10029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:00 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:00 smithi171 ceph-mon[37789]: pgmap v10029: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:00 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:01.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:01.058+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:44:01.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:01.058+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:44:01.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:01.058+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18186.093608 secs 2024-07-26T13:44:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:01 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:01 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18186.093608 secs 2024-07-26T13:44:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:01 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:01 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18186.093608 secs 2024-07-26T13:44:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:01 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:01 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:02 smithi044 ceph-mon[32404]: pgmap v10030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:02 smithi044 ceph-mon[32404]: pgmap v10031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:02.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:02 smithi116 ceph-mon[37097]: pgmap v10030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:02 smithi116 ceph-mon[37097]: pgmap v10031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:02 smithi171 ceph-mon[37789]: pgmap v10030: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:02 smithi171 ceph-mon[37789]: pgmap v10031: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:04 smithi044 ceph-mon[32404]: pgmap v10032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:04 smithi116 ceph-mon[37097]: pgmap v10032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:04 smithi171 ceph-mon[37789]: pgmap v10032: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:06 smithi044 ceph-mon[32404]: pgmap v10033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18191.093741 secs 2024-07-26T13:44:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:06 smithi116 ceph-mon[37097]: pgmap v10033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18191.093741 secs 2024-07-26T13:44:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:06 smithi171 ceph-mon[37789]: pgmap v10033: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18191.093741 secs 2024-07-26T13:44:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:08 smithi044 ceph-mon[32404]: pgmap v10034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:08 smithi116 ceph-mon[37097]: pgmap v10034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:08 smithi171 ceph-mon[37789]: pgmap v10034: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:10 smithi044 ceph-mon[32404]: pgmap v10035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:10 smithi116 ceph-mon[37097]: pgmap v10035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:10 smithi171 ceph-mon[37789]: pgmap v10035: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:44:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18196.093894 secs 2024-07-26T13:44:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18196.093894 secs 2024-07-26T13:44:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18196.093894 secs 2024-07-26T13:44:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:12 smithi044 ceph-mon[32404]: pgmap v10036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:12 smithi116 ceph-mon[37097]: pgmap v10036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:12 smithi171 ceph-mon[37789]: pgmap v10036: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:14 smithi044 ceph-mon[32404]: pgmap v10037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:14 smithi116 ceph-mon[37097]: pgmap v10037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:14 smithi171 ceph-mon[37789]: pgmap v10037: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:16.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:16.063+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:44:16.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:16.063+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:44:16.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:16.063+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:16 smithi044 ceph-mon[32404]: pgmap v10038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18201.093997 secs 2024-07-26T13:44:16.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:16 smithi116 ceph-mon[37097]: pgmap v10038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18201.093997 secs 2024-07-26T13:44:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:16 smithi171 ceph-mon[37789]: pgmap v10038: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18201.093997 secs 2024-07-26T13:44:17.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:17.970+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:44:17.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:17.970+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:44:17.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:17.971+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:44:17.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:17.971+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:44:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:18 smithi044 ceph-mon[32404]: pgmap v10039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:18.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:18 smithi116 ceph-mon[37097]: pgmap v10039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:18 smithi171 ceph-mon[37789]: pgmap v10039: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:20 smithi116 ceph-mon[37097]: pgmap v10040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:20 smithi171 ceph-mon[37789]: pgmap v10040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:20 smithi044 ceph-mon[32404]: pgmap v10040: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18206.094150 secs 2024-07-26T13:44:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18206.094150 secs 2024-07-26T13:44:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18206.094150 secs 2024-07-26T13:44:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:22 smithi171 ceph-mon[37789]: pgmap v10041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:22 smithi044 ceph-mon[32404]: pgmap v10041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:22 smithi116 ceph-mon[37097]: pgmap v10041: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:24 smithi171 ceph-mon[37789]: pgmap v10042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:24 smithi044 ceph-mon[32404]: pgmap v10042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:24 smithi116 ceph-mon[37097]: pgmap v10042: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:26 smithi171 ceph-mon[37789]: pgmap v10043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18211.094292 secs 2024-07-26T13:44:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:26 smithi044 ceph-mon[32404]: pgmap v10043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18211.094292 secs 2024-07-26T13:44:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:26 smithi116 ceph-mon[37097]: pgmap v10043: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18211.094292 secs 2024-07-26T13:44:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:28 smithi171 ceph-mon[37789]: pgmap v10044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:28 smithi044 ceph-mon[32404]: pgmap v10044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:28 smithi116 ceph-mon[37097]: pgmap v10044: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:30 smithi171 ceph-mon[37789]: pgmap v10045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:30 smithi044 ceph-mon[32404]: pgmap v10045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:30 smithi116 ceph-mon[37097]: pgmap v10045: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:31.064+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:44:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:31.064+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:44:31.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:31.064+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18216.094440 secs 2024-07-26T13:44:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18216.094440 secs 2024-07-26T13:44:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18216.094440 secs 2024-07-26T13:44:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:32 smithi171 ceph-mon[37789]: pgmap v10046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:32 smithi044 ceph-mon[32404]: pgmap v10046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:32 smithi116 ceph-mon[37097]: pgmap v10046: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:34 smithi171 ceph-mon[37789]: pgmap v10047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:34 smithi044 ceph-mon[32404]: pgmap v10047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:35.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:34 smithi116 ceph-mon[37097]: pgmap v10047: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:36 smithi171 ceph-mon[37789]: pgmap v10048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18221.094603 secs 2024-07-26T13:44:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:36 smithi044 ceph-mon[32404]: pgmap v10048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18221.094603 secs 2024-07-26T13:44:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:36 smithi116 ceph-mon[37097]: pgmap v10048: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18221.094603 secs 2024-07-26T13:44:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:38 smithi171 ceph-mon[37789]: pgmap v10049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:38 smithi044 ceph-mon[32404]: pgmap v10049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:38 smithi116 ceph-mon[37097]: pgmap v10049: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:40 smithi044 ceph-mon[32404]: pgmap v10050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:40 smithi116 ceph-mon[37097]: pgmap v10050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:40 smithi171 ceph-mon[37789]: pgmap v10050: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18226.094771 secs 2024-07-26T13:44:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18226.094771 secs 2024-07-26T13:44:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18226.094771 secs 2024-07-26T13:44:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:42 smithi044 ceph-mon[32404]: pgmap v10051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:42 smithi116 ceph-mon[37097]: pgmap v10051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:42 smithi171 ceph-mon[37789]: pgmap v10051: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:43 smithi044 ceph-mon[32404]: pgmap v10052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:43 smithi116 ceph-mon[37097]: pgmap v10052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:43 smithi171 ceph-mon[37789]: pgmap v10052: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:46.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:46.070+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:44:46.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:46.070+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:46.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:44:46.070+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:44:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:46 smithi044 ceph-mon[32404]: pgmap v10053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18231.094941 secs 2024-07-26T13:44:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:46 smithi116 ceph-mon[37097]: pgmap v10053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18231.094941 secs 2024-07-26T13:44:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:46 smithi171 ceph-mon[37789]: pgmap v10053: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18231.094941 secs 2024-07-26T13:44:48.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:48 smithi044 ceph-mon[32404]: pgmap v10054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:48.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:48 smithi116 ceph-mon[37097]: pgmap v10054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:48 smithi171 ceph-mon[37789]: pgmap v10054: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:50.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:50 smithi044 ceph-mon[32404]: pgmap v10055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:50.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:50 smithi116 ceph-mon[37097]: pgmap v10055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:50 smithi171 ceph-mon[37789]: pgmap v10055: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18236.095104 secs 2024-07-26T13:44:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18236.095104 secs 2024-07-26T13:44:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18236.095104 secs 2024-07-26T13:44:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:52 smithi116 ceph-mon[37097]: pgmap v10056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:52 smithi171 ceph-mon[37789]: pgmap v10056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:52 smithi044 ceph-mon[32404]: pgmap v10056: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:54 smithi171 ceph-mon[37789]: pgmap v10057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:54 smithi044 ceph-mon[32404]: pgmap v10057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:54.788 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:54 smithi116 ceph-mon[37097]: pgmap v10057: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:56 smithi171 ceph-mon[37789]: pgmap v10058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18241.095260 secs 2024-07-26T13:44:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:56 smithi044 ceph-mon[32404]: pgmap v10058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18241.095260 secs 2024-07-26T13:44:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:56 smithi116 ceph-mon[37097]: pgmap v10058: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18241.095260 secs 2024-07-26T13:44:58.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:44:58 smithi171 ceph-mon[37789]: pgmap v10059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:44:58 smithi044 ceph-mon[32404]: pgmap v10059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:44:58 smithi116 ceph-mon[37097]: pgmap v10059: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:00 smithi171 ceph-mon[37789]: pgmap v10060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:00 smithi044 ceph-mon[32404]: pgmap v10060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:00 smithi116 ceph-mon[37097]: pgmap v10060: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:01.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:01.071+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:45:01.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:01.071+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:01.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:01.071+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:45:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18246.095363 secs 2024-07-26T13:45:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18246.095363 secs 2024-07-26T13:45:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18246.095363 secs 2024-07-26T13:45:02.585 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:02 smithi116 ceph-mon[37097]: pgmap v10061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:02.585 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:02 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:02.585 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.585 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.585 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.586 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:02 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:02 smithi171 ceph-mon[37789]: pgmap v10061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:02 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:02 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:02 smithi044 ceph-mon[32404]: pgmap v10061: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:02.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:02 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:02.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:02.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:02 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:04 smithi044 ceph-mon[32404]: pgmap v10062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:04 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:04 smithi116 ceph-mon[37097]: pgmap v10062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:04 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:04 smithi171 ceph-mon[37789]: pgmap v10062: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:04.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:04 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:06 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:06 smithi171 ceph-mon[37789]: pgmap v10063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18251.095489 secs 2024-07-26T13:45:06.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:06 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:06 smithi044 ceph-mon[32404]: pgmap v10063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18251.095489 secs 2024-07-26T13:45:06.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:06 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:06 smithi116 ceph-mon[37097]: pgmap v10063: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18251.095489 secs 2024-07-26T13:45:07.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:07 smithi044 ceph-mon[32404]: pgmap v10064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:07 smithi044 ceph-mon[32404]: pgmap v10065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:07 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:07 smithi116 ceph-mon[37097]: pgmap v10064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:07 smithi116 ceph-mon[37097]: pgmap v10065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:07 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:07 smithi171 ceph-mon[37789]: pgmap v10064: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:07 smithi171 ceph-mon[37789]: pgmap v10065: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:07 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:09.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:09 smithi044 ceph-mon[32404]: pgmap v10066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:09.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:09 smithi116 ceph-mon[37097]: pgmap v10066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:09 smithi171 ceph-mon[37789]: pgmap v10066: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:11 smithi044 ceph-mon[32404]: pgmap v10067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:11.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18256.095663 secs 2024-07-26T13:45:11.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:11 smithi116 ceph-mon[37097]: pgmap v10067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18256.095663 secs 2024-07-26T13:45:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:11 smithi171 ceph-mon[37789]: pgmap v10067: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18256.095663 secs 2024-07-26T13:45:13.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:13 smithi044 ceph-mon[32404]: pgmap v10068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:13 smithi116 ceph-mon[37097]: pgmap v10068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:13 smithi171 ceph-mon[37789]: pgmap v10068: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:15 smithi116 ceph-mon[37097]: pgmap v10069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:15 smithi171 ceph-mon[37789]: pgmap v10069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:15 smithi044 ceph-mon[32404]: pgmap v10069: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:45:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:16.076+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:45:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:16.076+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:16.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:16.076+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:45:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18261.095800 secs 2024-07-26T13:45:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18261.095800 secs 2024-07-26T13:45:16.696 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18261.095800 secs 2024-07-26T13:45:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:17 smithi171 ceph-mon[37789]: pgmap v10070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:17.616 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:17 smithi116 ceph-mon[37097]: pgmap v10070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:17 smithi044 ceph-mon[32404]: pgmap v10070: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:19 smithi171 ceph-mon[37789]: pgmap v10071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:19 smithi044 ceph-mon[32404]: pgmap v10071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:19 smithi116 ceph-mon[37097]: pgmap v10071: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:21 smithi171 ceph-mon[37789]: pgmap v10072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18266.095954 secs 2024-07-26T13:45:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:21 smithi044 ceph-mon[32404]: pgmap v10072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18266.095954 secs 2024-07-26T13:45:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:21 smithi116 ceph-mon[37097]: pgmap v10072: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18266.095954 secs 2024-07-26T13:45:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:23 smithi171 ceph-mon[37789]: pgmap v10073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:23 smithi044 ceph-mon[32404]: pgmap v10073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:23 smithi116 ceph-mon[37097]: pgmap v10073: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:25 smithi171 ceph-mon[37789]: pgmap v10074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:25 smithi044 ceph-mon[32404]: pgmap v10074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:25 smithi116 ceph-mon[37097]: pgmap v10074: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18271.096107 secs 2024-07-26T13:45:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18271.096107 secs 2024-07-26T13:45:26.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18271.096107 secs 2024-07-26T13:45:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:27 smithi116 ceph-mon[37097]: pgmap v10075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:27 smithi171 ceph-mon[37789]: pgmap v10075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:27 smithi044 ceph-mon[32404]: pgmap v10075: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:29 smithi171 ceph-mon[37789]: pgmap v10076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:29.761 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:29 smithi044 ceph-mon[32404]: pgmap v10076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:29 smithi116 ceph-mon[37097]: pgmap v10076: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:31.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:31.077+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:45:31.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:31.077+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:31.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:31.077+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:45:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:31 smithi171 ceph-mon[37789]: pgmap v10077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18276.096277 secs 2024-07-26T13:45:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:31 smithi044 ceph-mon[32404]: pgmap v10077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18276.096277 secs 2024-07-26T13:45:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:31 smithi116 ceph-mon[37097]: pgmap v10077: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18276.096277 secs 2024-07-26T13:45:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:33 smithi044 ceph-mon[32404]: pgmap v10078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:33.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:33 smithi116 ceph-mon[37097]: pgmap v10078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:33 smithi171 ceph-mon[37789]: pgmap v10078: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:35.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:35 smithi044 ceph-mon[32404]: pgmap v10079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:35.953 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:35 smithi171 ceph-mon[37789]: pgmap v10079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:35.954 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:35 smithi116 ceph-mon[37097]: pgmap v10079: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18281.096435 secs 2024-07-26T13:45:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18281.096435 secs 2024-07-26T13:45:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18281.096435 secs 2024-07-26T13:45:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:37 smithi044 ceph-mon[32404]: pgmap v10080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:37 smithi116 ceph-mon[37097]: pgmap v10080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:37 smithi171 ceph-mon[37789]: pgmap v10080: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:39 smithi044 ceph-mon[32404]: pgmap v10081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:39 smithi116 ceph-mon[37097]: pgmap v10081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:39 smithi171 ceph-mon[37789]: pgmap v10081: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:41 smithi044 ceph-mon[32404]: pgmap v10082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18286.096586 secs 2024-07-26T13:45:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:41 smithi116 ceph-mon[37097]: pgmap v10082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18286.096586 secs 2024-07-26T13:45:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:41 smithi171 ceph-mon[37789]: pgmap v10082: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18286.096586 secs 2024-07-26T13:45:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:43 smithi044 ceph-mon[32404]: pgmap v10083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:43 smithi116 ceph-mon[37097]: pgmap v10083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:43 smithi171 ceph-mon[37789]: pgmap v10083: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:45 smithi044 ceph-mon[32404]: pgmap v10084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:45 smithi116 ceph-mon[37097]: pgmap v10084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:45 smithi171 ceph-mon[37789]: pgmap v10084: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:46.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:46.083+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:45:46.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:46.083+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:46.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:45:46.083+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:45:46.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18291.096743 secs 2024-07-26T13:45:46.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18291.096743 secs 2024-07-26T13:45:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18291.096743 secs 2024-07-26T13:45:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:47 smithi044 ceph-mon[32404]: pgmap v10085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:45:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:47 smithi116 ceph-mon[37097]: pgmap v10085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:45:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:47 smithi171 ceph-mon[37789]: pgmap v10085: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:45:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:49 smithi044 ceph-mon[32404]: pgmap v10086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:45:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:49 smithi116 ceph-mon[37097]: pgmap v10086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:45:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:49 smithi171 ceph-mon[37789]: pgmap v10086: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:45:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:51 smithi044 ceph-mon[32404]: pgmap v10087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:45:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18296.096856 secs 2024-07-26T13:45:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:51 smithi116 ceph-mon[37097]: pgmap v10087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:45:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18296.096856 secs 2024-07-26T13:45:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:51 smithi171 ceph-mon[37789]: pgmap v10087: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:45:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18296.096856 secs 2024-07-26T13:45:53.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:53 smithi044 ceph-mon[32404]: pgmap v10088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:53 smithi116 ceph-mon[37097]: pgmap v10088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:53 smithi171 ceph-mon[37789]: pgmap v10088: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:45:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:55 smithi044 ceph-mon[32404]: pgmap v10089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:55 smithi116 ceph-mon[37097]: pgmap v10089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:55 smithi171 ceph-mon[37789]: pgmap v10089: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:45:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18301.097011 secs 2024-07-26T13:45:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18301.097011 secs 2024-07-26T13:45:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18301.097011 secs 2024-07-26T13:45:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:57 smithi044 ceph-mon[32404]: pgmap v10090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:57 smithi116 ceph-mon[37097]: pgmap v10090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:57 smithi171 ceph-mon[37789]: pgmap v10090: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:45:59 smithi044 ceph-mon[32404]: pgmap v10091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:45:59 smithi116 ceph-mon[37097]: pgmap v10091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:45:59 smithi171 ceph-mon[37789]: pgmap v10091: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:01.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:01.084+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:46:01.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:01.084+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:01.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:01.084+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:46:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:01 smithi044 ceph-mon[32404]: pgmap v10092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18306.097171 secs 2024-07-26T13:46:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:01 smithi116 ceph-mon[37097]: pgmap v10092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18306.097171 secs 2024-07-26T13:46:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:01 smithi171 ceph-mon[37789]: pgmap v10092: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18306.097171 secs 2024-07-26T13:46:03.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:03 smithi044 ceph-mon[32404]: pgmap v10093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:03 smithi116 ceph-mon[37097]: pgmap v10093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:03 smithi171 ceph-mon[37789]: pgmap v10093: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:05 smithi044 ceph-mon[32404]: pgmap v10094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:05.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:05 smithi116 ceph-mon[37097]: pgmap v10094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:05 smithi171 ceph-mon[37789]: pgmap v10094: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18311.097313 secs 2024-07-26T13:46:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:06 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18311.097313 secs 2024-07-26T13:46:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:06 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18311.097313 secs 2024-07-26T13:46:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:06 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:07.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:07 smithi044 ceph-mon[32404]: pgmap v10095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:07 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:07 smithi116 ceph-mon[37097]: pgmap v10095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:07 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:07 smithi171 ceph-mon[37789]: pgmap v10095: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:07.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:07 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:08 smithi044 ceph-mon[32404]: pgmap v10096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:46:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:08 smithi044 ceph-mon[32404]: pgmap v10097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:08.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:08 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:08.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:08 smithi116 ceph-mon[37097]: pgmap v10096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:46:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:08 smithi116 ceph-mon[37097]: pgmap v10097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:08 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:08 smithi171 ceph-mon[37789]: pgmap v10096: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:46:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:08 smithi171 ceph-mon[37789]: pgmap v10097: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:08 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:10 smithi044 ceph-mon[32404]: pgmap v10098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:10 smithi116 ceph-mon[37097]: pgmap v10098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:10 smithi171 ceph-mon[37789]: pgmap v10098: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18316.097459 secs 2024-07-26T13:46:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18316.097459 secs 2024-07-26T13:46:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18316.097459 secs 2024-07-26T13:46:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:12 smithi044 ceph-mon[32404]: pgmap v10099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:12 smithi116 ceph-mon[37097]: pgmap v10099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:12 smithi171 ceph-mon[37789]: pgmap v10099: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:14 smithi044 ceph-mon[32404]: pgmap v10100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:14.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:14 smithi116 ceph-mon[37097]: pgmap v10100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:14 smithi171 ceph-mon[37789]: pgmap v10100: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:16.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:16.089+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:46:16.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:16.089+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:16.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:16.089+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:46:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:16 smithi116 ceph-mon[37097]: pgmap v10101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18321.097547 secs 2024-07-26T13:46:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:16 smithi171 ceph-mon[37789]: pgmap v10101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18321.097547 secs 2024-07-26T13:46:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:16 smithi044 ceph-mon[32404]: pgmap v10101: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:46:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18321.097547 secs 2024-07-26T13:46:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:18 smithi171 ceph-mon[37789]: pgmap v10102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:18 smithi044 ceph-mon[32404]: pgmap v10102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:18 smithi116 ceph-mon[37097]: pgmap v10102: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:20 smithi171 ceph-mon[37789]: pgmap v10103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:46:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:20 smithi044 ceph-mon[32404]: pgmap v10103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:46:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:20 smithi116 ceph-mon[37097]: pgmap v10103: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:46:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18326.097706 secs 2024-07-26T13:46:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18326.097706 secs 2024-07-26T13:46:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18326.097706 secs 2024-07-26T13:46:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:22 smithi171 ceph-mon[37789]: pgmap v10104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:22 smithi044 ceph-mon[32404]: pgmap v10104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:22 smithi116 ceph-mon[37097]: pgmap v10104: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:24 smithi171 ceph-mon[37789]: pgmap v10105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:24 smithi044 ceph-mon[32404]: pgmap v10105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:24 smithi116 ceph-mon[37097]: pgmap v10105: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:26 smithi171 ceph-mon[37789]: pgmap v10106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18331.097848 secs 2024-07-26T13:46:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:26 smithi044 ceph-mon[32404]: pgmap v10106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18331.097848 secs 2024-07-26T13:46:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:26 smithi116 ceph-mon[37097]: pgmap v10106: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18331.097848 secs 2024-07-26T13:46:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:28 smithi171 ceph-mon[37789]: pgmap v10107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:28 smithi044 ceph-mon[32404]: pgmap v10107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:29.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:28 smithi116 ceph-mon[37097]: pgmap v10107: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:30 smithi171 ceph-mon[37789]: pgmap v10108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:30 smithi044 ceph-mon[32404]: pgmap v10108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:31.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:30 smithi116 ceph-mon[37097]: pgmap v10108: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:31.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:31.090+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:46:31.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:31.090+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:31.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:31.090+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:46:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18336.098002 secs 2024-07-26T13:46:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18336.098002 secs 2024-07-26T13:46:32.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18336.098002 secs 2024-07-26T13:46:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:32 smithi171 ceph-mon[37789]: pgmap v10109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:32.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:32.996+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:46:32.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:32.996+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:46:32.998 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:32 smithi044 ceph-mon[32404]: pgmap v10109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:32.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:32.996+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:46:32.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:32.997+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:46:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:32 smithi116 ceph-mon[37097]: pgmap v10109: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:34 smithi044 ceph-mon[32404]: pgmap v10110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:46:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:34 smithi116 ceph-mon[37097]: pgmap v10110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:46:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:34 smithi171 ceph-mon[37789]: pgmap v10110: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:46:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:36 smithi044 ceph-mon[32404]: pgmap v10111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18341.098140 secs 2024-07-26T13:46:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:36 smithi116 ceph-mon[37097]: pgmap v10111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18341.098140 secs 2024-07-26T13:46:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:36 smithi171 ceph-mon[37789]: pgmap v10111: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:37.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18341.098140 secs 2024-07-26T13:46:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:38 smithi044 ceph-mon[32404]: pgmap v10112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:46:39.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:38 smithi116 ceph-mon[37097]: pgmap v10112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:46:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:38 smithi171 ceph-mon[37789]: pgmap v10112: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:46:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:40 smithi044 ceph-mon[32404]: pgmap v10113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:40 smithi116 ceph-mon[37097]: pgmap v10113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:40 smithi171 ceph-mon[37789]: pgmap v10113: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18346.098238 secs 2024-07-26T13:46:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18346.098238 secs 2024-07-26T13:46:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18346.098238 secs 2024-07-26T13:46:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:42 smithi044 ceph-mon[32404]: pgmap v10114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:42 smithi116 ceph-mon[37097]: pgmap v10114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:42 smithi171 ceph-mon[37789]: pgmap v10114: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:43 smithi044 ceph-mon[32404]: pgmap v10115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:44.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:43 smithi116 ceph-mon[37097]: pgmap v10115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:43 smithi171 ceph-mon[37789]: pgmap v10115: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:46.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:46.096+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:46:46.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:46.096+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:46:46.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:46:46.096+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:46.790 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:46 smithi171 ceph-mon[37789]: pgmap v10116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:46.790 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18351.098395 secs 2024-07-26T13:46:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:46 smithi044 ceph-mon[32404]: pgmap v10116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18351.098395 secs 2024-07-26T13:46:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:46 smithi116 ceph-mon[37097]: pgmap v10116: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18351.098395 secs 2024-07-26T13:46:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:48 smithi171 ceph-mon[37789]: pgmap v10117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:48 smithi044 ceph-mon[32404]: pgmap v10117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:48 smithi116 ceph-mon[37097]: pgmap v10117: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:50.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:50 smithi171 ceph-mon[37789]: pgmap v10118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:46:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:50 smithi044 ceph-mon[32404]: pgmap v10118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:46:51.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:50 smithi116 ceph-mon[37097]: pgmap v10118: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:46:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18356.098539 secs 2024-07-26T13:46:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18356.098539 secs 2024-07-26T13:46:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18356.098539 secs 2024-07-26T13:46:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:52 smithi171 ceph-mon[37789]: pgmap v10119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:53.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:52 smithi044 ceph-mon[32404]: pgmap v10119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:52 smithi116 ceph-mon[37097]: pgmap v10119: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:54 smithi171 ceph-mon[37789]: pgmap v10120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:54 smithi044 ceph-mon[32404]: pgmap v10120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:54 smithi116 ceph-mon[37097]: pgmap v10120: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:46:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:56 smithi171 ceph-mon[37789]: pgmap v10121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18361.098677 secs 2024-07-26T13:46:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:56 smithi044 ceph-mon[32404]: pgmap v10121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18361.098677 secs 2024-07-26T13:46:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:56 smithi116 ceph-mon[37097]: pgmap v10121: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18361.098677 secs 2024-07-26T13:46:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:58 smithi171 ceph-mon[37789]: pgmap v10122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:58 smithi044 ceph-mon[32404]: pgmap v10122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:46:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:58 smithi116 ceph-mon[37097]: pgmap v10122: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:46:59 smithi044 ceph-mon[32404]: pgmap v10123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:00.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:46:59 smithi116 ceph-mon[37097]: pgmap v10123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:46:59 smithi171 ceph-mon[37789]: pgmap v10123: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:01.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:01.097+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:47:01.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:01.097+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:47:01.099 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:01.097+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18366.098823 secs 2024-07-26T13:47:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18366.098823 secs 2024-07-26T13:47:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18366.098823 secs 2024-07-26T13:47:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:02 smithi171 ceph-mon[37789]: pgmap v10124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:02 smithi044 ceph-mon[32404]: pgmap v10124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:02.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:02 smithi116 ceph-mon[37097]: pgmap v10124: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:04 smithi171 ceph-mon[37789]: pgmap v10125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:47:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:04 smithi044 ceph-mon[32404]: pgmap v10125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:47:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:04 smithi116 ceph-mon[37097]: pgmap v10125: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:47:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:06 smithi171 ceph-mon[37789]: pgmap v10126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18371.098969 secs 2024-07-26T13:47:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:06 smithi044 ceph-mon[32404]: pgmap v10126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18371.098969 secs 2024-07-26T13:47:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:06 smithi116 ceph-mon[37097]: pgmap v10126: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18371.098969 secs 2024-07-26T13:47:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:08 smithi171 ceph-mon[37789]: pgmap v10127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:08.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:08 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:08 smithi044 ceph-mon[32404]: pgmap v10127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:08 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:09.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:08 smithi116 ceph-mon[37097]: pgmap v10127: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:08 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:09.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:09 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:10.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:09 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:09 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:10 smithi171 ceph-mon[37789]: pgmap v10128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:47:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:10 smithi171 ceph-mon[37789]: pgmap v10129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:10 smithi044 ceph-mon[32404]: pgmap v10128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:47:11.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:10 smithi044 ceph-mon[32404]: pgmap v10129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:10 smithi116 ceph-mon[37097]: pgmap v10128: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:47:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:10 smithi116 ceph-mon[37097]: pgmap v10129: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18376.099122 secs 2024-07-26T13:47:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18376.099122 secs 2024-07-26T13:47:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18376.099122 secs 2024-07-26T13:47:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:12 smithi044 ceph-mon[32404]: pgmap v10130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:12 smithi116 ceph-mon[37097]: pgmap v10130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:12 smithi171 ceph-mon[37789]: pgmap v10130: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:14 smithi044 ceph-mon[32404]: pgmap v10131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:47:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:14 smithi116 ceph-mon[37097]: pgmap v10131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:47:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:14 smithi171 ceph-mon[37789]: pgmap v10131: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:47:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:16.102+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:47:16.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:16.102+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:47:16.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:16.102+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:16.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:16 smithi171 ceph-mon[37789]: pgmap v10132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:16.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18381.099231 secs 2024-07-26T13:47:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:16 smithi044 ceph-mon[32404]: pgmap v10132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18381.099231 secs 2024-07-26T13:47:17.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:16 smithi116 ceph-mon[37097]: pgmap v10132: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18381.099231 secs 2024-07-26T13:47:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:18.004+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:47:18.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:18.004+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:47:18.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:18.004+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:47:18.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:18.004+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:47:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:18 smithi044 ceph-mon[32404]: pgmap v10133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:18 smithi116 ceph-mon[37097]: pgmap v10133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:18 smithi171 ceph-mon[37789]: pgmap v10133: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:20 smithi044 ceph-mon[32404]: pgmap v10134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:47:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:20 smithi116 ceph-mon[37097]: pgmap v10134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:47:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:20 smithi171 ceph-mon[37789]: pgmap v10134: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:47:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18386.099399 secs 2024-07-26T13:47:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18386.099399 secs 2024-07-26T13:47:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18386.099399 secs 2024-07-26T13:47:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:22 smithi044 ceph-mon[32404]: pgmap v10135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:22 smithi116 ceph-mon[37097]: pgmap v10135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:22 smithi171 ceph-mon[37789]: pgmap v10135: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:23 smithi044 ceph-mon[32404]: pgmap v10136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:23 smithi116 ceph-mon[37097]: pgmap v10136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:23 smithi171 ceph-mon[37789]: pgmap v10136: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:25 smithi171 ceph-mon[37789]: pgmap v10137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:25 smithi044 ceph-mon[32404]: pgmap v10137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:25 smithi116 ceph-mon[37097]: pgmap v10137: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18391.099554 secs 2024-07-26T13:47:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18391.099554 secs 2024-07-26T13:47:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18391.099554 secs 2024-07-26T13:47:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:27 smithi171 ceph-mon[37789]: pgmap v10138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:27 smithi044 ceph-mon[32404]: pgmap v10138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:27 smithi116 ceph-mon[37097]: pgmap v10138: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:29 smithi171 ceph-mon[37789]: pgmap v10139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:29 smithi044 ceph-mon[32404]: pgmap v10139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:29 smithi116 ceph-mon[37097]: pgmap v10139: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:31.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:31.103+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:47:31.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:31.103+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:47:31.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:31.103+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:31 smithi171 ceph-mon[37789]: pgmap v10140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18396.099711 secs 2024-07-26T13:47:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:31 smithi044 ceph-mon[32404]: pgmap v10140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18396.099711 secs 2024-07-26T13:47:32.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:31 smithi116 ceph-mon[37097]: pgmap v10140: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18396.099711 secs 2024-07-26T13:47:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:33 smithi044 ceph-mon[32404]: pgmap v10141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:33 smithi171 ceph-mon[37789]: pgmap v10141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:33 smithi116 ceph-mon[37097]: pgmap v10141: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:36.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:35 smithi171 ceph-mon[37789]: pgmap v10142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:35 smithi044 ceph-mon[32404]: pgmap v10142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:36.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:35 smithi116 ceph-mon[37097]: pgmap v10142: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18401.099828 secs 2024-07-26T13:47:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18401.099828 secs 2024-07-26T13:47:37.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18401.099828 secs 2024-07-26T13:47:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:37 smithi044 ceph-mon[32404]: pgmap v10143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:37 smithi116 ceph-mon[37097]: pgmap v10143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:37 smithi171 ceph-mon[37789]: pgmap v10143: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:39 smithi044 ceph-mon[32404]: pgmap v10144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:39 smithi116 ceph-mon[37097]: pgmap v10144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:39 smithi171 ceph-mon[37789]: pgmap v10144: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:41 smithi044 ceph-mon[32404]: pgmap v10145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18406.099981 secs 2024-07-26T13:47:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:41 smithi116 ceph-mon[37097]: pgmap v10145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18406.099981 secs 2024-07-26T13:47:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:41 smithi171 ceph-mon[37789]: pgmap v10145: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18406.099981 secs 2024-07-26T13:47:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:43 smithi044 ceph-mon[32404]: pgmap v10146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:43 smithi116 ceph-mon[37097]: pgmap v10146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:43 smithi171 ceph-mon[37789]: pgmap v10146: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:46.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:46.109+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:47:46.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:46.109+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:46.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:47:46.109+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:47:46.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:45 smithi171 ceph-mon[37789]: pgmap v10147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:45 smithi044 ceph-mon[32404]: pgmap v10147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:45 smithi116 ceph-mon[37097]: pgmap v10147: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18411.100136 secs 2024-07-26T13:47:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18411.100136 secs 2024-07-26T13:47:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18411.100136 secs 2024-07-26T13:47:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:47 smithi044 ceph-mon[32404]: pgmap v10148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:48.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:47 smithi116 ceph-mon[37097]: pgmap v10148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:47 smithi171 ceph-mon[37789]: pgmap v10148: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:49 smithi044 ceph-mon[32404]: pgmap v10149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:49 smithi116 ceph-mon[37097]: pgmap v10149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:49 smithi171 ceph-mon[37789]: pgmap v10149: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18416.100273 secs 2024-07-26T13:47:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:51 smithi044 ceph-mon[32404]: pgmap v10150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18416.100273 secs 2024-07-26T13:47:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:51 smithi116 ceph-mon[37097]: pgmap v10150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18416.100273 secs 2024-07-26T13:47:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:51 smithi171 ceph-mon[37789]: pgmap v10150: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:53 smithi044 ceph-mon[32404]: pgmap v10151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:54.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:53 smithi116 ceph-mon[37097]: pgmap v10151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:53 smithi171 ceph-mon[37789]: pgmap v10151: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:55 smithi044 ceph-mon[32404]: pgmap v10152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:55 smithi116 ceph-mon[37097]: pgmap v10152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:55 smithi171 ceph-mon[37789]: pgmap v10152: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:47:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18421.100428 secs 2024-07-26T13:47:57.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18421.100428 secs 2024-07-26T13:47:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18421.100428 secs 2024-07-26T13:47:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:57 smithi044 ceph-mon[32404]: pgmap v10153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:57 smithi116 ceph-mon[37097]: pgmap v10153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:57 smithi171 ceph-mon[37789]: pgmap v10153: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:47:59 smithi044 ceph-mon[32404]: pgmap v10154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:47:59 smithi116 ceph-mon[37097]: pgmap v10154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:47:59 smithi171 ceph-mon[37789]: pgmap v10154: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:01.110+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:48:01.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:01.110+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:01.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:01.110+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:48:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18426.100592 secs 2024-07-26T13:48:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:01 smithi044 ceph-mon[32404]: pgmap v10155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18426.100592 secs 2024-07-26T13:48:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:01 smithi116 ceph-mon[37097]: pgmap v10155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18426.100592 secs 2024-07-26T13:48:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:01 smithi171 ceph-mon[37789]: pgmap v10155: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:03 smithi044 ceph-mon[32404]: pgmap v10156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:03 smithi116 ceph-mon[37097]: pgmap v10156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:03 smithi171 ceph-mon[37789]: pgmap v10156: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:06.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:05 smithi044 ceph-mon[32404]: pgmap v10157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:48:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:05 smithi116 ceph-mon[37097]: pgmap v10157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:48:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:05 smithi171 ceph-mon[37789]: pgmap v10157: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:48:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18431.100733 secs 2024-07-26T13:48:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18431.100733 secs 2024-07-26T13:48:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18431.100733 secs 2024-07-26T13:48:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:07 smithi044 ceph-mon[32404]: pgmap v10158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:07 smithi116 ceph-mon[37097]: pgmap v10158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:07 smithi171 ceph-mon[37789]: pgmap v10158: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:09 smithi044 ceph-mon[32404]: pgmap v10159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:09 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:09 smithi116 ceph-mon[37097]: pgmap v10159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:10.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:09 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:09 smithi171 ceph-mon[37789]: pgmap v10159: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:48:10.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:09 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:10 smithi044 ceph-mon[32404]: pgmap v10160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:10 smithi044 ceph-mon[32404]: pgmap v10161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:48:11.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:10 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:10 smithi116 ceph-mon[37097]: pgmap v10160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:10 smithi116 ceph-mon[37097]: pgmap v10161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:48:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:10 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:48:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:10 smithi171 ceph-mon[37789]: pgmap v10160: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:48:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:10 smithi171 ceph-mon[37789]: pgmap v10161: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:48:11.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:10 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:48:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18436.100897 secs 2024-07-26T13:48:12.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18436.100897 secs 2024-07-26T13:48:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18436.100897 secs 2024-07-26T13:48:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:12 smithi044 ceph-mon[32404]: pgmap v10162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:12 smithi116 ceph-mon[37097]: pgmap v10162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:12 smithi171 ceph-mon[37789]: pgmap v10162: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:15 smithi171 ceph-mon[37789]: pgmap v10163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:15.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:15 smithi044 ceph-mon[32404]: pgmap v10163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:15 smithi116 ceph-mon[37097]: pgmap v10163: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:16.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:16.114+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:48:16.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:16.115+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:16.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:16.115+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:48:16.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18441.101056 secs 2024-07-26T13:48:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18441.101056 secs 2024-07-26T13:48:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18441.101056 secs 2024-07-26T13:48:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:17 smithi044 ceph-mon[32404]: pgmap v10164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:48:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:17 smithi116 ceph-mon[37097]: pgmap v10164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:48:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:17 smithi171 ceph-mon[37789]: pgmap v10164: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:48:19.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:19 smithi044 ceph-mon[32404]: pgmap v10165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:19.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:19 smithi116 ceph-mon[37097]: pgmap v10165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:19.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:19 smithi171 ceph-mon[37789]: pgmap v10165: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:48:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:21 smithi044 ceph-mon[32404]: pgmap v10166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:21.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18446.101211 secs 2024-07-26T13:48:21.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:21 smithi116 ceph-mon[37097]: pgmap v10166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18446.101211 secs 2024-07-26T13:48:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:21 smithi171 ceph-mon[37789]: pgmap v10166: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:21.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18446.101211 secs 2024-07-26T13:48:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:23 smithi044 ceph-mon[32404]: pgmap v10167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:23 smithi116 ceph-mon[37097]: pgmap v10167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:23 smithi171 ceph-mon[37789]: pgmap v10167: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:25 smithi044 ceph-mon[32404]: pgmap v10168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:25 smithi116 ceph-mon[37097]: pgmap v10168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:25 smithi171 ceph-mon[37789]: pgmap v10168: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18451.101338 secs 2024-07-26T13:48:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18451.101338 secs 2024-07-26T13:48:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18451.101338 secs 2024-07-26T13:48:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:27 smithi044 ceph-mon[32404]: pgmap v10169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:27 smithi116 ceph-mon[37097]: pgmap v10169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:27 smithi171 ceph-mon[37789]: pgmap v10169: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:29 smithi044 ceph-mon[32404]: pgmap v10170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:29 smithi116 ceph-mon[37097]: pgmap v10170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:29.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:29 smithi171 ceph-mon[37789]: pgmap v10170: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:31.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:31.116+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:48:31.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:31.116+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:48:31.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:31.116+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:31 smithi044 ceph-mon[32404]: pgmap v10171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18456.101474 secs 2024-07-26T13:48:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:31 smithi116 ceph-mon[37097]: pgmap v10171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18456.101474 secs 2024-07-26T13:48:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:31 smithi171 ceph-mon[37789]: pgmap v10171: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18456.101474 secs 2024-07-26T13:48:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:33 smithi044 ceph-mon[32404]: pgmap v10172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:33 smithi116 ceph-mon[37097]: pgmap v10172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:33 smithi171 ceph-mon[37789]: pgmap v10172: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:35 smithi044 ceph-mon[32404]: pgmap v10173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:35 smithi116 ceph-mon[37097]: pgmap v10173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:35 smithi171 ceph-mon[37789]: pgmap v10173: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18461.101678 secs 2024-07-26T13:48:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18461.101678 secs 2024-07-26T13:48:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18461.101678 secs 2024-07-26T13:48:37.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:37 smithi044 ceph-mon[32404]: pgmap v10174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:37.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:37 smithi116 ceph-mon[37097]: pgmap v10174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:37 smithi171 ceph-mon[37789]: pgmap v10174: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:39.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:39 smithi044 ceph-mon[32404]: pgmap v10175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:39.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:39 smithi116 ceph-mon[37097]: pgmap v10175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:39 smithi171 ceph-mon[37789]: pgmap v10175: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:41 smithi044 ceph-mon[32404]: pgmap v10176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18466.101824 secs 2024-07-26T13:48:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:41 smithi116 ceph-mon[37097]: pgmap v10176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18466.101824 secs 2024-07-26T13:48:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:41 smithi171 ceph-mon[37789]: pgmap v10176: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:41.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18466.101824 secs 2024-07-26T13:48:43.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:43 smithi044 ceph-mon[32404]: pgmap v10177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:43.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:43 smithi116 ceph-mon[37097]: pgmap v10177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:43 smithi171 ceph-mon[37789]: pgmap v10177: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:45.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:45 smithi044 ceph-mon[32404]: pgmap v10178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:45.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:45 smithi116 ceph-mon[37097]: pgmap v10178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:45 smithi171 ceph-mon[37789]: pgmap v10178: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:46.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:46.121+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:48:46.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:46.121+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:46.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:48:46.121+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:48:46.487 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18471.101969 secs 2024-07-26T13:48:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18471.101969 secs 2024-07-26T13:48:46.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18471.101969 secs 2024-07-26T13:48:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:47 smithi044 ceph-mon[32404]: pgmap v10179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:47.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:47 smithi116 ceph-mon[37097]: pgmap v10179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:47 smithi171 ceph-mon[37789]: pgmap v10179: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:49 smithi044 ceph-mon[32404]: pgmap v10180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:49 smithi116 ceph-mon[37097]: pgmap v10180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:49 smithi171 ceph-mon[37789]: pgmap v10180: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:51 smithi044 ceph-mon[32404]: pgmap v10181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18476.102131 secs 2024-07-26T13:48:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:51 smithi116 ceph-mon[37097]: pgmap v10181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18476.102131 secs 2024-07-26T13:48:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:51 smithi171 ceph-mon[37789]: pgmap v10181: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18476.102131 secs 2024-07-26T13:48:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:53 smithi044 ceph-mon[32404]: pgmap v10182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:53.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:53 smithi116 ceph-mon[37097]: pgmap v10182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:53 smithi171 ceph-mon[37789]: pgmap v10182: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:55 smithi044 ceph-mon[32404]: pgmap v10183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:55 smithi116 ceph-mon[37097]: pgmap v10183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:55 smithi171 ceph-mon[37789]: pgmap v10183: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:48:56.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18481.102267 secs 2024-07-26T13:48:56.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18481.102267 secs 2024-07-26T13:48:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18481.102267 secs 2024-07-26T13:48:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:57 smithi044 ceph-mon[32404]: pgmap v10184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:57 smithi116 ceph-mon[37097]: pgmap v10184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:57 smithi171 ceph-mon[37789]: pgmap v10184: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:59.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:48:59 smithi044 ceph-mon[32404]: pgmap v10185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:59.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:48:59 smithi116 ceph-mon[37097]: pgmap v10185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:48:59 smithi171 ceph-mon[37789]: pgmap v10185: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:01.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:01.122+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:49:01.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:01.122+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:01.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:01.122+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:49:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:01 smithi044 ceph-mon[32404]: pgmap v10186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18486.102427 secs 2024-07-26T13:49:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:01 smithi116 ceph-mon[37097]: pgmap v10186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18486.102427 secs 2024-07-26T13:49:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:01 smithi171 ceph-mon[37789]: pgmap v10186: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18486.102427 secs 2024-07-26T13:49:03.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:03 smithi044 ceph-mon[32404]: pgmap v10187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:03.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:03 smithi116 ceph-mon[37097]: pgmap v10187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:03.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:03 smithi171 ceph-mon[37789]: pgmap v10187: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:05.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:05 smithi116 ceph-mon[37097]: pgmap v10188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:05.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:05 smithi171 ceph-mon[37789]: pgmap v10188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:05 smithi044 ceph-mon[32404]: pgmap v10188: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18491.102559 secs 2024-07-26T13:49:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18491.102559 secs 2024-07-26T13:49:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18491.102559 secs 2024-07-26T13:49:07.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:07 smithi171 ceph-mon[37789]: pgmap v10189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:07.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:07 smithi044 ceph-mon[32404]: pgmap v10189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:07 smithi116 ceph-mon[37097]: pgmap v10189: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:09.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:09 smithi171 ceph-mon[37789]: pgmap v10190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:09.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:09 smithi044 ceph-mon[32404]: pgmap v10190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:09 smithi116 ceph-mon[37097]: pgmap v10190: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:10 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:10 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:10 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:11 smithi171 ceph-mon[37789]: pgmap v10191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18496.102697 secs 2024-07-26T13:49:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:11.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:11 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:11 smithi044 ceph-mon[32404]: pgmap v10191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18496.102697 secs 2024-07-26T13:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:11 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:49:11.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:11 smithi116 ceph-mon[37097]: pgmap v10191: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18496.102697 secs 2024-07-26T13:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:11 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:49:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:12 smithi171 ceph-mon[37789]: pgmap v10192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:12.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:12 smithi171 ceph-mon[37789]: pgmap v10193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:12 smithi044 ceph-mon[32404]: pgmap v10192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:12.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:12 smithi044 ceph-mon[32404]: pgmap v10193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:12 smithi116 ceph-mon[37097]: pgmap v10192: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:12.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:12 smithi116 ceph-mon[37097]: pgmap v10193: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:14 smithi171 ceph-mon[37789]: pgmap v10194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:14 smithi044 ceph-mon[32404]: pgmap v10194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:14 smithi116 ceph-mon[37097]: pgmap v10194: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:16.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:16.127+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:49:16.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:16.127+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:49:16.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:16.127+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:16 smithi044 ceph-mon[32404]: pgmap v10195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18501.102786 secs 2024-07-26T13:49:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:16 smithi171 ceph-mon[37789]: pgmap v10195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18501.102786 secs 2024-07-26T13:49:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:16 smithi116 ceph-mon[37097]: pgmap v10195: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18501.102786 secs 2024-07-26T13:49:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:18 smithi171 ceph-mon[37789]: pgmap v10196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:18 smithi044 ceph-mon[32404]: pgmap v10196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:18 smithi116 ceph-mon[37097]: pgmap v10196: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:20.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:20 smithi044 ceph-mon[32404]: pgmap v10197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:20.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:20 smithi116 ceph-mon[37097]: pgmap v10197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:20 smithi171 ceph-mon[37789]: pgmap v10197: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18506.102925 secs 2024-07-26T13:49:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18506.102925 secs 2024-07-26T13:49:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18506.102925 secs 2024-07-26T13:49:22.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:22 smithi044 ceph-mon[32404]: pgmap v10198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:22 smithi116 ceph-mon[37097]: pgmap v10198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:22.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:22 smithi171 ceph-mon[37789]: pgmap v10198: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:24 smithi044 ceph-mon[32404]: pgmap v10199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:24 smithi116 ceph-mon[37097]: pgmap v10199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:24 smithi171 ceph-mon[37789]: pgmap v10199: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:26 smithi044 ceph-mon[32404]: pgmap v10200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18511.103035 secs 2024-07-26T13:49:26.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:26 smithi116 ceph-mon[37097]: pgmap v10200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18511.103035 secs 2024-07-26T13:49:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:26 smithi171 ceph-mon[37789]: pgmap v10200: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18511.103035 secs 2024-07-26T13:49:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:28 smithi044 ceph-mon[32404]: pgmap v10201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:28 smithi116 ceph-mon[37097]: pgmap v10201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:28 smithi171 ceph-mon[37789]: pgmap v10201: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:30 smithi044 ceph-mon[32404]: pgmap v10202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:30.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:30 smithi116 ceph-mon[37097]: pgmap v10202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:30 smithi171 ceph-mon[37789]: pgmap v10202: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:31.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:31.128+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:49:31.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:31.128+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:49:31.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:31.128+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18516.103192 secs 2024-07-26T13:49:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18516.103192 secs 2024-07-26T13:49:31.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18516.103192 secs 2024-07-26T13:49:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:32 smithi044 ceph-mon[32404]: pgmap v10203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:32.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:32 smithi116 ceph-mon[37097]: pgmap v10203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:32 smithi171 ceph-mon[37789]: pgmap v10203: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:34 smithi044 ceph-mon[32404]: pgmap v10204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:34 smithi116 ceph-mon[37097]: pgmap v10204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:34 smithi171 ceph-mon[37789]: pgmap v10204: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:36 smithi044 ceph-mon[32404]: pgmap v10205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:36.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18521.103330 secs 2024-07-26T13:49:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:36 smithi116 ceph-mon[37097]: pgmap v10205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18521.103330 secs 2024-07-26T13:49:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:36 smithi171 ceph-mon[37789]: pgmap v10205: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18521.103330 secs 2024-07-26T13:49:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:38 smithi044 ceph-mon[32404]: pgmap v10206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:38.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:38 smithi116 ceph-mon[37097]: pgmap v10206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:38 smithi171 ceph-mon[37789]: pgmap v10206: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:40 smithi044 ceph-mon[32404]: pgmap v10207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:40 smithi116 ceph-mon[37097]: pgmap v10207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:40.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:40 smithi171 ceph-mon[37789]: pgmap v10207: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18526.103494 secs 2024-07-26T13:49:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18526.103494 secs 2024-07-26T13:49:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18526.103494 secs 2024-07-26T13:49:42.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:42 smithi044 ceph-mon[32404]: pgmap v10208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:42 smithi116 ceph-mon[37097]: pgmap v10208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:42 smithi171 ceph-mon[37789]: pgmap v10208: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:44 smithi044 ceph-mon[32404]: pgmap v10209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:44 smithi116 ceph-mon[37097]: pgmap v10209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:44 smithi171 ceph-mon[37789]: pgmap v10209: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:46.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:46.133+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:49:46.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:46.133+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:49:46.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:49:46.133+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:46 smithi044 ceph-mon[32404]: pgmap v10210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:46.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18531.103611 secs 2024-07-26T13:49:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:46 smithi116 ceph-mon[37097]: pgmap v10210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18531.103611 secs 2024-07-26T13:49:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:46 smithi171 ceph-mon[37789]: pgmap v10210: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18531.103611 secs 2024-07-26T13:49:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:48 smithi044 ceph-mon[32404]: pgmap v10211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:48 smithi116 ceph-mon[37097]: pgmap v10211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:48 smithi171 ceph-mon[37789]: pgmap v10211: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:50 smithi044 ceph-mon[32404]: pgmap v10212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:50 smithi116 ceph-mon[37097]: pgmap v10212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:50 smithi171 ceph-mon[37789]: pgmap v10212: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18536.103746 secs 2024-07-26T13:49:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18536.103746 secs 2024-07-26T13:49:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18536.103746 secs 2024-07-26T13:49:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:52 smithi044 ceph-mon[32404]: pgmap v10213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:52 smithi116 ceph-mon[37097]: pgmap v10213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:52 smithi171 ceph-mon[37789]: pgmap v10213: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:54 smithi044 ceph-mon[32404]: pgmap v10214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:54 smithi116 ceph-mon[37097]: pgmap v10214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:54 smithi171 ceph-mon[37789]: pgmap v10214: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:56 smithi044 ceph-mon[32404]: pgmap v10215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18541.103907 secs 2024-07-26T13:49:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:56 smithi116 ceph-mon[37097]: pgmap v10215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18541.103907 secs 2024-07-26T13:49:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:56 smithi171 ceph-mon[37789]: pgmap v10215: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:56.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18541.103907 secs 2024-07-26T13:49:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:49:58 smithi044 ceph-mon[32404]: pgmap v10216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:49:58 smithi116 ceph-mon[37097]: pgmap v10216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:49:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:49:58 smithi171 ceph-mon[37789]: pgmap v10216: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:00 smithi044 ceph-mon[32404]: pgmap v10217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:00.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:00 smithi116 ceph-mon[37097]: pgmap v10217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:00 smithi171 ceph-mon[37789]: pgmap v10217: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:00.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:01.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:01.134+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:50:01.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:01.134+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:50:01.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:01.134+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18546.104056 secs 2024-07-26T13:50:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18546.104056 secs 2024-07-26T13:50:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18546.104056 secs 2024-07-26T13:50:02.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:02 smithi044 ceph-mon[32404]: pgmap v10218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:02.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:02 smithi116 ceph-mon[37097]: pgmap v10218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:02 smithi171 ceph-mon[37789]: pgmap v10218: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:04 smithi116 ceph-mon[37097]: pgmap v10219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:04 smithi171 ceph-mon[37789]: pgmap v10219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:04 smithi044 ceph-mon[32404]: pgmap v10219: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:06 smithi171 ceph-mon[37789]: pgmap v10220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18551.104168 secs 2024-07-26T13:50:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:06 smithi044 ceph-mon[32404]: pgmap v10220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18551.104168 secs 2024-07-26T13:50:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:06 smithi116 ceph-mon[37097]: pgmap v10220: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18551.104168 secs 2024-07-26T13:50:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:08 smithi171 ceph-mon[37789]: pgmap v10221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:09.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:08 smithi044 ceph-mon[32404]: pgmap v10221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:08 smithi116 ceph-mon[37097]: pgmap v10221: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:10 smithi171 ceph-mon[37789]: pgmap v10222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:10 smithi044 ceph-mon[32404]: pgmap v10222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:10 smithi116 ceph-mon[37097]: pgmap v10222: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18556.104323 secs 2024-07-26T13:50:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:11 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18556.104323 secs 2024-07-26T13:50:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:11 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:12.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18556.104323 secs 2024-07-26T13:50:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:11 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:12 smithi171 ceph-mon[37789]: pgmap v10223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:12 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:12.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:12 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:50:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:12 smithi044 ceph-mon[32404]: pgmap v10223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:12 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:13.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:12 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:50:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:12 smithi116 ceph-mon[37097]: pgmap v10223: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:12 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:12 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:50:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:13 smithi171 ceph-mon[37789]: pgmap v10224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:13.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:13 smithi171 ceph-mon[37789]: pgmap v10225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:13 smithi044 ceph-mon[32404]: pgmap v10224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:13 smithi044 ceph-mon[32404]: pgmap v10225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:13 smithi116 ceph-mon[37097]: pgmap v10224: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:13 smithi116 ceph-mon[37097]: pgmap v10225: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:15.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:14 smithi044 ceph-mon[32404]: pgmap v10226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:14 smithi116 ceph-mon[37097]: pgmap v10226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:14 smithi171 ceph-mon[37789]: pgmap v10226: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:16.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:16.140+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:50:16.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:16.140+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:50:16.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:16.140+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18561.104411 secs 2024-07-26T13:50:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18561.104411 secs 2024-07-26T13:50:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18561.104411 secs 2024-07-26T13:50:17.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:17 smithi171 ceph-mon[37789]: pgmap v10227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:17 smithi044 ceph-mon[32404]: pgmap v10227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:17 smithi116 ceph-mon[37097]: pgmap v10227: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:18.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:18.038+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:50:18.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:18.038+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:50:18.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:18.038+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:50:18.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:18.038+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:50:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:19 smithi171 ceph-mon[37789]: pgmap v10228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:19 smithi044 ceph-mon[32404]: pgmap v10228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:19 smithi116 ceph-mon[37097]: pgmap v10228: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:21 smithi171 ceph-mon[37789]: pgmap v10229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18566.104567 secs 2024-07-26T13:50:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:21 smithi044 ceph-mon[32404]: pgmap v10229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18566.104567 secs 2024-07-26T13:50:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:21 smithi116 ceph-mon[37097]: pgmap v10229: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18566.104567 secs 2024-07-26T13:50:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:23 smithi171 ceph-mon[37789]: pgmap v10230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:23 smithi044 ceph-mon[32404]: pgmap v10230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:23 smithi116 ceph-mon[37097]: pgmap v10230: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:25 smithi044 ceph-mon[32404]: pgmap v10231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:26.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:25 smithi116 ceph-mon[37097]: pgmap v10231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:25 smithi171 ceph-mon[37789]: pgmap v10231: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:26.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18571.104744 secs 2024-07-26T13:50:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18571.104744 secs 2024-07-26T13:50:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18571.104744 secs 2024-07-26T13:50:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:27 smithi171 ceph-mon[37789]: pgmap v10232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:27 smithi044 ceph-mon[32404]: pgmap v10232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:27 smithi116 ceph-mon[37097]: pgmap v10232: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:29 smithi044 ceph-mon[32404]: pgmap v10233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:29 smithi116 ceph-mon[37097]: pgmap v10233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:29 smithi171 ceph-mon[37789]: pgmap v10233: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:31.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:31.140+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:50:31.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:31.140+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:50:31.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:31.140+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:31 smithi044 ceph-mon[32404]: pgmap v10234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18576.104894 secs 2024-07-26T13:50:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:31 smithi116 ceph-mon[37097]: pgmap v10234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18576.104894 secs 2024-07-26T13:50:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:31 smithi171 ceph-mon[37789]: pgmap v10234: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18576.104894 secs 2024-07-26T13:50:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:33 smithi044 ceph-mon[32404]: pgmap v10235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:33 smithi116 ceph-mon[37097]: pgmap v10235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:33 smithi171 ceph-mon[37789]: pgmap v10235: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:35 smithi044 ceph-mon[32404]: pgmap v10236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:35 smithi116 ceph-mon[37097]: pgmap v10236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:35 smithi171 ceph-mon[37789]: pgmap v10236: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18581.104969 secs 2024-07-26T13:50:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18581.104969 secs 2024-07-26T13:50:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18581.104969 secs 2024-07-26T13:50:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:37 smithi044 ceph-mon[32404]: pgmap v10237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:37 smithi116 ceph-mon[37097]: pgmap v10237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:37 smithi171 ceph-mon[37789]: pgmap v10237: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:39 smithi044 ceph-mon[32404]: pgmap v10238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:39 smithi116 ceph-mon[37097]: pgmap v10238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:39 smithi171 ceph-mon[37789]: pgmap v10238: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:41 smithi044 ceph-mon[32404]: pgmap v10239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18586.105115 secs 2024-07-26T13:50:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:41 smithi116 ceph-mon[37097]: pgmap v10239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18586.105115 secs 2024-07-26T13:50:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:41 smithi171 ceph-mon[37789]: pgmap v10239: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18586.105115 secs 2024-07-26T13:50:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:43 smithi044 ceph-mon[32404]: pgmap v10240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:43 smithi116 ceph-mon[37097]: pgmap v10240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:43 smithi171 ceph-mon[37789]: pgmap v10240: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:45 smithi044 ceph-mon[32404]: pgmap v10241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:45 smithi116 ceph-mon[37097]: pgmap v10241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:45 smithi171 ceph-mon[37789]: pgmap v10241: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:46.145+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:50:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:46.145+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:46.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:50:46.145+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:50:46.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18591.105277 secs 2024-07-26T13:50:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18591.105277 secs 2024-07-26T13:50:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18591.105277 secs 2024-07-26T13:50:48.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:47 smithi044 ceph-mon[32404]: pgmap v10242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:47 smithi116 ceph-mon[37097]: pgmap v10242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:47 smithi171 ceph-mon[37789]: pgmap v10242: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:48 smithi044 ceph-mon[32404]: pgmap v10243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:48 smithi116 ceph-mon[37097]: pgmap v10243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:48 smithi171 ceph-mon[37789]: pgmap v10243: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:50:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:51 smithi171 ceph-mon[37789]: pgmap v10244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18596.105438 secs 2024-07-26T13:50:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:51 smithi044 ceph-mon[32404]: pgmap v10244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18596.105438 secs 2024-07-26T13:50:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:51 smithi116 ceph-mon[37097]: pgmap v10244: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18596.105438 secs 2024-07-26T13:50:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:53 smithi171 ceph-mon[37789]: pgmap v10245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:53 smithi044 ceph-mon[32404]: pgmap v10245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:53 smithi116 ceph-mon[37097]: pgmap v10245: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:50:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:54 smithi044 ceph-mon[32404]: pgmap v10246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:55.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:54 smithi116 ceph-mon[37097]: pgmap v10246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:54 smithi171 ceph-mon[37789]: pgmap v10246: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:50:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18601.105583 secs 2024-07-26T13:50:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18601.105583 secs 2024-07-26T13:50:56.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18601.105583 secs 2024-07-26T13:50:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:57 smithi171 ceph-mon[37789]: pgmap v10247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:57 smithi044 ceph-mon[32404]: pgmap v10247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:57 smithi116 ceph-mon[37097]: pgmap v10247: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:50:59 smithi171 ceph-mon[37789]: pgmap v10248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:51:00.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:50:59 smithi044 ceph-mon[32404]: pgmap v10248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:51:00.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:50:59 smithi116 ceph-mon[37097]: pgmap v10248: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:51:01.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:01.146+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:51:01.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:01.146+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:01.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:01.146+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:51:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:01 smithi044 ceph-mon[32404]: pgmap v10249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18606.105730 secs 2024-07-26T13:51:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:01 smithi116 ceph-mon[37097]: pgmap v10249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18606.105730 secs 2024-07-26T13:51:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:01 smithi171 ceph-mon[37789]: pgmap v10249: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18606.105730 secs 2024-07-26T13:51:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:03 smithi044 ceph-mon[32404]: pgmap v10250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:03 smithi116 ceph-mon[37097]: pgmap v10250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:03 smithi171 ceph-mon[37789]: pgmap v10250: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:05 smithi044 ceph-mon[32404]: pgmap v10251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:05 smithi116 ceph-mon[37097]: pgmap v10251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:05 smithi171 ceph-mon[37789]: pgmap v10251: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:07.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18611.105882 secs 2024-07-26T13:51:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18611.105882 secs 2024-07-26T13:51:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18611.105882 secs 2024-07-26T13:51:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:07 smithi044 ceph-mon[32404]: pgmap v10252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:07 smithi116 ceph-mon[37097]: pgmap v10252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:07 smithi171 ceph-mon[37789]: pgmap v10252: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:09 smithi044 ceph-mon[32404]: pgmap v10253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:51:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:09 smithi116 ceph-mon[37097]: pgmap v10253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:51:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:09 smithi171 ceph-mon[37789]: pgmap v10253: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:51:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:11 smithi044 ceph-mon[32404]: pgmap v10254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18616.106050 secs 2024-07-26T13:51:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:11 smithi116 ceph-mon[37097]: pgmap v10254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18616.106050 secs 2024-07-26T13:51:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:11 smithi171 ceph-mon[37789]: pgmap v10254: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18616.106050 secs 2024-07-26T13:51:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:13 smithi044 ceph-mon[32404]: pgmap v10255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:14.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:13 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:13 smithi116 ceph-mon[37097]: pgmap v10255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:13 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:13 smithi171 ceph-mon[37789]: pgmap v10255: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:13 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:15.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:15.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:15.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:14 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:51:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:15.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:14 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:51:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:15.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:14 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:51:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:15 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:51:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:15 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:51:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:15 smithi044 ceph-mon[32404]: pgmap v10256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:16.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:15 smithi044 ceph-mon[32404]: pgmap v10257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:15 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:51:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:15 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:51:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:15 smithi116 ceph-mon[37097]: pgmap v10256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:15 smithi116 ceph-mon[37097]: pgmap v10257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:15 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T13:51:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:15 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T13:51:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:15 smithi171 ceph-mon[37789]: pgmap v10256: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:16.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:15 smithi171 ceph-mon[37789]: pgmap v10257: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:16.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:16.152+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:51:16.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:16.152+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:16.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:16.152+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:51:16.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18621.106200 secs 2024-07-26T13:51:16.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18621.106200 secs 2024-07-26T13:51:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18621.106200 secs 2024-07-26T13:51:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:17 smithi044 ceph-mon[32404]: pgmap v10258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:17 smithi116 ceph-mon[37097]: pgmap v10258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:17 smithi171 ceph-mon[37789]: pgmap v10258: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:19 smithi044 ceph-mon[32404]: pgmap v10259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:20.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:19 smithi116 ceph-mon[37097]: pgmap v10259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:19 smithi171 ceph-mon[37789]: pgmap v10259: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:21 smithi044 ceph-mon[32404]: pgmap v10260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18626.106366 secs 2024-07-26T13:51:22.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:21 smithi116 ceph-mon[37097]: pgmap v10260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18626.106366 secs 2024-07-26T13:51:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:21 smithi171 ceph-mon[37789]: pgmap v10260: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18626.106366 secs 2024-07-26T13:51:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:23 smithi044 ceph-mon[32404]: pgmap v10261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:24.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:23 smithi116 ceph-mon[37097]: pgmap v10261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:23 smithi171 ceph-mon[37789]: pgmap v10261: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:25 smithi044 ceph-mon[32404]: pgmap v10262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:26.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:25 smithi116 ceph-mon[37097]: pgmap v10262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:25 smithi171 ceph-mon[37789]: pgmap v10262: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18631.106514 secs 2024-07-26T13:51:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:26 smithi044 ceph-mon[32404]: pgmap v10263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18631.106514 secs 2024-07-26T13:51:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:26 smithi116 ceph-mon[37097]: pgmap v10263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18631.106514 secs 2024-07-26T13:51:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:26 smithi171 ceph-mon[37789]: pgmap v10263: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:29.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:28 smithi044 ceph-mon[32404]: pgmap v10264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:28 smithi116 ceph-mon[37097]: pgmap v10264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:28 smithi171 ceph-mon[37789]: pgmap v10264: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:31.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:31.152+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:51:31.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:31.152+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:31.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:31.152+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:51:31.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:30 smithi044 ceph-mon[32404]: pgmap v10265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:30 smithi116 ceph-mon[37097]: pgmap v10265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:30 smithi171 ceph-mon[37789]: pgmap v10265: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18636.106675 secs 2024-07-26T13:51:32.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18636.106675 secs 2024-07-26T13:51:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18636.106675 secs 2024-07-26T13:51:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:32 smithi044 ceph-mon[32404]: pgmap v10266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:33.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:32 smithi116 ceph-mon[37097]: pgmap v10266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:32 smithi171 ceph-mon[37789]: pgmap v10266: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:35.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:34 smithi044 ceph-mon[32404]: pgmap v10267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:34 smithi116 ceph-mon[37097]: pgmap v10267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:34 smithi171 ceph-mon[37789]: pgmap v10267: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18641.106819 secs 2024-07-26T13:51:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:36 smithi044 ceph-mon[32404]: pgmap v10268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:51:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18641.106819 secs 2024-07-26T13:51:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:36 smithi116 ceph-mon[37097]: pgmap v10268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:51:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18641.106819 secs 2024-07-26T13:51:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:36 smithi171 ceph-mon[37789]: pgmap v10268: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:51:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:38 smithi044 ceph-mon[32404]: pgmap v10269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:38 smithi116 ceph-mon[37097]: pgmap v10269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:38 smithi171 ceph-mon[37789]: pgmap v10269: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:40 smithi044 ceph-mon[32404]: pgmap v10270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:51:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:40 smithi116 ceph-mon[37097]: pgmap v10270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:51:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:40 smithi171 ceph-mon[37789]: pgmap v10270: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:51:42.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18646.106972 secs 2024-07-26T13:51:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18646.106972 secs 2024-07-26T13:51:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18646.106972 secs 2024-07-26T13:51:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:42 smithi044 ceph-mon[32404]: pgmap v10271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:42 smithi116 ceph-mon[37097]: pgmap v10271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:42 smithi171 ceph-mon[37789]: pgmap v10271: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:51:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:44 smithi044 ceph-mon[32404]: pgmap v10272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:44 smithi116 ceph-mon[37097]: pgmap v10272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:44 smithi171 ceph-mon[37789]: pgmap v10272: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:46.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:46.158+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:51:46.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:46.158+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:46.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:51:46.158+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:51:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18651.107135 secs 2024-07-26T13:51:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:46 smithi044 ceph-mon[32404]: pgmap v10273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18651.107135 secs 2024-07-26T13:51:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:46 smithi116 ceph-mon[37097]: pgmap v10273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18651.107135 secs 2024-07-26T13:51:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:46 smithi171 ceph-mon[37789]: pgmap v10273: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:48 smithi044 ceph-mon[32404]: pgmap v10274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:48 smithi116 ceph-mon[37097]: pgmap v10274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:48 smithi171 ceph-mon[37789]: pgmap v10274: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:50 smithi044 ceph-mon[32404]: pgmap v10275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:51.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:50 smithi116 ceph-mon[37097]: pgmap v10275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:50 smithi171 ceph-mon[37789]: pgmap v10275: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:51:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18656.107302 secs 2024-07-26T13:51:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18656.107302 secs 2024-07-26T13:51:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18656.107302 secs 2024-07-26T13:51:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:52 smithi044 ceph-mon[32404]: pgmap v10276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:52 smithi116 ceph-mon[37097]: pgmap v10276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:52 smithi171 ceph-mon[37789]: pgmap v10276: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:54 smithi044 ceph-mon[32404]: pgmap v10277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:54 smithi116 ceph-mon[37097]: pgmap v10277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:54 smithi171 ceph-mon[37789]: pgmap v10277: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18661.107465 secs 2024-07-26T13:51:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:56 smithi044 ceph-mon[32404]: pgmap v10278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18661.107465 secs 2024-07-26T13:51:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:56 smithi116 ceph-mon[37097]: pgmap v10278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18661.107465 secs 2024-07-26T13:51:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:56 smithi171 ceph-mon[37789]: pgmap v10278: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:51:58 smithi044 ceph-mon[32404]: pgmap v10279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:51:58 smithi116 ceph-mon[37097]: pgmap v10279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:51:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:51:58 smithi171 ceph-mon[37789]: pgmap v10279: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:01.160+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:52:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:01.160+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:52:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:01.160+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:00 smithi044 ceph-mon[32404]: pgmap v10280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:00 smithi116 ceph-mon[37097]: pgmap v10280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:00 smithi171 ceph-mon[37789]: pgmap v10280: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18666.107666 secs 2024-07-26T13:52:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18666.107666 secs 2024-07-26T13:52:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18666.107666 secs 2024-07-26T13:52:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:02 smithi044 ceph-mon[32404]: pgmap v10281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:03.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:02 smithi116 ceph-mon[37097]: pgmap v10281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:02 smithi171 ceph-mon[37789]: pgmap v10281: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:04 smithi044 ceph-mon[32404]: pgmap v10282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:04 smithi116 ceph-mon[37097]: pgmap v10282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:04 smithi171 ceph-mon[37789]: pgmap v10282: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:07 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18671.107829 secs 2024-07-26T13:52:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:07 smithi044 ceph-mon[32404]: pgmap v10283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:52:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:07 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18671.107829 secs 2024-07-26T13:52:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:07 smithi116 ceph-mon[37097]: pgmap v10283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:52:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:07 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18671.107829 secs 2024-07-26T13:52:07.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:07 smithi171 ceph-mon[37789]: pgmap v10283: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:52:09.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:09 smithi044 ceph-mon[32404]: pgmap v10284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:09.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:09 smithi116 ceph-mon[37097]: pgmap v10284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:09 smithi171 ceph-mon[37789]: pgmap v10284: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:11 smithi116 ceph-mon[37097]: pgmap v10285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:11 smithi171 ceph-mon[37789]: pgmap v10285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:11 smithi044 ceph-mon[32404]: pgmap v10285: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:12 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18676.107967 secs 2024-07-26T13:52:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:12 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18676.107967 secs 2024-07-26T13:52:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:12 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18676.107967 secs 2024-07-26T13:52:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:13 smithi171 ceph-mon[37789]: pgmap v10286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:13.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:13 smithi044 ceph-mon[32404]: pgmap v10286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:13.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:13 smithi116 ceph-mon[37097]: pgmap v10286: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: pgmap v10287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:52:15.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:52:15.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:15 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:52:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: pgmap v10287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:52:15.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:52:15.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:52:15.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:15 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:52:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: pgmap v10287: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.526 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:52:15.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T13:52:15.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T13:52:15.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T13:52:15.528 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:15 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T13:52:16.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:16.165+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:52:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:16.166+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:52:16.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:16.166+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:16 smithi171 ceph-mon[37789]: pgmap v10288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:52:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:16 smithi171 ceph-mon[37789]: pgmap v10289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:52:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18681.108135 secs 2024-07-26T13:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:16 smithi044 ceph-mon[32404]: pgmap v10288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:16 smithi044 ceph-mon[32404]: pgmap v10289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:52:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18681.108135 secs 2024-07-26T13:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:16 smithi116 ceph-mon[37097]: pgmap v10288: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:16 smithi116 ceph-mon[37097]: pgmap v10289: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:52:16.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18681.108135 secs 2024-07-26T13:52:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:18 smithi044 ceph-mon[32404]: pgmap v10290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:18 smithi116 ceph-mon[37097]: pgmap v10290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:18 smithi171 ceph-mon[37789]: pgmap v10290: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:20 smithi044 ceph-mon[32404]: pgmap v10291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:20.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:20 smithi116 ceph-mon[37097]: pgmap v10291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:20 smithi171 ceph-mon[37789]: pgmap v10291: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18686.108285 secs 2024-07-26T13:52:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18686.108285 secs 2024-07-26T13:52:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18686.108285 secs 2024-07-26T13:52:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:22 smithi044 ceph-mon[32404]: pgmap v10292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:52:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:22 smithi116 ceph-mon[37097]: pgmap v10292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:52:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:22 smithi171 ceph-mon[37789]: pgmap v10292: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:52:24.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:24 smithi044 ceph-mon[32404]: pgmap v10293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:24 smithi116 ceph-mon[37097]: pgmap v10293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:24.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:24 smithi171 ceph-mon[37789]: pgmap v10293: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:52:26.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:26 smithi044 ceph-mon[32404]: pgmap v10294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18691.108454 secs 2024-07-26T13:52:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:26 smithi116 ceph-mon[37097]: pgmap v10294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18691.108454 secs 2024-07-26T13:52:26.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:26 smithi171 ceph-mon[37789]: pgmap v10294: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:26.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18691.108454 secs 2024-07-26T13:52:28.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:28 smithi044 ceph-mon[32404]: pgmap v10295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:28.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:28 smithi116 ceph-mon[37097]: pgmap v10295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:28.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:28 smithi171 ceph-mon[37789]: pgmap v10295: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:30.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:30 smithi044 ceph-mon[32404]: pgmap v10296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:30.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:30 smithi116 ceph-mon[37097]: pgmap v10296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:30.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:30 smithi171 ceph-mon[37789]: pgmap v10296: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:31.166+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:52:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:31.166+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:52:31.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:31.166+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:31.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18696.108596 secs 2024-07-26T13:52:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18696.108596 secs 2024-07-26T13:52:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18696.108596 secs 2024-07-26T13:52:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:32 smithi044 ceph-mon[32404]: pgmap v10297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:32 smithi116 ceph-mon[37097]: pgmap v10297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:32 smithi171 ceph-mon[37789]: pgmap v10297: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:34.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:34 smithi044 ceph-mon[32404]: pgmap v10298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:34 smithi116 ceph-mon[37097]: pgmap v10298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:34 smithi171 ceph-mon[37789]: pgmap v10298: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:36.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:36 smithi044 ceph-mon[32404]: pgmap v10299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:36.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18701.108734 secs 2024-07-26T13:52:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:36 smithi116 ceph-mon[37097]: pgmap v10299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:36.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18701.108734 secs 2024-07-26T13:52:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:36 smithi171 ceph-mon[37789]: pgmap v10299: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18701.108734 secs 2024-07-26T13:52:38.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:38 smithi044 ceph-mon[32404]: pgmap v10300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:38.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:38 smithi116 ceph-mon[37097]: pgmap v10300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:38 smithi171 ceph-mon[37789]: pgmap v10300: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:40.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:40 smithi044 ceph-mon[32404]: pgmap v10301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:40.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:40 smithi116 ceph-mon[37097]: pgmap v10301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:40 smithi171 ceph-mon[37789]: pgmap v10301: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18706.108884 secs 2024-07-26T13:52:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18706.108884 secs 2024-07-26T13:52:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18706.108884 secs 2024-07-26T13:52:42.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:42 smithi116 ceph-mon[37097]: pgmap v10302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:42 smithi171 ceph-mon[37789]: pgmap v10302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:42 smithi044 ceph-mon[32404]: pgmap v10302: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:44 smithi171 ceph-mon[37789]: pgmap v10303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:44 smithi044 ceph-mon[32404]: pgmap v10303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:44 smithi116 ceph-mon[37097]: pgmap v10303: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:46.169+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:52:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:46.169+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:52:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:52:46.169+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:46 smithi044 ceph-mon[32404]: pgmap v10304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:46.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18711.109037 secs 2024-07-26T13:52:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:46 smithi171 ceph-mon[37789]: pgmap v10304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18711.109037 secs 2024-07-26T13:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:46 smithi116 ceph-mon[37097]: pgmap v10304: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18711.109037 secs 2024-07-26T13:52:48.574 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:48 smithi116 ceph-mon[37097]: pgmap v10305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:48.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:48 smithi171 ceph-mon[37789]: pgmap v10305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:48 smithi044 ceph-mon[32404]: pgmap v10305: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:50 smithi171 ceph-mon[37789]: pgmap v10306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:50 smithi044 ceph-mon[32404]: pgmap v10306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:50.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:50 smithi116 ceph-mon[37097]: pgmap v10306: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18716.109184 secs 2024-07-26T13:52:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18716.109184 secs 2024-07-26T13:52:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18716.109184 secs 2024-07-26T13:52:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:52 smithi171 ceph-mon[37789]: pgmap v10307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:52 smithi044 ceph-mon[32404]: pgmap v10307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:52 smithi116 ceph-mon[37097]: pgmap v10307: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:54.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:54 smithi171 ceph-mon[37789]: pgmap v10308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:54 smithi044 ceph-mon[32404]: pgmap v10308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:54 smithi116 ceph-mon[37097]: pgmap v10308: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:56 smithi171 ceph-mon[37789]: pgmap v10309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:56.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18721.109326 secs 2024-07-26T13:52:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:56 smithi044 ceph-mon[32404]: pgmap v10309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18721.109326 secs 2024-07-26T13:52:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:56 smithi116 ceph-mon[37097]: pgmap v10309: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18721.109326 secs 2024-07-26T13:52:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:52:58 smithi171 ceph-mon[37789]: pgmap v10310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:52:58 smithi044 ceph-mon[32404]: pgmap v10310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:58.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:52:58 smithi116 ceph-mon[37097]: pgmap v10310: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:00.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:00 smithi171 ceph-mon[37789]: pgmap v10311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:00 smithi044 ceph-mon[32404]: pgmap v10311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:00 smithi116 ceph-mon[37097]: pgmap v10311: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:01.171+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:53:01.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:01.171+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:53:01.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:01.171+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18726.109471 secs 2024-07-26T13:53:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18726.109471 secs 2024-07-26T13:53:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18726.109471 secs 2024-07-26T13:53:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:02 smithi044 ceph-mon[32404]: pgmap v10312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:02.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:02 smithi116 ceph-mon[37097]: pgmap v10312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:02 smithi171 ceph-mon[37789]: pgmap v10312: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:04 smithi044 ceph-mon[32404]: pgmap v10313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:04.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:04 smithi116 ceph-mon[37097]: pgmap v10313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:04 smithi171 ceph-mon[37789]: pgmap v10313: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:06 smithi044 ceph-mon[32404]: pgmap v10314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18731.109611 secs 2024-07-26T13:53:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:06 smithi116 ceph-mon[37097]: pgmap v10314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18731.109611 secs 2024-07-26T13:53:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:06 smithi171 ceph-mon[37789]: pgmap v10314: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:06.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18731.109611 secs 2024-07-26T13:53:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:08 smithi044 ceph-mon[32404]: pgmap v10315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:08 smithi116 ceph-mon[37097]: pgmap v10315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:08 smithi171 ceph-mon[37789]: pgmap v10315: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:10 smithi044 ceph-mon[32404]: pgmap v10316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:10 smithi116 ceph-mon[37097]: pgmap v10316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:10 smithi171 ceph-mon[37789]: pgmap v10316: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18736.109733 secs 2024-07-26T13:53:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18736.109733 secs 2024-07-26T13:53:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18736.109733 secs 2024-07-26T13:53:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:12 smithi044 ceph-mon[32404]: pgmap v10317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:12.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:12 smithi116 ceph-mon[37097]: pgmap v10317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:12 smithi171 ceph-mon[37789]: pgmap v10317: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:14.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:14 smithi044 ceph-mon[32404]: pgmap v10318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:14 smithi116 ceph-mon[37097]: pgmap v10318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:14.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:14 smithi171 ceph-mon[37789]: pgmap v10318: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:15 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:15 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:15 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:16.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:16.176+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:53:16.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:16.176+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:53:16.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:16.176+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:16 smithi044 ceph-mon[32404]: pgmap v10319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18741.109835 secs 2024-07-26T13:53:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:16.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:16 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:53:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:16 smithi116 ceph-mon[37097]: pgmap v10319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18741.109835 secs 2024-07-26T13:53:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:16.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:16 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:53:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:16 smithi171 ceph-mon[37789]: pgmap v10319: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18741.109835 secs 2024-07-26T13:53:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:16.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:16 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:53:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:17 smithi044 ceph-mon[32404]: pgmap v10320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:17 smithi116 ceph-mon[37097]: pgmap v10320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:17 smithi171 ceph-mon[37789]: pgmap v10320: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:18.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:18.070+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:53:18.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:18.070+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:53:18.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:18.070+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:53:18.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:18.071+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:53:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:19 smithi044 ceph-mon[32404]: pgmap v10321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:19 smithi116 ceph-mon[37097]: pgmap v10321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:19 smithi171 ceph-mon[37789]: pgmap v10321: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:21 smithi044 ceph-mon[32404]: pgmap v10322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18746.109958 secs 2024-07-26T13:53:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:21 smithi116 ceph-mon[37097]: pgmap v10322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18746.109958 secs 2024-07-26T13:53:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:21 smithi171 ceph-mon[37789]: pgmap v10322: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18746.109958 secs 2024-07-26T13:53:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:23 smithi044 ceph-mon[32404]: pgmap v10323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:23 smithi116 ceph-mon[37097]: pgmap v10323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:23 smithi171 ceph-mon[37789]: pgmap v10323: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:25.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:25 smithi044 ceph-mon[32404]: pgmap v10324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:25.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:25 smithi116 ceph-mon[37097]: pgmap v10324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:25 smithi171 ceph-mon[37789]: pgmap v10324: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18751.110091 secs 2024-07-26T13:53:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18751.110091 secs 2024-07-26T13:53:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18751.110091 secs 2024-07-26T13:53:27.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:27 smithi044 ceph-mon[32404]: pgmap v10325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:27 smithi116 ceph-mon[37097]: pgmap v10325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:27 smithi171 ceph-mon[37789]: pgmap v10325: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:29.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:29 smithi044 ceph-mon[32404]: pgmap v10326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:29.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:29 smithi116 ceph-mon[37097]: pgmap v10326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:29 smithi171 ceph-mon[37789]: pgmap v10326: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:31.177+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:53:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:31.177+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:31.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:31.177+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:53:31.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:31 smithi044 ceph-mon[32404]: pgmap v10327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:31.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18756.110251 secs 2024-07-26T13:53:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:31 smithi116 ceph-mon[37097]: pgmap v10327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18756.110251 secs 2024-07-26T13:53:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:31 smithi171 ceph-mon[37789]: pgmap v10327: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18756.110251 secs 2024-07-26T13:53:33.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:33 smithi044 ceph-mon[32404]: pgmap v10328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:33 smithi116 ceph-mon[37097]: pgmap v10328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:33 smithi171 ceph-mon[37789]: pgmap v10328: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:35.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:35 smithi044 ceph-mon[32404]: pgmap v10329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:35.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:35 smithi116 ceph-mon[37097]: pgmap v10329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:35 smithi171 ceph-mon[37789]: pgmap v10329: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18761.110414 secs 2024-07-26T13:53:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18761.110414 secs 2024-07-26T13:53:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18761.110414 secs 2024-07-26T13:53:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:37 smithi044 ceph-mon[32404]: pgmap v10330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:37 smithi116 ceph-mon[37097]: pgmap v10330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:37 smithi171 ceph-mon[37789]: pgmap v10330: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:39 smithi044 ceph-mon[32404]: pgmap v10331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:39.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:39 smithi116 ceph-mon[37097]: pgmap v10331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:39.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:39 smithi171 ceph-mon[37789]: pgmap v10331: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:41 smithi044 ceph-mon[32404]: pgmap v10332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18766.110525 secs 2024-07-26T13:53:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:41 smithi116 ceph-mon[37097]: pgmap v10332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18766.110525 secs 2024-07-26T13:53:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:41 smithi171 ceph-mon[37789]: pgmap v10332: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18766.110525 secs 2024-07-26T13:53:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:43 smithi044 ceph-mon[32404]: pgmap v10333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:43 smithi116 ceph-mon[37097]: pgmap v10333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:43 smithi171 ceph-mon[37789]: pgmap v10333: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:45 smithi044 ceph-mon[32404]: pgmap v10334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:45.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:45 smithi116 ceph-mon[37097]: pgmap v10334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:45 smithi171 ceph-mon[37789]: pgmap v10334: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:46.182+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:53:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:46.182+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:53:46.182+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:53:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18771.110678 secs 2024-07-26T13:53:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18771.110678 secs 2024-07-26T13:53:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18771.110678 secs 2024-07-26T13:53:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:47 smithi116 ceph-mon[37097]: pgmap v10335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:47 smithi171 ceph-mon[37789]: pgmap v10335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:47 smithi044 ceph-mon[32404]: pgmap v10335: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:49 smithi171 ceph-mon[37789]: pgmap v10336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:50.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:49 smithi044 ceph-mon[32404]: pgmap v10336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:50.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:49 smithi116 ceph-mon[37097]: pgmap v10336: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:51 smithi171 ceph-mon[37789]: pgmap v10337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18776.110814 secs 2024-07-26T13:53:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:51 smithi044 ceph-mon[32404]: pgmap v10337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18776.110814 secs 2024-07-26T13:53:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:51 smithi116 ceph-mon[37097]: pgmap v10337: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18776.110814 secs 2024-07-26T13:53:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:53 smithi171 ceph-mon[37789]: pgmap v10338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:53 smithi044 ceph-mon[32404]: pgmap v10338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:53 smithi116 ceph-mon[37097]: pgmap v10338: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:53:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:55 smithi171 ceph-mon[37789]: pgmap v10339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:55 smithi044 ceph-mon[32404]: pgmap v10339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:56.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:55 smithi116 ceph-mon[37097]: pgmap v10339: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18781.110977 secs 2024-07-26T13:53:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18781.110977 secs 2024-07-26T13:53:57.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18781.110977 secs 2024-07-26T13:53:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:57 smithi171 ceph-mon[37789]: pgmap v10340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:58.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:57 smithi044 ceph-mon[32404]: pgmap v10340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:57 smithi116 ceph-mon[37097]: pgmap v10340: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:53:59 smithi171 ceph-mon[37789]: pgmap v10341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:53:59 smithi044 ceph-mon[32404]: pgmap v10341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:53:59 smithi116 ceph-mon[37097]: pgmap v10341: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:01.184+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:54:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:01.184+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:01.184+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:54:01.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:01 smithi044 ceph-mon[32404]: pgmap v10342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:01.770 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18786.111140 secs 2024-07-26T13:54:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:01 smithi171 ceph-mon[37789]: pgmap v10342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18786.111140 secs 2024-07-26T13:54:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:01 smithi116 ceph-mon[37097]: pgmap v10342: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18786.111140 secs 2024-07-26T13:54:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:03 smithi044 ceph-mon[32404]: pgmap v10343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:03 smithi116 ceph-mon[37097]: pgmap v10343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:03 smithi171 ceph-mon[37789]: pgmap v10343: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:05 smithi044 ceph-mon[32404]: pgmap v10344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:05 smithi116 ceph-mon[37097]: pgmap v10344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:05 smithi171 ceph-mon[37789]: pgmap v10344: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18791.111268 secs 2024-07-26T13:54:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18791.111268 secs 2024-07-26T13:54:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18791.111268 secs 2024-07-26T13:54:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:07 smithi044 ceph-mon[32404]: pgmap v10345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:07 smithi116 ceph-mon[37097]: pgmap v10345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:07 smithi171 ceph-mon[37789]: pgmap v10345: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:09 smithi044 ceph-mon[32404]: pgmap v10346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:09 smithi116 ceph-mon[37097]: pgmap v10346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:09 smithi171 ceph-mon[37789]: pgmap v10346: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:11 smithi044 ceph-mon[32404]: pgmap v10347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18796.111430 secs 2024-07-26T13:54:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:11 smithi116 ceph-mon[37097]: pgmap v10347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18796.111430 secs 2024-07-26T13:54:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:11 smithi171 ceph-mon[37789]: pgmap v10347: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18796.111430 secs 2024-07-26T13:54:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:13 smithi044 ceph-mon[32404]: pgmap v10348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:14.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:13 smithi116 ceph-mon[37097]: pgmap v10348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:13 smithi171 ceph-mon[37789]: pgmap v10348: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:16.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:15 smithi044 ceph-mon[32404]: pgmap v10349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:15 smithi116 ceph-mon[37097]: pgmap v10349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:15 smithi171 ceph-mon[37789]: pgmap v10349: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:16.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:16.189+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:54:16.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:16.189+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:16.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:16.189+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:54:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18801.111583 secs 2024-07-26T13:54:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:16 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18801.111583 secs 2024-07-26T13:54:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:16 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18801.111583 secs 2024-07-26T13:54:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:16 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:18.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:17 smithi044 ceph-mon[32404]: pgmap v10350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:18.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:18.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:17 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:17 smithi116 ceph-mon[37097]: pgmap v10350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:17 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:17 smithi171 ceph-mon[37789]: pgmap v10350: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:17 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:18 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:19.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:18 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:18 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:18 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:18 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:18 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:19.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:19 smithi116 ceph-mon[37097]: pgmap v10351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5 KiB/s rd, 9 op/s 2024-07-26T13:54:19.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:19 smithi116 ceph-mon[37097]: pgmap v10352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T13:54:19.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:19 smithi171 ceph-mon[37789]: pgmap v10351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5 KiB/s rd, 9 op/s 2024-07-26T13:54:19.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:19 smithi171 ceph-mon[37789]: pgmap v10352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T13:54:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:19 smithi044 ceph-mon[32404]: pgmap v10351: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5 KiB/s rd, 9 op/s 2024-07-26T13:54:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:19 smithi044 ceph-mon[32404]: pgmap v10352: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T13:54:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:21 smithi044 ceph-mon[32404]: pgmap v10353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18806.111746 secs 2024-07-26T13:54:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:21 smithi116 ceph-mon[37097]: pgmap v10353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18806.111746 secs 2024-07-26T13:54:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:21 smithi171 ceph-mon[37789]: pgmap v10353: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18806.111746 secs 2024-07-26T13:54:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:23 smithi044 ceph-mon[32404]: pgmap v10354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:23 smithi116 ceph-mon[37097]: pgmap v10354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:23 smithi171 ceph-mon[37789]: pgmap v10354: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:25 smithi044 ceph-mon[32404]: pgmap v10355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:25 smithi116 ceph-mon[37097]: pgmap v10355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:25 smithi171 ceph-mon[37789]: pgmap v10355: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18811.111856 secs 2024-07-26T13:54:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18811.111856 secs 2024-07-26T13:54:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18811.111856 secs 2024-07-26T13:54:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:27 smithi044 ceph-mon[32404]: pgmap v10356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:27 smithi116 ceph-mon[37097]: pgmap v10356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:27 smithi171 ceph-mon[37789]: pgmap v10356: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:54:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:29 smithi044 ceph-mon[32404]: pgmap v10357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:54:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:29 smithi116 ceph-mon[37097]: pgmap v10357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:54:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:29 smithi171 ceph-mon[37789]: pgmap v10357: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:54:31.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:31.190+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:54:31.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:31.190+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:31.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:31.190+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:54:31.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:31 smithi044 ceph-mon[32404]: pgmap v10358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:31.970 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18816.112017 secs 2024-07-26T13:54:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:31 smithi116 ceph-mon[37097]: pgmap v10358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18816.112017 secs 2024-07-26T13:54:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:31 smithi171 ceph-mon[37789]: pgmap v10358: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18816.112017 secs 2024-07-26T13:54:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:33 smithi044 ceph-mon[32404]: pgmap v10359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:33 smithi116 ceph-mon[37097]: pgmap v10359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:34.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:33 smithi171 ceph-mon[37789]: pgmap v10359: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:35 smithi044 ceph-mon[32404]: pgmap v10360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:35 smithi116 ceph-mon[37097]: pgmap v10360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:35 smithi171 ceph-mon[37789]: pgmap v10360: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18821.112164 secs 2024-07-26T13:54:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18821.112164 secs 2024-07-26T13:54:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18821.112164 secs 2024-07-26T13:54:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:37 smithi044 ceph-mon[32404]: pgmap v10361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:37 smithi116 ceph-mon[37097]: pgmap v10361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:37 smithi171 ceph-mon[37789]: pgmap v10361: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:38 smithi044 ceph-mon[32404]: pgmap v10362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:38 smithi116 ceph-mon[37097]: pgmap v10362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:39.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:38 smithi171 ceph-mon[37789]: pgmap v10362: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:41 smithi044 ceph-mon[32404]: pgmap v10363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18826.112291 secs 2024-07-26T13:54:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:41 smithi116 ceph-mon[37097]: pgmap v10363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18826.112291 secs 2024-07-26T13:54:41.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:41 smithi171 ceph-mon[37789]: pgmap v10363: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18826.112291 secs 2024-07-26T13:54:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:43 smithi044 ceph-mon[32404]: pgmap v10364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:43 smithi116 ceph-mon[37097]: pgmap v10364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:43.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:43 smithi171 ceph-mon[37789]: pgmap v10364: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:45 smithi044 ceph-mon[32404]: pgmap v10365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:45 smithi116 ceph-mon[37097]: pgmap v10365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:45 smithi171 ceph-mon[37789]: pgmap v10365: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:46.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:46.195+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:54:46.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:46.195+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:46.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:46.195+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:54:46.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18831.112451 secs 2024-07-26T13:54:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18831.112451 secs 2024-07-26T13:54:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18831.112451 secs 2024-07-26T13:54:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:47 smithi044 ceph-mon[32404]: pgmap v10366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:47 smithi116 ceph-mon[37097]: pgmap v10366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:47 smithi171 ceph-mon[37789]: pgmap v10366: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:48.086+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:54:48.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:48.086+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:54:48.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:48.086+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf40006460 con 0x7fcf6c1057b0 2024-07-26T13:54:48.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:54:48.086+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:54:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:49 smithi044 ceph-mon[32404]: pgmap v10367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:49 smithi116 ceph-mon[37097]: pgmap v10367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:49.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:49 smithi171 ceph-mon[37789]: pgmap v10367: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:51 smithi044 ceph-mon[32404]: pgmap v10368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18836.112587 secs 2024-07-26T13:54:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:51 smithi116 ceph-mon[37097]: pgmap v10368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18836.112587 secs 2024-07-26T13:54:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:51 smithi171 ceph-mon[37789]: pgmap v10368: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:51.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18836.112587 secs 2024-07-26T13:54:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:53 smithi044 ceph-mon[32404]: pgmap v10369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:53 smithi116 ceph-mon[37097]: pgmap v10369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:53.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:53 smithi171 ceph-mon[37789]: pgmap v10369: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:54:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:55 smithi171 ceph-mon[37789]: pgmap v10370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:55 smithi044 ceph-mon[32404]: pgmap v10370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:55.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:55 smithi116 ceph-mon[37097]: pgmap v10370: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:54:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18841.112721 secs 2024-07-26T13:54:56.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18841.112721 secs 2024-07-26T13:54:56.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18841.112721 secs 2024-07-26T13:54:57.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:57 smithi044 ceph-mon[32404]: pgmap v10371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:57.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:57 smithi116 ceph-mon[37097]: pgmap v10371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:57 smithi171 ceph-mon[37789]: pgmap v10371: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:54:59 smithi044 ceph-mon[32404]: pgmap v10372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:54:59.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:54:59 smithi116 ceph-mon[37097]: pgmap v10372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:54:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:54:59 smithi171 ceph-mon[37789]: pgmap v10372: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:55:01.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:01.196+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:55:01.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:01.196+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:55:01.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:01.196+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:01.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:01 smithi044 ceph-mon[32404]: pgmap v10373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:01.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18846.112893 secs 2024-07-26T13:55:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:01 smithi116 ceph-mon[37097]: pgmap v10373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18846.112893 secs 2024-07-26T13:55:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:01 smithi171 ceph-mon[37789]: pgmap v10373: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18846.112893 secs 2024-07-26T13:55:03.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:03 smithi044 ceph-mon[32404]: pgmap v10374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:55:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:03 smithi116 ceph-mon[37097]: pgmap v10374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:55:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:03 smithi171 ceph-mon[37789]: pgmap v10374: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:55:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:05 smithi044 ceph-mon[32404]: pgmap v10375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:55:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:05 smithi116 ceph-mon[37097]: pgmap v10375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:55:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:05 smithi171 ceph-mon[37789]: pgmap v10375: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:55:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18851.113045 secs 2024-07-26T13:55:06.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18851.113045 secs 2024-07-26T13:55:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18851.113045 secs 2024-07-26T13:55:07.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:07 smithi044 ceph-mon[32404]: pgmap v10376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:55:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:07 smithi116 ceph-mon[37097]: pgmap v10376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:55:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:07 smithi171 ceph-mon[37789]: pgmap v10376: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:55:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:09 smithi116 ceph-mon[37097]: pgmap v10377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:55:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:09 smithi171 ceph-mon[37789]: pgmap v10377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:55:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:09 smithi044 ceph-mon[32404]: pgmap v10377: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:55:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:11 smithi116 ceph-mon[37097]: pgmap v10378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18856.113189 secs 2024-07-26T13:55:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:11 smithi171 ceph-mon[37789]: pgmap v10378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18856.113189 secs 2024-07-26T13:55:12.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:11 smithi044 ceph-mon[32404]: pgmap v10378: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:12.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18856.113189 secs 2024-07-26T13:55:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:13 smithi171 ceph-mon[37789]: pgmap v10379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:13 smithi044 ceph-mon[32404]: pgmap v10379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:13 smithi116 ceph-mon[37097]: pgmap v10379: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:15.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:15 smithi044 ceph-mon[32404]: pgmap v10380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:55:15.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:15 smithi116 ceph-mon[37097]: pgmap v10380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:55:15.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:15 smithi171 ceph-mon[37789]: pgmap v10380: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:55:16.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:16.199+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:55:16.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:16.199+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:16.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:16.199+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:55:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18861.113348 secs 2024-07-26T13:55:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18861.113348 secs 2024-07-26T13:55:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18861.113348 secs 2024-07-26T13:55:17.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:17 smithi044 ceph-mon[32404]: pgmap v10381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:17.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:17 smithi116 ceph-mon[37097]: pgmap v10381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:17 smithi171 ceph-mon[37789]: pgmap v10381: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:19.763 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:19 smithi171 ceph-mon[37789]: pgmap v10382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:19.763 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:19 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:19.763 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.764 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.764 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.764 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:19 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:19 smithi044 ceph-mon[32404]: pgmap v10382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:19 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:19 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:19 smithi116 ceph-mon[37097]: pgmap v10382: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:19.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:19 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:19.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:19.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:19 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:21 smithi044 ceph-mon[32404]: pgmap v10383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:21 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18866.113440 secs 2024-07-26T13:55:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:21 smithi116 ceph-mon[37097]: pgmap v10383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:21 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18866.113440 secs 2024-07-26T13:55:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:21 smithi171 ceph-mon[37789]: pgmap v10383: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:21 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18866.113440 secs 2024-07-26T13:55:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:23 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:23.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:23 smithi044 ceph-mon[32404]: pgmap v10384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:23.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:23 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:23.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:23 smithi116 ceph-mon[37097]: pgmap v10384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:23 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:23.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:23 smithi171 ceph-mon[37789]: pgmap v10384: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:24 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:24 smithi044 ceph-mon[32404]: pgmap v10385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:24.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:24 smithi044 ceph-mon[32404]: pgmap v10386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:24.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:24 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:24 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:24 smithi116 ceph-mon[37097]: pgmap v10385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:24.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:24 smithi116 ceph-mon[37097]: pgmap v10386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:24.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:24 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:24 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:24 smithi171 ceph-mon[37789]: pgmap v10385: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:24 smithi171 ceph-mon[37789]: pgmap v10386: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:24.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:24 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:26 smithi171 ceph-mon[37789]: pgmap v10387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18871.113576 secs 2024-07-26T13:55:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:26 smithi044 ceph-mon[32404]: pgmap v10387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18871.113576 secs 2024-07-26T13:55:27.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:26 smithi116 ceph-mon[37097]: pgmap v10387: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18871.113576 secs 2024-07-26T13:55:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:28 smithi171 ceph-mon[37789]: pgmap v10388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:29.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:28 smithi044 ceph-mon[32404]: pgmap v10388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:28 smithi116 ceph-mon[37097]: pgmap v10388: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:30 smithi171 ceph-mon[37789]: pgmap v10389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:31.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:30 smithi044 ceph-mon[32404]: pgmap v10389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:31.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:30 smithi116 ceph-mon[37097]: pgmap v10389: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:31.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:31.200+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:55:31.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:31.200+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:31.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:31.200+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:55:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18876.113741 secs 2024-07-26T13:55:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18876.113741 secs 2024-07-26T13:55:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18876.113741 secs 2024-07-26T13:55:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:32 smithi171 ceph-mon[37789]: pgmap v10390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:33.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:32 smithi044 ceph-mon[32404]: pgmap v10390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:33.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:32 smithi116 ceph-mon[37097]: pgmap v10390: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:34 smithi171 ceph-mon[37789]: pgmap v10391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:55:34.945 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:34 smithi044 ceph-mon[32404]: pgmap v10391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:55:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:34 smithi116 ceph-mon[37097]: pgmap v10391: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:55:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:36 smithi171 ceph-mon[37789]: pgmap v10392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18881.113902 secs 2024-07-26T13:55:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:36 smithi044 ceph-mon[32404]: pgmap v10392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:37.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18881.113902 secs 2024-07-26T13:55:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:36 smithi116 ceph-mon[37097]: pgmap v10392: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18881.113902 secs 2024-07-26T13:55:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:38 smithi171 ceph-mon[37789]: pgmap v10393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:55:39.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:38 smithi044 ceph-mon[32404]: pgmap v10393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:55:39.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:38 smithi116 ceph-mon[37097]: pgmap v10393: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:55:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:40 smithi171 ceph-mon[37789]: pgmap v10394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:55:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:40 smithi044 ceph-mon[32404]: pgmap v10394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:55:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:40 smithi116 ceph-mon[37097]: pgmap v10394: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:55:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18886.114075 secs 2024-07-26T13:55:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18886.114075 secs 2024-07-26T13:55:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18886.114075 secs 2024-07-26T13:55:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:42 smithi044 ceph-mon[32404]: pgmap v10395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:55:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:42 smithi116 ceph-mon[37097]: pgmap v10395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:55:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:42 smithi171 ceph-mon[37789]: pgmap v10395: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:55:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:44 smithi044 ceph-mon[32404]: pgmap v10396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:55:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:44 smithi116 ceph-mon[37097]: pgmap v10396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:55:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:44 smithi171 ceph-mon[37789]: pgmap v10396: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:55:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:46.205+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:55:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:46.205+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:55:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:55:46.205+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:46 smithi044 ceph-mon[32404]: pgmap v10397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18891.114168 secs 2024-07-26T13:55:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:46 smithi116 ceph-mon[37097]: pgmap v10397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18891.114168 secs 2024-07-26T13:55:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:46 smithi171 ceph-mon[37789]: pgmap v10397: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18891.114168 secs 2024-07-26T13:55:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:48 smithi044 ceph-mon[32404]: pgmap v10398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:48 smithi116 ceph-mon[37097]: pgmap v10398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:48 smithi171 ceph-mon[37789]: pgmap v10398: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:50.889 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:50 smithi171 ceph-mon[37789]: pgmap v10399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:55:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:50 smithi044 ceph-mon[32404]: pgmap v10399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:55:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:50 smithi116 ceph-mon[37097]: pgmap v10399: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:55:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18896.114314 secs 2024-07-26T13:55:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18896.114314 secs 2024-07-26T13:55:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18896.114314 secs 2024-07-26T13:55:53.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:52 smithi044 ceph-mon[32404]: pgmap v10400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:53.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:52 smithi116 ceph-mon[37097]: pgmap v10400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:52 smithi171 ceph-mon[37789]: pgmap v10400: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:54 smithi044 ceph-mon[32404]: pgmap v10401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:54 smithi116 ceph-mon[37097]: pgmap v10401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:55.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:54 smithi171 ceph-mon[37789]: pgmap v10401: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:56 smithi044 ceph-mon[32404]: pgmap v10402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18901.114476 secs 2024-07-26T13:55:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:56 smithi116 ceph-mon[37097]: pgmap v10402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18901.114476 secs 2024-07-26T13:55:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:56 smithi171 ceph-mon[37789]: pgmap v10402: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18901.114476 secs 2024-07-26T13:55:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:55:58 smithi044 ceph-mon[32404]: pgmap v10403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:55:58 smithi116 ceph-mon[37097]: pgmap v10403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:55:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:55:58 smithi171 ceph-mon[37789]: pgmap v10403: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:00 smithi044 ceph-mon[32404]: pgmap v10404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:00 smithi116 ceph-mon[37097]: pgmap v10404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:00 smithi171 ceph-mon[37789]: pgmap v10404: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:01.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:01.206+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:56:01.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:01.206+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:01.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:01.206+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:56:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18906.114672 secs 2024-07-26T13:56:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18906.114672 secs 2024-07-26T13:56:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18906.114672 secs 2024-07-26T13:56:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:02 smithi044 ceph-mon[32404]: pgmap v10405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:02 smithi116 ceph-mon[37097]: pgmap v10405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:02 smithi171 ceph-mon[37789]: pgmap v10405: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:04 smithi044 ceph-mon[32404]: pgmap v10406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:04 smithi116 ceph-mon[37097]: pgmap v10406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:04 smithi171 ceph-mon[37789]: pgmap v10406: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:06 smithi044 ceph-mon[32404]: pgmap v10407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18911.114820 secs 2024-07-26T13:56:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:06 smithi116 ceph-mon[37097]: pgmap v10407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18911.114820 secs 2024-07-26T13:56:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:06 smithi171 ceph-mon[37789]: pgmap v10407: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18911.114820 secs 2024-07-26T13:56:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:08 smithi044 ceph-mon[32404]: pgmap v10408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:08 smithi116 ceph-mon[37097]: pgmap v10408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:08 smithi171 ceph-mon[37789]: pgmap v10408: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:11.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:10 smithi044 ceph-mon[32404]: pgmap v10409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:11.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:10 smithi116 ceph-mon[37097]: pgmap v10409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:10 smithi171 ceph-mon[37789]: pgmap v10409: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18916.114918 secs 2024-07-26T13:56:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18916.114918 secs 2024-07-26T13:56:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18916.114918 secs 2024-07-26T13:56:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:12 smithi044 ceph-mon[32404]: pgmap v10410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:13.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:12 smithi116 ceph-mon[37097]: pgmap v10410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:12 smithi171 ceph-mon[37789]: pgmap v10410: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:14.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.779+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:56:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.779+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:56:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.779+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf580318f0 tx=0x7fcf58087040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:56:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.780+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf580318f0 tx=0x7fcf58087040 comp rx=0 tx=0).stop 2024-07-26T13:56:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.780+0000 7fcf71200640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.44:6800/2852968588, fault. 2024-07-26T13:56:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.780+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 msgr2=0x7fcf540c42d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:56:14.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.780+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540c1eb0 0x7fcf540c42d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:56:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.780+0000 7fcf6a000640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:56:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.781+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:56:14.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:14.782+0000 7fcf71200640 1 --2- 172.21.15.44:0/1872798322 >> [v2:172.21.15.44:6800/2852968588,v1:172.21.15.44:6801/2852968588] conn(0x7fcf540bf8c0 0x7fcf540c4810 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcf54003700 tx=0x7fcf58074040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:56:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:14 smithi044 ceph-mon[32404]: pgmap v10411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:56:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:14 smithi116 ceph-mon[37097]: pgmap v10411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:56:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:14 smithi171 ceph-mon[37789]: pgmap v10411: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:56:16.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:16.207+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:56:16.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:16.208+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:56:16.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:16.208+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:16 smithi044 ceph-mon[32404]: pgmap v10412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18921.115086 secs 2024-07-26T13:56:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:16 smithi116 ceph-mon[37097]: pgmap v10412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18921.115086 secs 2024-07-26T13:56:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:16 smithi171 ceph-mon[37789]: pgmap v10412: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18921.115086 secs 2024-07-26T13:56:18.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:18.102+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T13:56:18.103 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:18.102+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:56:18.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:18.102+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T13:56:18.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:18.103+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:56:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:18 smithi044 ceph-mon[32404]: pgmap v10413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:19.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:18 smithi116 ceph-mon[37097]: pgmap v10413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:18 smithi171 ceph-mon[37789]: pgmap v10413: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:20 smithi044 ceph-mon[32404]: pgmap v10414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:20 smithi116 ceph-mon[37097]: pgmap v10414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:20 smithi171 ceph-mon[37789]: pgmap v10414: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18926.115241 secs 2024-07-26T13:56:22.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18926.115241 secs 2024-07-26T13:56:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18926.115241 secs 2024-07-26T13:56:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:22 smithi044 ceph-mon[32404]: pgmap v10415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:22 smithi116 ceph-mon[37097]: pgmap v10415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:22 smithi171 ceph-mon[37789]: pgmap v10415: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:23 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:24.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:23 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:23 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:25.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:24 smithi044 ceph-mon[32404]: pgmap v10416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:56:25.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:24 smithi116 ceph-mon[37097]: pgmap v10416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:56:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:24 smithi171 ceph-mon[37789]: pgmap v10416: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:56:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:25 smithi044 ceph-mon[32404]: pgmap v10417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:56:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:25 smithi044 ceph-mon[32404]: pgmap v10418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:26.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:25 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:25 smithi116 ceph-mon[37097]: pgmap v10417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:25 smithi116 ceph-mon[37097]: pgmap v10418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:25 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:25 smithi171 ceph-mon[37789]: pgmap v10417: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:25 smithi171 ceph-mon[37789]: pgmap v10418: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:25 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18931.115332 secs 2024-07-26T13:56:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18931.115332 secs 2024-07-26T13:56:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18931.115332 secs 2024-07-26T13:56:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:27 smithi171 ceph-mon[37789]: pgmap v10419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:27 smithi044 ceph-mon[32404]: pgmap v10419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:27 smithi116 ceph-mon[37097]: pgmap v10419: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:29 smithi044 ceph-mon[32404]: pgmap v10420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:30.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:29 smithi116 ceph-mon[37097]: pgmap v10420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:29 smithi171 ceph-mon[37789]: pgmap v10420: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:31.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:31.209+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:56:31.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:31.209+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:56:31.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:31.209+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18936.115496 secs 2024-07-26T13:56:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:31 smithi044 ceph-mon[32404]: pgmap v10421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18936.115496 secs 2024-07-26T13:56:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:31 smithi116 ceph-mon[37097]: pgmap v10421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18936.115496 secs 2024-07-26T13:56:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:31 smithi171 ceph-mon[37789]: pgmap v10421: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:56:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:33 smithi044 ceph-mon[32404]: pgmap v10422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:56:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:33 smithi116 ceph-mon[37097]: pgmap v10422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:56:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:33 smithi171 ceph-mon[37789]: pgmap v10422: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:56:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:35 smithi044 ceph-mon[32404]: pgmap v10423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:35 smithi116 ceph-mon[37097]: pgmap v10423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:35 smithi171 ceph-mon[37789]: pgmap v10423: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18941.115669 secs 2024-07-26T13:56:37.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18941.115669 secs 2024-07-26T13:56:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18941.115669 secs 2024-07-26T13:56:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:37 smithi044 ceph-mon[32404]: pgmap v10424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:37 smithi116 ceph-mon[37097]: pgmap v10424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:37 smithi171 ceph-mon[37789]: pgmap v10424: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:39 smithi044 ceph-mon[32404]: pgmap v10425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:39 smithi116 ceph-mon[37097]: pgmap v10425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:39 smithi171 ceph-mon[37789]: pgmap v10425: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18946.115776 secs 2024-07-26T13:56:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:41 smithi044 ceph-mon[32404]: pgmap v10426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18946.115776 secs 2024-07-26T13:56:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:41 smithi116 ceph-mon[37097]: pgmap v10426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18946.115776 secs 2024-07-26T13:56:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:41 smithi171 ceph-mon[37789]: pgmap v10426: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:43 smithi044 ceph-mon[32404]: pgmap v10427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:43 smithi116 ceph-mon[37097]: pgmap v10427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:43 smithi171 ceph-mon[37789]: pgmap v10427: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:46.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:46.215+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:56:46.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:46.215+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:56:46.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:56:46.215+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:46.217 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:45 smithi044 ceph-mon[32404]: pgmap v10428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:45 smithi116 ceph-mon[37097]: pgmap v10428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:45 smithi171 ceph-mon[37789]: pgmap v10428: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18951.115868 secs 2024-07-26T13:56:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18951.115868 secs 2024-07-26T13:56:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18951.115868 secs 2024-07-26T13:56:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:47 smithi044 ceph-mon[32404]: pgmap v10429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:47 smithi116 ceph-mon[37097]: pgmap v10429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:47 smithi171 ceph-mon[37789]: pgmap v10429: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:49 smithi044 ceph-mon[32404]: pgmap v10430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:50.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:49 smithi116 ceph-mon[37097]: pgmap v10430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:49 smithi171 ceph-mon[37789]: pgmap v10430: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18956.116029 secs 2024-07-26T13:56:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:51 smithi044 ceph-mon[32404]: pgmap v10431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18956.116029 secs 2024-07-26T13:56:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:51 smithi116 ceph-mon[37097]: pgmap v10431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18956.116029 secs 2024-07-26T13:56:52.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:51 smithi171 ceph-mon[37789]: pgmap v10431: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:53 smithi044 ceph-mon[32404]: pgmap v10432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:53 smithi116 ceph-mon[37097]: pgmap v10432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:53 smithi171 ceph-mon[37789]: pgmap v10432: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:55 smithi044 ceph-mon[32404]: pgmap v10433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:56.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:55 smithi116 ceph-mon[37097]: pgmap v10433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:55 smithi171 ceph-mon[37789]: pgmap v10433: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18961.116190 secs 2024-07-26T13:56:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18961.116190 secs 2024-07-26T13:56:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18961.116190 secs 2024-07-26T13:56:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:57 smithi044 ceph-mon[32404]: pgmap v10434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:57 smithi116 ceph-mon[37097]: pgmap v10434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:57 smithi171 ceph-mon[37789]: pgmap v10434: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:00.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:56:59 smithi044 ceph-mon[32404]: pgmap v10435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:00.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:56:59 smithi116 ceph-mon[37097]: pgmap v10435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:56:59 smithi171 ceph-mon[37789]: pgmap v10435: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:01.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:01.216+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:57:01.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:01.216+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:57:01.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:01.216+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18966.116343 secs 2024-07-26T13:57:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:01 smithi044 ceph-mon[32404]: pgmap v10436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18966.116343 secs 2024-07-26T13:57:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:01 smithi116 ceph-mon[37097]: pgmap v10436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18966.116343 secs 2024-07-26T13:57:02.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:01 smithi171 ceph-mon[37789]: pgmap v10436: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:03 smithi044 ceph-mon[32404]: pgmap v10437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:04.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:03 smithi116 ceph-mon[37097]: pgmap v10437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:03 smithi171 ceph-mon[37789]: pgmap v10437: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:06 smithi044 ceph-mon[32404]: pgmap v10438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:06.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18971.116515 secs 2024-07-26T13:57:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:06 smithi116 ceph-mon[37097]: pgmap v10438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:06.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18971.116515 secs 2024-07-26T13:57:06.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:06 smithi171 ceph-mon[37789]: pgmap v10438: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:06.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18971.116515 secs 2024-07-26T13:57:08.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:08 smithi044 ceph-mon[32404]: pgmap v10439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:08.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:08 smithi116 ceph-mon[37097]: pgmap v10439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:08.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:08 smithi171 ceph-mon[37789]: pgmap v10439: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:10.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:10 smithi044 ceph-mon[32404]: pgmap v10440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:10.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:10 smithi116 ceph-mon[37097]: pgmap v10440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:10.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:10 smithi171 ceph-mon[37789]: pgmap v10440: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18976.116704 secs 2024-07-26T13:57:11.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18976.116704 secs 2024-07-26T13:57:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18976.116704 secs 2024-07-26T13:57:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:12 smithi044 ceph-mon[32404]: pgmap v10441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:12.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:12 smithi116 ceph-mon[37097]: pgmap v10441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:12 smithi171 ceph-mon[37789]: pgmap v10441: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:14.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:14 smithi044 ceph-mon[32404]: pgmap v10442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:14.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:14 smithi116 ceph-mon[37097]: pgmap v10442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:14.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:14 smithi171 ceph-mon[37789]: pgmap v10442: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:16.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:16.221+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:57:16.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:16.222+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:57:16.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:16.222+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:16 smithi044 ceph-mon[32404]: pgmap v10443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:16.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18981.116865 secs 2024-07-26T13:57:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:16 smithi116 ceph-mon[37097]: pgmap v10443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:16.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18981.116865 secs 2024-07-26T13:57:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:16 smithi171 ceph-mon[37789]: pgmap v10443: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:16.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18981.116865 secs 2024-07-26T13:57:18.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:18 smithi044 ceph-mon[32404]: pgmap v10444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:18.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:18 smithi116 ceph-mon[37097]: pgmap v10444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:18.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:18 smithi171 ceph-mon[37789]: pgmap v10444: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:20.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:20 smithi044 ceph-mon[32404]: pgmap v10445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:20.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:20 smithi116 ceph-mon[37097]: pgmap v10445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:20.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:20 smithi171 ceph-mon[37789]: pgmap v10445: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18986.117009 secs 2024-07-26T13:57:21.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18986.117009 secs 2024-07-26T13:57:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18986.117009 secs 2024-07-26T13:57:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:22 smithi044 ceph-mon[32404]: pgmap v10446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:22 smithi116 ceph-mon[37097]: pgmap v10446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:22 smithi171 ceph-mon[37789]: pgmap v10446: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:24.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:24 smithi044 ceph-mon[32404]: pgmap v10447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:24.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:24 smithi116 ceph-mon[37097]: pgmap v10447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:24 smithi171 ceph-mon[37789]: pgmap v10447: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:25 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:25.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:25 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:25 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:26 smithi044 ceph-mon[32404]: pgmap v10448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:57:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18991.117891 secs 2024-07-26T13:57:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:26.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:26.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:26 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:26 smithi116 ceph-mon[37097]: pgmap v10448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:57:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18991.117891 secs 2024-07-26T13:57:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:26.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:26 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:26 smithi171 ceph-mon[37789]: pgmap v10448: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:57:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18991.117891 secs 2024-07-26T13:57:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:26 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:27 smithi044 ceph-mon[32404]: pgmap v10449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:57:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:27 smithi044 ceph-mon[32404]: pgmap v10450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:57:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:27 smithi116 ceph-mon[37097]: pgmap v10449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:57:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:27 smithi116 ceph-mon[37097]: pgmap v10450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:57:27.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:27 smithi171 ceph-mon[37789]: pgmap v10449: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:57:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:27 smithi171 ceph-mon[37789]: pgmap v10450: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:57:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:29 smithi044 ceph-mon[32404]: pgmap v10451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:29 smithi116 ceph-mon[37097]: pgmap v10451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:29.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:29 smithi171 ceph-mon[37789]: pgmap v10451: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:31.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:31.223+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:57:31.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:31.223+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:57:31.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:31.223+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:31 smithi044 ceph-mon[32404]: pgmap v10452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:31.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 18996.118024 secs 2024-07-26T13:57:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:31 smithi116 ceph-mon[37097]: pgmap v10452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 18996.118024 secs 2024-07-26T13:57:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:31 smithi171 ceph-mon[37789]: pgmap v10452: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:31.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 18996.118024 secs 2024-07-26T13:57:33.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:33 smithi044 ceph-mon[32404]: pgmap v10453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:57:33.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:33 smithi116 ceph-mon[37097]: pgmap v10453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:57:33.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:33 smithi171 ceph-mon[37789]: pgmap v10453: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:57:35.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:35 smithi044 ceph-mon[32404]: pgmap v10454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:57:35.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:35 smithi116 ceph-mon[37097]: pgmap v10454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:57:35.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:35 smithi171 ceph-mon[37789]: pgmap v10454: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:57:36.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19001.118154 secs 2024-07-26T13:57:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19001.118154 secs 2024-07-26T13:57:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19001.118154 secs 2024-07-26T13:57:37.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:37 smithi171 ceph-mon[37789]: pgmap v10455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:37.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:37 smithi044 ceph-mon[32404]: pgmap v10455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:37.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:37 smithi116 ceph-mon[37097]: pgmap v10455: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:39.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:39 smithi171 ceph-mon[37789]: pgmap v10456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:39.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:39 smithi044 ceph-mon[32404]: pgmap v10456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:39.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:39 smithi116 ceph-mon[37097]: pgmap v10456: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:41 smithi171 ceph-mon[37789]: pgmap v10457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19006.118287 secs 2024-07-26T13:57:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:41 smithi044 ceph-mon[32404]: pgmap v10457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:41.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19006.118287 secs 2024-07-26T13:57:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:41 smithi116 ceph-mon[37097]: pgmap v10457: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:41.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19006.118287 secs 2024-07-26T13:57:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:43 smithi171 ceph-mon[37789]: pgmap v10458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:43 smithi044 ceph-mon[32404]: pgmap v10458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:43 smithi116 ceph-mon[37097]: pgmap v10458: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:45 smithi171 ceph-mon[37789]: pgmap v10459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:45.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:45 smithi044 ceph-mon[32404]: pgmap v10459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:45.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:45 smithi116 ceph-mon[37097]: pgmap v10459: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:46.229+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:57:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:46.229+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:57:46.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:57:46.229+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:46.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19011.118426 secs 2024-07-26T13:57:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19011.118426 secs 2024-07-26T13:57:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19011.118426 secs 2024-07-26T13:57:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:47 smithi171 ceph-mon[37789]: pgmap v10460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:57:47.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:47 smithi044 ceph-mon[32404]: pgmap v10460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:57:47.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:47 smithi116 ceph-mon[37097]: pgmap v10460: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:57:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:49 smithi171 ceph-mon[37789]: pgmap v10461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:57:49.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:49 smithi044 ceph-mon[32404]: pgmap v10461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:57:49.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:49 smithi116 ceph-mon[37097]: pgmap v10461: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:57:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:51 smithi171 ceph-mon[37789]: pgmap v10462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:57:51.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19016.118527 secs 2024-07-26T13:57:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:51 smithi044 ceph-mon[32404]: pgmap v10462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:57:51.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19016.118527 secs 2024-07-26T13:57:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:51 smithi116 ceph-mon[37097]: pgmap v10462: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:57:51.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19016.118527 secs 2024-07-26T13:57:53.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:53 smithi171 ceph-mon[37789]: pgmap v10463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:53.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:53 smithi044 ceph-mon[32404]: pgmap v10463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:53.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:53 smithi116 ceph-mon[37097]: pgmap v10463: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:55 smithi044 ceph-mon[32404]: pgmap v10464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:55.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:55 smithi116 ceph-mon[37097]: pgmap v10464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:55 smithi171 ceph-mon[37789]: pgmap v10464: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19021.118926 secs 2024-07-26T13:57:56.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19021.118926 secs 2024-07-26T13:57:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19021.118926 secs 2024-07-26T13:57:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:57 smithi044 ceph-mon[32404]: pgmap v10465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:57.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:57 smithi116 ceph-mon[37097]: pgmap v10465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:57 smithi171 ceph-mon[37789]: pgmap v10465: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:59.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:57:59 smithi044 ceph-mon[32404]: pgmap v10466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:59.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:57:59 smithi116 ceph-mon[37097]: pgmap v10466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:57:59 smithi171 ceph-mon[37789]: pgmap v10466: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:01.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:01.229+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:58:01.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:01.229+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:58:01.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:01.229+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:01 smithi044 ceph-mon[32404]: pgmap v10467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:01.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19026.119043 secs 2024-07-26T13:58:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:01 smithi116 ceph-mon[37097]: pgmap v10467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:01.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19026.119043 secs 2024-07-26T13:58:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:01 smithi171 ceph-mon[37789]: pgmap v10467: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:01.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19026.119043 secs 2024-07-26T13:58:03.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:03 smithi044 ceph-mon[32404]: pgmap v10468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:03.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:03 smithi116 ceph-mon[37097]: pgmap v10468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:03.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:03 smithi171 ceph-mon[37789]: pgmap v10468: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:05.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:05 smithi044 ceph-mon[32404]: pgmap v10469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:05.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:05 smithi116 ceph-mon[37097]: pgmap v10469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:05.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:05 smithi171 ceph-mon[37789]: pgmap v10469: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:06.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19031.119174 secs 2024-07-26T13:58:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19031.119174 secs 2024-07-26T13:58:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19031.119174 secs 2024-07-26T13:58:07.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:07 smithi044 ceph-mon[32404]: pgmap v10470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:07 smithi116 ceph-mon[37097]: pgmap v10470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:07.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:07 smithi171 ceph-mon[37789]: pgmap v10470: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:09.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:09 smithi044 ceph-mon[32404]: pgmap v10471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:09.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:09 smithi116 ceph-mon[37097]: pgmap v10471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:09.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:09 smithi171 ceph-mon[37789]: pgmap v10471: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:11 smithi044 ceph-mon[32404]: pgmap v10472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:11.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19036.119286 secs 2024-07-26T13:58:11.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:11 smithi116 ceph-mon[37097]: pgmap v10472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19036.119286 secs 2024-07-26T13:58:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:11 smithi171 ceph-mon[37789]: pgmap v10472: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19036.119286 secs 2024-07-26T13:58:13.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:13 smithi044 ceph-mon[32404]: pgmap v10473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:13.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:13 smithi116 ceph-mon[37097]: pgmap v10473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:13.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:13 smithi171 ceph-mon[37789]: pgmap v10473: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:15.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:15 smithi044 ceph-mon[32404]: pgmap v10474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:15 smithi116 ceph-mon[37097]: pgmap v10474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:15.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:15 smithi171 ceph-mon[37789]: pgmap v10474: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:16.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:16.234+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:58:16.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:16.235+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:58:16.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:16.235+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:16.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19041.119396 secs 2024-07-26T13:58:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19041.119396 secs 2024-07-26T13:58:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19041.119396 secs 2024-07-26T13:58:17.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:17 smithi044 ceph-mon[32404]: pgmap v10475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:17.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:17 smithi116 ceph-mon[37097]: pgmap v10475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:17.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:17 smithi171 ceph-mon[37789]: pgmap v10475: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:19.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:19 smithi044 ceph-mon[32404]: pgmap v10476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:19.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:19 smithi116 ceph-mon[37097]: pgmap v10476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:19.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:19 smithi171 ceph-mon[37789]: pgmap v10476: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:21 smithi044 ceph-mon[32404]: pgmap v10477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:21.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19046.119496 secs 2024-07-26T13:58:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:21 smithi116 ceph-mon[37097]: pgmap v10477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19046.119496 secs 2024-07-26T13:58:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:21 smithi171 ceph-mon[37789]: pgmap v10477: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19046.119496 secs 2024-07-26T13:58:23.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:23 smithi044 ceph-mon[32404]: pgmap v10478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:23.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:23 smithi116 ceph-mon[37097]: pgmap v10478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:23 smithi171 ceph-mon[37789]: pgmap v10478: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:25.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:25 smithi044 ceph-mon[32404]: pgmap v10479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:25.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:25 smithi116 ceph-mon[37097]: pgmap v10479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:25.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:25 smithi171 ceph-mon[37789]: pgmap v10479: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19051.119581 secs 2024-07-26T13:58:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:26 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19051.119581 secs 2024-07-26T13:58:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:26 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19051.119581 secs 2024-07-26T13:58:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:26 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:27 smithi044 ceph-mon[32404]: pgmap v10480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:27.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:27 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:58:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:27 smithi116 ceph-mon[37097]: pgmap v10480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:27.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:27 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:58:27.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:27 smithi171 ceph-mon[37789]: pgmap v10480: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:27.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:27 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:58:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:28 smithi044 ceph-mon[32404]: pgmap v10481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:58:28.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:28 smithi044 ceph-mon[32404]: pgmap v10482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:28 smithi116 ceph-mon[37097]: pgmap v10481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:58:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:28 smithi116 ceph-mon[37097]: pgmap v10482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:28 smithi171 ceph-mon[37789]: pgmap v10481: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:58:28.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:28 smithi171 ceph-mon[37789]: pgmap v10482: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:30 smithi044 ceph-mon[32404]: pgmap v10483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:30 smithi116 ceph-mon[37097]: pgmap v10483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:30 smithi171 ceph-mon[37789]: pgmap v10483: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:31.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:31.237+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:58:31.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:31.237+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:58:31.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:31.237+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19056.119676 secs 2024-07-26T13:58:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19056.119676 secs 2024-07-26T13:58:31.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19056.119676 secs 2024-07-26T13:58:32.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:32 smithi044 ceph-mon[32404]: pgmap v10484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:32.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:32 smithi116 ceph-mon[37097]: pgmap v10484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:32 smithi171 ceph-mon[37789]: pgmap v10484: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:34 smithi044 ceph-mon[32404]: pgmap v10485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:34.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:34 smithi116 ceph-mon[37097]: pgmap v10485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:34.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:34 smithi171 ceph-mon[37789]: pgmap v10485: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:36.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:36 smithi044 ceph-mon[32404]: pgmap v10486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19061.119762 secs 2024-07-26T13:58:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:36 smithi116 ceph-mon[37097]: pgmap v10486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19061.119762 secs 2024-07-26T13:58:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:36 smithi171 ceph-mon[37789]: pgmap v10486: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:36.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19061.119762 secs 2024-07-26T13:58:38.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:38 smithi044 ceph-mon[32404]: pgmap v10487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:58:38.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:38 smithi116 ceph-mon[37097]: pgmap v10487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:58:38.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:38 smithi171 ceph-mon[37789]: pgmap v10487: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:58:40.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:40 smithi044 ceph-mon[32404]: pgmap v10488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:40 smithi116 ceph-mon[37097]: pgmap v10488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:40.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:40 smithi171 ceph-mon[37789]: pgmap v10488: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:41.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19066.119846 secs 2024-07-26T13:58:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19066.119846 secs 2024-07-26T13:58:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19066.119846 secs 2024-07-26T13:58:42.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:42 smithi171 ceph-mon[37789]: pgmap v10489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:42 smithi044 ceph-mon[32404]: pgmap v10489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:42 smithi116 ceph-mon[37097]: pgmap v10489: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:44.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:44 smithi171 ceph-mon[37789]: pgmap v10490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:44 smithi044 ceph-mon[32404]: pgmap v10490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:44 smithi116 ceph-mon[37097]: pgmap v10490: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:46.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:46.242+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:58:46.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:46.242+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:58:46.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:58:46.242+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:46 smithi171 ceph-mon[37789]: pgmap v10491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:46.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19071.119935 secs 2024-07-26T13:58:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:46 smithi044 ceph-mon[32404]: pgmap v10491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19071.119935 secs 2024-07-26T13:58:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:46 smithi116 ceph-mon[37097]: pgmap v10491: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19071.119935 secs 2024-07-26T13:58:48.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:48 smithi171 ceph-mon[37789]: pgmap v10492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:49.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:48 smithi044 ceph-mon[32404]: pgmap v10492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:48 smithi116 ceph-mon[37097]: pgmap v10492: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:50.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:50 smithi171 ceph-mon[37789]: pgmap v10493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:50 smithi044 ceph-mon[32404]: pgmap v10493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:51.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:50 smithi116 ceph-mon[37097]: pgmap v10493: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:58:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19076.120101 secs 2024-07-26T13:58:51.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19076.120101 secs 2024-07-26T13:58:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19076.120101 secs 2024-07-26T13:58:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:52 smithi171 ceph-mon[37789]: pgmap v10494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:53.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:52 smithi044 ceph-mon[32404]: pgmap v10494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:52 smithi116 ceph-mon[37097]: pgmap v10494: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:54.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:54 smithi171 ceph-mon[37789]: pgmap v10495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:55.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:54 smithi044 ceph-mon[32404]: pgmap v10495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:55.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:54 smithi116 ceph-mon[37097]: pgmap v10495: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:56 smithi171 ceph-mon[37789]: pgmap v10496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19081.120245 secs 2024-07-26T13:58:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:56 smithi044 ceph-mon[32404]: pgmap v10496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19081.120245 secs 2024-07-26T13:58:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:56 smithi116 ceph-mon[37097]: pgmap v10496: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19081.120245 secs 2024-07-26T13:58:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:58:58 smithi171 ceph-mon[37789]: pgmap v10497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:58:58 smithi044 ceph-mon[32404]: pgmap v10497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:58:58 smithi116 ceph-mon[37097]: pgmap v10497: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:01.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:00 smithi044 ceph-mon[32404]: pgmap v10498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:00 smithi116 ceph-mon[37097]: pgmap v10498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:00 smithi171 ceph-mon[37789]: pgmap v10498: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:01.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:01.242+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:59:01.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:01.242+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:59:01.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:01.242+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19086.120389 secs 2024-07-26T13:59:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19086.120389 secs 2024-07-26T13:59:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19086.120389 secs 2024-07-26T13:59:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:02 smithi044 ceph-mon[32404]: pgmap v10499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:03.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:02 smithi116 ceph-mon[37097]: pgmap v10499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:02 smithi171 ceph-mon[37789]: pgmap v10499: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:04 smithi044 ceph-mon[32404]: pgmap v10500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:05.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:04 smithi116 ceph-mon[37097]: pgmap v10500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:04 smithi171 ceph-mon[37789]: pgmap v10500: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:06 smithi044 ceph-mon[32404]: pgmap v10501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19091.120537 secs 2024-07-26T13:59:07.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:06 smithi116 ceph-mon[37097]: pgmap v10501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19091.120537 secs 2024-07-26T13:59:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:06 smithi171 ceph-mon[37789]: pgmap v10501: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19091.120537 secs 2024-07-26T13:59:09.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:08 smithi044 ceph-mon[32404]: pgmap v10502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:09.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:08 smithi116 ceph-mon[37097]: pgmap v10502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:08 smithi171 ceph-mon[37789]: pgmap v10502: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:11.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:10 smithi044 ceph-mon[32404]: pgmap v10503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:11.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:10 smithi116 ceph-mon[37097]: pgmap v10503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:10 smithi171 ceph-mon[37789]: pgmap v10503: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19096.120694 secs 2024-07-26T13:59:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19096.120694 secs 2024-07-26T13:59:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19096.120694 secs 2024-07-26T13:59:13.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:12 smithi044 ceph-mon[32404]: pgmap v10504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:13.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:12 smithi116 ceph-mon[37097]: pgmap v10504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:12 smithi171 ceph-mon[37789]: pgmap v10504: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:15.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:14 smithi044 ceph-mon[32404]: pgmap v10505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:15.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:14 smithi116 ceph-mon[37097]: pgmap v10505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:14 smithi171 ceph-mon[37789]: pgmap v10505: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:16.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:16.247+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:59:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:16.248+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:59:16.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:16.248+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:16 smithi044 ceph-mon[32404]: pgmap v10506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:17.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19101.120863 secs 2024-07-26T13:59:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:16 smithi116 ceph-mon[37097]: pgmap v10506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19101.120863 secs 2024-07-26T13:59:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:16 smithi171 ceph-mon[37789]: pgmap v10506: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19101.120863 secs 2024-07-26T13:59:18.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:18.135+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:59:18.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:18.135+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T13:59:18.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:18.135+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T13:59:18.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:18.136+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T13:59:19.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:18 smithi044 ceph-mon[32404]: pgmap v10507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:19.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:18 smithi116 ceph-mon[37097]: pgmap v10507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:18 smithi171 ceph-mon[37789]: pgmap v10507: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:21.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:20 smithi044 ceph-mon[32404]: pgmap v10508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:21.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:20 smithi116 ceph-mon[37097]: pgmap v10508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:20 smithi171 ceph-mon[37789]: pgmap v10508: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:21.988 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19106.120963 secs 2024-07-26T13:59:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19106.120963 secs 2024-07-26T13:59:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19106.120963 secs 2024-07-26T13:59:23.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:22 smithi044 ceph-mon[32404]: pgmap v10509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:23.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:22 smithi116 ceph-mon[37097]: pgmap v10509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:22 smithi171 ceph-mon[37789]: pgmap v10509: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:25.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:24 smithi044 ceph-mon[32404]: pgmap v10510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:25.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:24 smithi116 ceph-mon[37097]: pgmap v10510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:24 smithi171 ceph-mon[37789]: pgmap v10510: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:26 smithi044 ceph-mon[32404]: pgmap v10511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19111.121172 secs 2024-07-26T13:59:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:26 smithi116 ceph-mon[37097]: pgmap v10511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19111.121172 secs 2024-07-26T13:59:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:26 smithi171 ceph-mon[37789]: pgmap v10511: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:27.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19111.121172 secs 2024-07-26T13:59:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:27 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:28.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:27 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:27 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:28 smithi044 ceph-mon[32404]: pgmap v10512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:29.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:28 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:29.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:28 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:59:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:28 smithi116 ceph-mon[37097]: pgmap v10512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:29.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:28 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:29.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:28 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:59:29.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:28 smithi171 ceph-mon[37789]: pgmap v10512: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:29.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:29.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:28 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:29.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:28 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:59:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:29 smithi044 ceph-mon[32404]: pgmap v10513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:30.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:29 smithi044 ceph-mon[32404]: pgmap v10514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:29 smithi116 ceph-mon[37097]: pgmap v10513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:29 smithi116 ceph-mon[37097]: pgmap v10514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:29 smithi171 ceph-mon[37789]: pgmap v10513: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:29 smithi171 ceph-mon[37789]: pgmap v10514: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:31.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:31.249+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:59:31.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:31.249+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:59:31.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:31.249+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:31 smithi044 ceph-mon[32404]: pgmap v10515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19116.121309 secs 2024-07-26T13:59:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:31 smithi116 ceph-mon[37097]: pgmap v10515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19116.121309 secs 2024-07-26T13:59:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:31 smithi171 ceph-mon[37789]: pgmap v10515: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19116.121309 secs 2024-07-26T13:59:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:33 smithi044 ceph-mon[32404]: pgmap v10516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:59:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:33 smithi116 ceph-mon[37097]: pgmap v10516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:59:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:33 smithi171 ceph-mon[37789]: pgmap v10516: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:59:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:35 smithi044 ceph-mon[32404]: pgmap v10517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:36.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:35 smithi116 ceph-mon[37097]: pgmap v10517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:35 smithi171 ceph-mon[37789]: pgmap v10517: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:59:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19121.121472 secs 2024-07-26T13:59:37.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19121.121472 secs 2024-07-26T13:59:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19121.121472 secs 2024-07-26T13:59:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:37 smithi044 ceph-mon[32404]: pgmap v10518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:59:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:37 smithi116 ceph-mon[37097]: pgmap v10518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:59:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:37 smithi171 ceph-mon[37789]: pgmap v10518: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:59:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:39 smithi044 ceph-mon[32404]: pgmap v10519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:39 smithi116 ceph-mon[37097]: pgmap v10519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:39 smithi171 ceph-mon[37789]: pgmap v10519: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:42.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:41 smithi044 ceph-mon[32404]: pgmap v10520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19126.121672 secs 2024-07-26T13:59:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:41 smithi116 ceph-mon[37097]: pgmap v10520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19126.121672 secs 2024-07-26T13:59:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:41 smithi171 ceph-mon[37789]: pgmap v10520: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19126.121672 secs 2024-07-26T13:59:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:43 smithi116 ceph-mon[37097]: pgmap v10521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:43 smithi171 ceph-mon[37789]: pgmap v10521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:43 smithi044 ceph-mon[32404]: pgmap v10521: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:45 smithi116 ceph-mon[37097]: pgmap v10522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:45 smithi171 ceph-mon[37789]: pgmap v10522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:46.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:46.255+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T13:59:46.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:46.255+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T13:59:46.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T13:59:46.255+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:46.257 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:45 smithi044 ceph-mon[32404]: pgmap v10522: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19131.121778 secs 2024-07-26T13:59:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19131.121778 secs 2024-07-26T13:59:47.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19131.121778 secs 2024-07-26T13:59:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:47 smithi171 ceph-mon[37789]: pgmap v10523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:47 smithi044 ceph-mon[32404]: pgmap v10523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:47 smithi116 ceph-mon[37097]: pgmap v10523: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:49 smithi171 ceph-mon[37789]: pgmap v10524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:59:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:49 smithi044 ceph-mon[32404]: pgmap v10524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:59:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:49 smithi116 ceph-mon[37097]: pgmap v10524: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:59:51.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:50 smithi116 ceph-mon[37097]: pgmap v10525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:51.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:50 smithi044 ceph-mon[32404]: pgmap v10525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:50 smithi171 ceph-mon[37789]: pgmap v10525: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:52.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19136.121944 secs 2024-07-26T13:59:52.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19136.121944 secs 2024-07-26T13:59:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19136.121944 secs 2024-07-26T13:59:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:52 smithi044 ceph-mon[32404]: pgmap v10526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:53.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:52 smithi116 ceph-mon[37097]: pgmap v10526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:52 smithi171 ceph-mon[37789]: pgmap v10526: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:56.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:55 smithi044 ceph-mon[32404]: pgmap v10527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:56.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:55 smithi116 ceph-mon[37097]: pgmap v10527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:55 smithi171 ceph-mon[37789]: pgmap v10527: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19141.122111 secs 2024-07-26T13:59:57.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19141.122111 secs 2024-07-26T13:59:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19141.122111 secs 2024-07-26T13:59:57.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:57 smithi044 ceph-mon[32404]: pgmap v10528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:58.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:57 smithi116 ceph-mon[37097]: pgmap v10528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:58.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:57 smithi171 ceph-mon[37789]: pgmap v10528: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 13:59:59 smithi044 ceph-mon[32404]: pgmap v10529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:00:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 13:59:59 smithi116 ceph-mon[37097]: pgmap v10529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:00:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 13:59:59 smithi171 ceph-mon[37789]: pgmap v10529: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:00:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:00 smithi044 ceph-mon[32404]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:00 smithi116 ceph-mon[37097]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:00 smithi171 ceph-mon[37789]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:01.256+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:00:01.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:01.256+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:01.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:01.256+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:00:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:01 smithi044 ceph-mon[32404]: pgmap v10530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19146.122235 secs 2024-07-26T14:00:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:01 smithi116 ceph-mon[37097]: pgmap v10530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19146.122235 secs 2024-07-26T14:00:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:01 smithi171 ceph-mon[37789]: pgmap v10530: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:02.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19146.122235 secs 2024-07-26T14:00:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:03 smithi044 ceph-mon[32404]: pgmap v10531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:03 smithi116 ceph-mon[37097]: pgmap v10531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:03 smithi171 ceph-mon[37789]: pgmap v10531: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:05 smithi044 ceph-mon[32404]: pgmap v10532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:05 smithi116 ceph-mon[37097]: pgmap v10532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:05 smithi171 ceph-mon[37789]: pgmap v10532: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19151.122406 secs 2024-07-26T14:00:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19151.122406 secs 2024-07-26T14:00:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19151.122406 secs 2024-07-26T14:00:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:07 smithi044 ceph-mon[32404]: pgmap v10533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:07 smithi116 ceph-mon[37097]: pgmap v10533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:07 smithi171 ceph-mon[37789]: pgmap v10533: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:09 smithi044 ceph-mon[32404]: pgmap v10534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:00:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:09 smithi116 ceph-mon[37097]: pgmap v10534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:00:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:09 smithi171 ceph-mon[37789]: pgmap v10534: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:00:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:11 smithi044 ceph-mon[32404]: pgmap v10535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19156.122544 secs 2024-07-26T14:00:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:11 smithi116 ceph-mon[37097]: pgmap v10535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19156.122544 secs 2024-07-26T14:00:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:11 smithi171 ceph-mon[37789]: pgmap v10535: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19156.122544 secs 2024-07-26T14:00:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:13 smithi044 ceph-mon[32404]: pgmap v10536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:14.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:13 smithi116 ceph-mon[37097]: pgmap v10536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:13 smithi171 ceph-mon[37789]: pgmap v10536: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:16.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:15 smithi044 ceph-mon[32404]: pgmap v10537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:15 smithi116 ceph-mon[37097]: pgmap v10537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:15 smithi171 ceph-mon[37789]: pgmap v10537: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:16.261+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:00:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:16.261+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:00:16.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:16.261+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19161.122709 secs 2024-07-26T14:00:17.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19161.122709 secs 2024-07-26T14:00:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19161.122709 secs 2024-07-26T14:00:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:17 smithi044 ceph-mon[32404]: pgmap v10538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:17 smithi116 ceph-mon[37097]: pgmap v10538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:17 smithi171 ceph-mon[37789]: pgmap v10538: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:19 smithi044 ceph-mon[32404]: pgmap v10539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:19 smithi116 ceph-mon[37097]: pgmap v10539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:19 smithi171 ceph-mon[37789]: pgmap v10539: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:22.018 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:21 smithi044 ceph-mon[32404]: pgmap v10540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:22.019 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19166.122838 secs 2024-07-26T14:00:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:21 smithi116 ceph-mon[37097]: pgmap v10540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19166.122838 secs 2024-07-26T14:00:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:21 smithi171 ceph-mon[37789]: pgmap v10540: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19166.122838 secs 2024-07-26T14:00:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:23 smithi044 ceph-mon[32404]: pgmap v10541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:23 smithi116 ceph-mon[37097]: pgmap v10541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:23 smithi171 ceph-mon[37789]: pgmap v10541: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:25 smithi044 ceph-mon[32404]: pgmap v10542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:25 smithi116 ceph-mon[37097]: pgmap v10542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:25 smithi171 ceph-mon[37789]: pgmap v10542: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19171.122988 secs 2024-07-26T14:00:27.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19171.122988 secs 2024-07-26T14:00:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19171.122988 secs 2024-07-26T14:00:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:27 smithi044 ceph-mon[32404]: pgmap v10543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:28.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:27 smithi116 ceph-mon[37097]: pgmap v10543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:27 smithi171 ceph-mon[37789]: pgmap v10543: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:29 smithi044 ceph-mon[32404]: pgmap v10544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:29 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:29 smithi116 ceph-mon[37097]: pgmap v10544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:29 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:29 smithi171 ceph-mon[37789]: pgmap v10544: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:29 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:31.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:31.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:31.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:30 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:00:31.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:31.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:31.371 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:30 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:00:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:31.263+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:00:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:31.263+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:00:31.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:31.263+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:31.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:31.371 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:31.372 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:30 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:00:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:31 smithi171 ceph-mon[37789]: pgmap v10545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:00:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:31 smithi171 ceph-mon[37789]: pgmap v10546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19176.123130 secs 2024-07-26T14:00:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:31 smithi044 ceph-mon[32404]: pgmap v10545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:31 smithi044 ceph-mon[32404]: pgmap v10546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19176.123130 secs 2024-07-26T14:00:32.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:31 smithi116 ceph-mon[37097]: pgmap v10545: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:31 smithi116 ceph-mon[37097]: pgmap v10546: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19176.123130 secs 2024-07-26T14:00:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:33 smithi171 ceph-mon[37789]: pgmap v10547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:33 smithi044 ceph-mon[32404]: pgmap v10547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:33 smithi116 ceph-mon[37097]: pgmap v10547: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:35 smithi171 ceph-mon[37789]: pgmap v10548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:35 smithi044 ceph-mon[32404]: pgmap v10548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:35 smithi116 ceph-mon[37097]: pgmap v10548: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:37.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19181.123288 secs 2024-07-26T14:00:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19181.123288 secs 2024-07-26T14:00:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19181.123288 secs 2024-07-26T14:00:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:37 smithi171 ceph-mon[37789]: pgmap v10549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:37 smithi044 ceph-mon[32404]: pgmap v10549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:38.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:37 smithi116 ceph-mon[37097]: pgmap v10549: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:00:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:39 smithi171 ceph-mon[37789]: pgmap v10550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:00:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:39 smithi044 ceph-mon[32404]: pgmap v10550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:00:40.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:39 smithi116 ceph-mon[37097]: pgmap v10550: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:00:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:40 smithi044 ceph-mon[32404]: pgmap v10551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:40 smithi116 ceph-mon[37097]: pgmap v10551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:40 smithi171 ceph-mon[37789]: pgmap v10551: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19186.123446 secs 2024-07-26T14:00:42.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19186.123446 secs 2024-07-26T14:00:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19186.123446 secs 2024-07-26T14:00:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:42 smithi044 ceph-mon[32404]: pgmap v10552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:42 smithi116 ceph-mon[37097]: pgmap v10552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:42 smithi171 ceph-mon[37789]: pgmap v10552: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:44 smithi044 ceph-mon[32404]: pgmap v10553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:44 smithi116 ceph-mon[37097]: pgmap v10553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:44 smithi171 ceph-mon[37789]: pgmap v10553: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:46.266 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:46.265+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:00:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:46.265+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:46.267 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:00:46.265+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:00:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19191.123611 secs 2024-07-26T14:00:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:46 smithi044 ceph-mon[32404]: pgmap v10554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19191.123611 secs 2024-07-26T14:00:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:46 smithi116 ceph-mon[37097]: pgmap v10554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19191.123611 secs 2024-07-26T14:00:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:46 smithi171 ceph-mon[37789]: pgmap v10554: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:48 smithi044 ceph-mon[32404]: pgmap v10555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:48 smithi116 ceph-mon[37097]: pgmap v10555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:48 smithi171 ceph-mon[37789]: pgmap v10555: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:51.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:50 smithi044 ceph-mon[32404]: pgmap v10556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:51.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:50 smithi116 ceph-mon[37097]: pgmap v10556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:50 smithi171 ceph-mon[37789]: pgmap v10556: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:52.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19196.123774 secs 2024-07-26T14:00:52.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19196.123774 secs 2024-07-26T14:00:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19196.123774 secs 2024-07-26T14:00:53.238 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:52 smithi171 ceph-mon[37789]: pgmap v10557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:53.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:52 smithi044 ceph-mon[32404]: pgmap v10557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:53.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:52 smithi116 ceph-mon[37097]: pgmap v10557: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:54 smithi044 ceph-mon[32404]: pgmap v10558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:55.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:54 smithi116 ceph-mon[37097]: pgmap v10558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:54 smithi171 ceph-mon[37789]: pgmap v10558: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19201.123919 secs 2024-07-26T14:00:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:56 smithi044 ceph-mon[32404]: pgmap v10559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19201.123919 secs 2024-07-26T14:00:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:56 smithi116 ceph-mon[37097]: pgmap v10559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19201.123919 secs 2024-07-26T14:00:57.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:56 smithi171 ceph-mon[37789]: pgmap v10559: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:59.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:00:58 smithi044 ceph-mon[32404]: pgmap v10560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:00:58 smithi116 ceph-mon[37097]: pgmap v10560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:00:58 smithi171 ceph-mon[37789]: pgmap v10560: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:01.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:00 smithi044 ceph-mon[32404]: pgmap v10561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:01.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:01.266+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:01:01.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:01.266+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:01.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:01.266+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:01:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:00 smithi116 ceph-mon[37097]: pgmap v10561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:00 smithi171 ceph-mon[37789]: pgmap v10561: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:02.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19206.124084 secs 2024-07-26T14:01:02.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19206.124084 secs 2024-07-26T14:01:02.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19206.124084 secs 2024-07-26T14:01:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:02 smithi044 ceph-mon[32404]: pgmap v10562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:02 smithi116 ceph-mon[37097]: pgmap v10562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:02 smithi171 ceph-mon[37789]: pgmap v10562: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:04 smithi044 ceph-mon[32404]: pgmap v10563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:04 smithi116 ceph-mon[37097]: pgmap v10563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:04 smithi171 ceph-mon[37789]: pgmap v10563: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:07.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19211.124256 secs 2024-07-26T14:01:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:06 smithi044 ceph-mon[32404]: pgmap v10564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19211.124256 secs 2024-07-26T14:01:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:06 smithi116 ceph-mon[37097]: pgmap v10564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19211.124256 secs 2024-07-26T14:01:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:06 smithi171 ceph-mon[37789]: pgmap v10564: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:08 smithi044 ceph-mon[32404]: pgmap v10565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:09.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:08 smithi116 ceph-mon[37097]: pgmap v10565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:08 smithi171 ceph-mon[37789]: pgmap v10565: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:10 smithi044 ceph-mon[32404]: pgmap v10566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:01:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:10 smithi116 ceph-mon[37097]: pgmap v10566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:01:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:10 smithi171 ceph-mon[37789]: pgmap v10566: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:01:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19216.124407 secs 2024-07-26T14:01:12.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19216.124407 secs 2024-07-26T14:01:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19216.124407 secs 2024-07-26T14:01:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:12 smithi044 ceph-mon[32404]: pgmap v10567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:12 smithi116 ceph-mon[37097]: pgmap v10567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:12 smithi171 ceph-mon[37789]: pgmap v10567: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:01:15.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:15 smithi044 ceph-mon[32404]: pgmap v10568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:15 smithi116 ceph-mon[37097]: pgmap v10568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:15 smithi171 ceph-mon[37789]: pgmap v10568: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:16.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:16.271+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:01:16.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:16.271+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:01:16.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:16.271+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:17 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19221.124557 secs 2024-07-26T14:01:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:17 smithi044 ceph-mon[32404]: pgmap v10569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:17.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:17 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19221.124557 secs 2024-07-26T14:01:17.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:17 smithi116 ceph-mon[37097]: pgmap v10569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:17.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:17 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19221.124557 secs 2024-07-26T14:01:17.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:17 smithi171 ceph-mon[37789]: pgmap v10569: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:19 smithi044 ceph-mon[32404]: pgmap v10570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:19 smithi116 ceph-mon[37097]: pgmap v10570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:19.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:19 smithi171 ceph-mon[37789]: pgmap v10570: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:21 smithi116 ceph-mon[37097]: pgmap v10571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:21 smithi171 ceph-mon[37789]: pgmap v10571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:21 smithi044 ceph-mon[32404]: pgmap v10571: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:22 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19226.124728 secs 2024-07-26T14:01:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:22 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19226.124728 secs 2024-07-26T14:01:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:22 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19226.124728 secs 2024-07-26T14:01:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:23 smithi171 ceph-mon[37789]: pgmap v10572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:23.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:23 smithi044 ceph-mon[32404]: pgmap v10572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:23.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:23 smithi116 ceph-mon[37097]: pgmap v10572: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:25.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:25 smithi171 ceph-mon[37789]: pgmap v10573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:25.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:25 smithi044 ceph-mon[32404]: pgmap v10573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:25.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:25 smithi116 ceph-mon[37097]: pgmap v10573: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:27 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19231.124905 secs 2024-07-26T14:01:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:27 smithi171 ceph-mon[37789]: pgmap v10574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:27.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:27 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19231.124905 secs 2024-07-26T14:01:27.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:27 smithi044 ceph-mon[32404]: pgmap v10574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:27 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19231.124905 secs 2024-07-26T14:01:27.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:27 smithi116 ceph-mon[37097]: pgmap v10574: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:29 smithi171 ceph-mon[37789]: pgmap v10575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:29.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:29 smithi044 ceph-mon[32404]: pgmap v10575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:29.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:29 smithi116 ceph-mon[37097]: pgmap v10575: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:31.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:31.272+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:01:31.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:31.272+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:01:31.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:31.272+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:31.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:31 smithi044 ceph-mon[32404]: pgmap v10576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:31.318 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:31 smithi171 ceph-mon[37789]: pgmap v10576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:31 smithi116 ceph-mon[37097]: pgmap v10576: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:31.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19236.125060 secs 2024-07-26T14:01:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T14:01:32.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T14:01:32.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:32.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:32.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: pgmap v10577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:01:32.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: pgmap v10578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:32.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:01:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19236.125060 secs 2024-07-26T14:01:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.517 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T14:01:32.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T14:01:32.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:32.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:32.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: pgmap v10577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:01:32.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: pgmap v10578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:32.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:01:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19236.125060 secs 2024-07-26T14:01:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi044 to 177.5M 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi044 to 186154393: error parsing value: Value '186154393' is below minimum 939524096 2024-07-26T14:01:32.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:32.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:32.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: pgmap v10577: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:01:32.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: pgmap v10578: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:32.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:01:34.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:34 smithi116 ceph-mon[37097]: pgmap v10579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:34 smithi171 ceph-mon[37789]: pgmap v10579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:34.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:34 smithi044 ceph-mon[32404]: pgmap v10579: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:36 smithi171 ceph-mon[37789]: pgmap v10580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:36.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19241.125234 secs 2024-07-26T14:01:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:36 smithi044 ceph-mon[32404]: pgmap v10580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:36.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19241.125234 secs 2024-07-26T14:01:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:36 smithi116 ceph-mon[37097]: pgmap v10580: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:36.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19241.125234 secs 2024-07-26T14:01:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:38 smithi171 ceph-mon[37789]: pgmap v10581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:38.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:38 smithi044 ceph-mon[32404]: pgmap v10581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:38.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:38 smithi116 ceph-mon[37097]: pgmap v10581: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:40.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:40 smithi171 ceph-mon[37789]: pgmap v10582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:40.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:40 smithi044 ceph-mon[32404]: pgmap v10582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:40.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:40 smithi116 ceph-mon[37097]: pgmap v10582: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:01:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19246.125395 secs 2024-07-26T14:01:41.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19246.125395 secs 2024-07-26T14:01:41.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19246.125395 secs 2024-07-26T14:01:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:42 smithi171 ceph-mon[37789]: pgmap v10583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:01:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:42 smithi044 ceph-mon[32404]: pgmap v10583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:01:42.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:42 smithi116 ceph-mon[37097]: pgmap v10583: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:01:44.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:44 smithi171 ceph-mon[37789]: pgmap v10584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:44.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:44 smithi044 ceph-mon[32404]: pgmap v10584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:44.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:44 smithi116 ceph-mon[37097]: pgmap v10584: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:46.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:46.278+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:01:46.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:46.278+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:01:46.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:01:46.278+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:46.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:46 smithi171 ceph-mon[37789]: pgmap v10585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:46.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19251.125557 secs 2024-07-26T14:01:46.657 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:46 smithi044 ceph-mon[32404]: pgmap v10585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:46.658 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19251.125557 secs 2024-07-26T14:01:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:46 smithi116 ceph-mon[37097]: pgmap v10585: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:46.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19251.125557 secs 2024-07-26T14:01:48.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:48 smithi171 ceph-mon[37789]: pgmap v10586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:48.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:48 smithi044 ceph-mon[32404]: pgmap v10586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:48.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:48 smithi116 ceph-mon[37097]: pgmap v10586: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:50.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:50 smithi171 ceph-mon[37789]: pgmap v10587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:50.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:50 smithi044 ceph-mon[32404]: pgmap v10587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:50.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:50 smithi116 ceph-mon[37097]: pgmap v10587: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:51.673 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19256.125702 secs 2024-07-26T14:01:51.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19256.125702 secs 2024-07-26T14:01:51.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19256.125702 secs 2024-07-26T14:01:52.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:52 smithi044 ceph-mon[32404]: pgmap v10588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:52.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:52 smithi116 ceph-mon[37097]: pgmap v10588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:52.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:52 smithi171 ceph-mon[37789]: pgmap v10588: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:54.639 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:54 smithi171 ceph-mon[37789]: pgmap v10589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:54.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:54 smithi044 ceph-mon[32404]: pgmap v10589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:54.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:54 smithi116 ceph-mon[37097]: pgmap v10589: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:56 smithi044 ceph-mon[32404]: pgmap v10590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:56.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19261.125803 secs 2024-07-26T14:01:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:56 smithi116 ceph-mon[37097]: pgmap v10590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:56.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19261.125803 secs 2024-07-26T14:01:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:56 smithi171 ceph-mon[37789]: pgmap v10590: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19261.125803 secs 2024-07-26T14:01:58.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:01:58 smithi044 ceph-mon[32404]: pgmap v10591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:58.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:01:58 smithi116 ceph-mon[37097]: pgmap v10591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:01:58.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:01:58 smithi171 ceph-mon[37789]: pgmap v10591: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:00.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:00 smithi044 ceph-mon[32404]: pgmap v10592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:00.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:00 smithi116 ceph-mon[37097]: pgmap v10592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:00.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:00 smithi171 ceph-mon[37789]: pgmap v10592: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:01.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:01.279+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:02:01.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:01.279+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:02:01.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:01.279+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:01.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19266.125947 secs 2024-07-26T14:02:01.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19266.125947 secs 2024-07-26T14:02:01.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19266.125947 secs 2024-07-26T14:02:02.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:02 smithi044 ceph-mon[32404]: pgmap v10593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:02.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:02 smithi116 ceph-mon[37097]: pgmap v10593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:02.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:02 smithi171 ceph-mon[37789]: pgmap v10593: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:04.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:04 smithi044 ceph-mon[32404]: pgmap v10594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:04.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:04 smithi116 ceph-mon[37097]: pgmap v10594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:04.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:04 smithi171 ceph-mon[37789]: pgmap v10594: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:06 smithi044 ceph-mon[32404]: pgmap v10595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:06.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19271.126097 secs 2024-07-26T14:02:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:06 smithi116 ceph-mon[37097]: pgmap v10595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:06.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19271.126097 secs 2024-07-26T14:02:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:06 smithi171 ceph-mon[37789]: pgmap v10595: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19271.126097 secs 2024-07-26T14:02:08.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:08 smithi044 ceph-mon[32404]: pgmap v10596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:08.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:08 smithi116 ceph-mon[37097]: pgmap v10596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:08.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:08 smithi171 ceph-mon[37789]: pgmap v10596: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:10.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:10 smithi044 ceph-mon[32404]: pgmap v10597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:10.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:10 smithi116 ceph-mon[37097]: pgmap v10597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:10.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:10 smithi171 ceph-mon[37789]: pgmap v10597: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:11.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19276.126256 secs 2024-07-26T14:02:11.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19276.126256 secs 2024-07-26T14:02:11.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19276.126256 secs 2024-07-26T14:02:12.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:12 smithi044 ceph-mon[32404]: pgmap v10598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:12.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:12 smithi116 ceph-mon[37097]: pgmap v10598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:12.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:12 smithi171 ceph-mon[37789]: pgmap v10598: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:14.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:14 smithi044 ceph-mon[32404]: pgmap v10599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:14.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:14 smithi116 ceph-mon[37097]: pgmap v10599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:14.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:14 smithi171 ceph-mon[37789]: pgmap v10599: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:16.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:16.284+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:02:16.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:16.284+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:02:16.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:16.284+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:16.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:16 smithi044 ceph-mon[32404]: pgmap v10600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:16.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19281.126423 secs 2024-07-26T14:02:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:16 smithi116 ceph-mon[37097]: pgmap v10600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:16.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19281.126423 secs 2024-07-26T14:02:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:16 smithi171 ceph-mon[37789]: pgmap v10600: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:16.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19281.126423 secs 2024-07-26T14:02:18.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:18.168+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T14:02:18.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:18.168+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T14:02:18.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:18.169+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T14:02:18.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:18.169+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T14:02:18.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:18 smithi044 ceph-mon[32404]: pgmap v10601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:18.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:18 smithi116 ceph-mon[37097]: pgmap v10601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:18.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:18 smithi171 ceph-mon[37789]: pgmap v10601: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:20.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:20 smithi044 ceph-mon[32404]: pgmap v10602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:20.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:20 smithi116 ceph-mon[37097]: pgmap v10602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:20.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:20 smithi171 ceph-mon[37789]: pgmap v10602: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:21.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19286.126573 secs 2024-07-26T14:02:21.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19286.126573 secs 2024-07-26T14:02:21.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19286.126573 secs 2024-07-26T14:02:22.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:22 smithi044 ceph-mon[32404]: pgmap v10603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:22.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:22 smithi116 ceph-mon[37097]: pgmap v10603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:22.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:22 smithi171 ceph-mon[37789]: pgmap v10603: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:24.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:24 smithi044 ceph-mon[32404]: pgmap v10604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:24.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:24 smithi116 ceph-mon[37097]: pgmap v10604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:24.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:24 smithi171 ceph-mon[37789]: pgmap v10604: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:26.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:26 smithi044 ceph-mon[32404]: pgmap v10605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:26.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19291.126726 secs 2024-07-26T14:02:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:26 smithi116 ceph-mon[37097]: pgmap v10605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:26.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19291.126726 secs 2024-07-26T14:02:26.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:26 smithi171 ceph-mon[37789]: pgmap v10605: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:26.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19291.126726 secs 2024-07-26T14:02:28.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:28 smithi044 ceph-mon[32404]: pgmap v10606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:28.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:28 smithi116 ceph-mon[37097]: pgmap v10606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:28.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:28 smithi171 ceph-mon[37789]: pgmap v10606: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:30.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:30 smithi044 ceph-mon[32404]: pgmap v10607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:30.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:30 smithi116 ceph-mon[37097]: pgmap v10607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:30.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:30 smithi171 ceph-mon[37789]: pgmap v10607: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:31.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:31.285+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:02:31.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:31.286+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:02:31.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:31.286+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:31.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19296.126894 secs 2024-07-26T14:02:31.667 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:31 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19296.126894 secs 2024-07-26T14:02:31.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:31 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19296.126894 secs 2024-07-26T14:02:31.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:31 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:32.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: pgmap v10608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.768 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.769 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:32 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: pgmap v10608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:32 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: pgmap v10608: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:32.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:32 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:02:33.762 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: pgmap v10609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:33.763 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: pgmap v10610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:02:33.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:33 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:02:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T14:02:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T14:02:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T14:02:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:02:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:33.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: pgmap v10609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: pgmap v10610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:02:33.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:33 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:02:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-07-26T14:02:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-07-26T14:02:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi116 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-07-26T14:02:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: Unable to set osd_memory_target on smithi171 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T14:02:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: pgmap v10609: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: pgmap v10610: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:02:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:33 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:02:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:35 smithi171 ceph-mon[37789]: pgmap v10611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:36.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:35 smithi044 ceph-mon[32404]: pgmap v10611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:36.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:35 smithi116 ceph-mon[37097]: pgmap v10611: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:36.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19301.127038 secs 2024-07-26T14:02:37.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19301.127038 secs 2024-07-26T14:02:37.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19301.127038 secs 2024-07-26T14:02:37.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:37 smithi171 ceph-mon[37789]: pgmap v10612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:02:38.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:37 smithi044 ceph-mon[32404]: pgmap v10612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:02:38.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:37 smithi116 ceph-mon[37097]: pgmap v10612: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:02:40.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:39 smithi044 ceph-mon[32404]: pgmap v10613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:40.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:39 smithi116 ceph-mon[37097]: pgmap v10613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:40.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:39 smithi171 ceph-mon[37789]: pgmap v10613: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:41 smithi044 ceph-mon[32404]: pgmap v10614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19306.127202 secs 2024-07-26T14:02:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:41 smithi116 ceph-mon[37097]: pgmap v10614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19306.127202 secs 2024-07-26T14:02:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:41 smithi171 ceph-mon[37789]: pgmap v10614: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:02:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19306.127202 secs 2024-07-26T14:02:44.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:43 smithi044 ceph-mon[32404]: pgmap v10615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:02:44.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:43 smithi116 ceph-mon[37097]: pgmap v10615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:02:44.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:43 smithi171 ceph-mon[37789]: pgmap v10615: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:02:46.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:45 smithi044 ceph-mon[32404]: pgmap v10616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:46.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:45 smithi116 ceph-mon[37097]: pgmap v10616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:46.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:45 smithi171 ceph-mon[37789]: pgmap v10616: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:46.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:46.292+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:02:46.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:46.292+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:02:46.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:02:46.292+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19311.127355 secs 2024-07-26T14:02:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19311.127355 secs 2024-07-26T14:02:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19311.127355 secs 2024-07-26T14:02:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:47 smithi044 ceph-mon[32404]: pgmap v10617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:48.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:47 smithi116 ceph-mon[37097]: pgmap v10617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:47 smithi171 ceph-mon[37789]: pgmap v10617: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:49 smithi044 ceph-mon[32404]: pgmap v10618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:49 smithi116 ceph-mon[37097]: pgmap v10618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:49 smithi171 ceph-mon[37789]: pgmap v10618: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:51 smithi044 ceph-mon[32404]: pgmap v10619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19316.127509 secs 2024-07-26T14:02:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:51 smithi116 ceph-mon[37097]: pgmap v10619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19316.127509 secs 2024-07-26T14:02:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:51 smithi171 ceph-mon[37789]: pgmap v10619: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19316.127509 secs 2024-07-26T14:02:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:53 smithi044 ceph-mon[32404]: pgmap v10620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:54.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:53 smithi116 ceph-mon[37097]: pgmap v10620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:53 smithi171 ceph-mon[37789]: pgmap v10620: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:02:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:55 smithi171 ceph-mon[37789]: pgmap v10621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:55 smithi044 ceph-mon[32404]: pgmap v10621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:55.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:55 smithi116 ceph-mon[37097]: pgmap v10621: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:02:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19321.127675 secs 2024-07-26T14:02:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19321.127675 secs 2024-07-26T14:02:56.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19321.127675 secs 2024-07-26T14:02:57.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:57 smithi171 ceph-mon[37789]: pgmap v10622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:57 smithi044 ceph-mon[32404]: pgmap v10622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:57.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:57 smithi116 ceph-mon[37097]: pgmap v10622: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:59.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:02:59 smithi171 ceph-mon[37789]: pgmap v10623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:02:59 smithi044 ceph-mon[32404]: pgmap v10623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:02:59 smithi116 ceph-mon[37097]: pgmap v10623: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:01.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:01.293+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:03:01.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:01.293+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:03:01.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:01.293+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:01 smithi044 ceph-mon[32404]: pgmap v10624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19326.127821 secs 2024-07-26T14:03:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:01 smithi116 ceph-mon[37097]: pgmap v10624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19326.127821 secs 2024-07-26T14:03:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:01 smithi171 ceph-mon[37789]: pgmap v10624: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19326.127821 secs 2024-07-26T14:03:03.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:03.177+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T14:03:03.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:03.177+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T14:03:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:03.178+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T14:03:03.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:03.178+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T14:03:03.977 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:03 smithi044 ceph-mon[32404]: pgmap v10625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:03 smithi116 ceph-mon[37097]: pgmap v10625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:03 smithi171 ceph-mon[37789]: pgmap v10625: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:05 smithi044 ceph-mon[32404]: pgmap v10626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:05 smithi116 ceph-mon[37097]: pgmap v10626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:05 smithi171 ceph-mon[37789]: pgmap v10626: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19331.127962 secs 2024-07-26T14:03:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19331.127962 secs 2024-07-26T14:03:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19331.127962 secs 2024-07-26T14:03:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:07 smithi044 ceph-mon[32404]: pgmap v10627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:08.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:07 smithi116 ceph-mon[37097]: pgmap v10627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:07 smithi171 ceph-mon[37789]: pgmap v10627: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:09 smithi044 ceph-mon[32404]: pgmap v10628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:03:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:09 smithi116 ceph-mon[37097]: pgmap v10628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:03:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:09 smithi171 ceph-mon[37789]: pgmap v10628: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:03:12.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:11 smithi044 ceph-mon[32404]: pgmap v10629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:03:12.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19336.128134 secs 2024-07-26T14:03:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:11 smithi116 ceph-mon[37097]: pgmap v10629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:03:12.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19336.128134 secs 2024-07-26T14:03:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:11 smithi171 ceph-mon[37789]: pgmap v10629: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:03:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19336.128134 secs 2024-07-26T14:03:14.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:13 smithi044 ceph-mon[32404]: pgmap v10630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:03:14.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:13 smithi116 ceph-mon[37097]: pgmap v10630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:03:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:13 smithi171 ceph-mon[37789]: pgmap v10630: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:03:16.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:15 smithi044 ceph-mon[32404]: pgmap v10631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:03:16.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:15 smithi116 ceph-mon[37097]: pgmap v10631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:03:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:15 smithi171 ceph-mon[37789]: pgmap v10631: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:03:16.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:16.299+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:03:16.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:16.299+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:03:16.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:16.299+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:17.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19341.128302 secs 2024-07-26T14:03:17.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19341.128302 secs 2024-07-26T14:03:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19341.128302 secs 2024-07-26T14:03:18.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:17 smithi044 ceph-mon[32404]: pgmap v10632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:18.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:17 smithi116 ceph-mon[37097]: pgmap v10632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:17 smithi171 ceph-mon[37789]: pgmap v10632: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:20.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:19 smithi044 ceph-mon[32404]: pgmap v10633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:03:20.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:19 smithi116 ceph-mon[37097]: pgmap v10633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:03:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:19 smithi171 ceph-mon[37789]: pgmap v10633: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:03:22.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:21 smithi044 ceph-mon[32404]: pgmap v10634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:22.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19346.128470 secs 2024-07-26T14:03:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:21 smithi116 ceph-mon[37097]: pgmap v10634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:22.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19346.128470 secs 2024-07-26T14:03:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:21 smithi171 ceph-mon[37789]: pgmap v10634: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19346.128470 secs 2024-07-26T14:03:24.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:23 smithi044 ceph-mon[32404]: pgmap v10635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:24.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:23 smithi116 ceph-mon[37097]: pgmap v10635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:23 smithi171 ceph-mon[37789]: pgmap v10635: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:26.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:25 smithi044 ceph-mon[32404]: pgmap v10636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:26.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:25 smithi116 ceph-mon[37097]: pgmap v10636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:25 smithi171 ceph-mon[37789]: pgmap v10636: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19351.128656 secs 2024-07-26T14:03:27.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19351.128656 secs 2024-07-26T14:03:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19351.128656 secs 2024-07-26T14:03:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:27 smithi044 ceph-mon[32404]: pgmap v10637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:28.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:27 smithi116 ceph-mon[37097]: pgmap v10637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:28.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:27 smithi171 ceph-mon[37789]: pgmap v10637: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:30.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:29 smithi044 ceph-mon[32404]: pgmap v10638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:30.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:29 smithi116 ceph-mon[37097]: pgmap v10638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:29 smithi171 ceph-mon[37789]: pgmap v10638: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:31.300+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:03:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:31.300+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:03:31.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:31.300+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:32.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:31 smithi044 ceph-mon[32404]: pgmap v10639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:32.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19356.128786 secs 2024-07-26T14:03:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:31 smithi116 ceph-mon[37097]: pgmap v10639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:32.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19356.128786 secs 2024-07-26T14:03:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:31 smithi171 ceph-mon[37789]: pgmap v10639: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19356.128786 secs 2024-07-26T14:03:34.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:33 smithi044 ceph-mon[32404]: pgmap v10640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:34.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:33 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:34.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:33 smithi116 ceph-mon[37097]: pgmap v10640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:34.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:33 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:33 smithi171 ceph-mon[37789]: pgmap v10640: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:34.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:33 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:35.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:34 smithi044 ceph-mon[32404]: pgmap v10641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:35.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:03:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:34 smithi116 ceph-mon[37097]: pgmap v10641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:35.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:03:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:34 smithi171 ceph-mon[37789]: pgmap v10641: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:03:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19361.128946 secs 2024-07-26T14:03:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:36 smithi044 ceph-mon[32404]: pgmap v10642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19361.128946 secs 2024-07-26T14:03:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:36 smithi116 ceph-mon[37097]: pgmap v10642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19361.128946 secs 2024-07-26T14:03:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:36 smithi171 ceph-mon[37789]: pgmap v10642: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:37 smithi044 ceph-mon[32404]: pgmap v10643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:03:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:37 smithi116 ceph-mon[37097]: pgmap v10643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:03:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:37 smithi171 ceph-mon[37789]: pgmap v10643: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:03:41.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:40 smithi044 ceph-mon[32404]: pgmap v10644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:40 smithi116 ceph-mon[37097]: pgmap v10644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:41.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:40 smithi171 ceph-mon[37789]: pgmap v10644: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19366.129112 secs 2024-07-26T14:03:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:41 smithi044 ceph-mon[32404]: pgmap v10645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19366.129112 secs 2024-07-26T14:03:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:41 smithi116 ceph-mon[37097]: pgmap v10645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19366.129112 secs 2024-07-26T14:03:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:41 smithi171 ceph-mon[37789]: pgmap v10645: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:43 smithi044 ceph-mon[32404]: pgmap v10646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:44.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:43 smithi116 ceph-mon[37097]: pgmap v10646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:43 smithi171 ceph-mon[37789]: pgmap v10646: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:46.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:45 smithi044 ceph-mon[32404]: pgmap v10647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:46.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:45 smithi116 ceph-mon[37097]: pgmap v10647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:46.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:46.306+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:03:46.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:46.306+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:03:46.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:03:46.306+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:45 smithi171 ceph-mon[37789]: pgmap v10647: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19371.129273 secs 2024-07-26T14:03:47.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19371.129273 secs 2024-07-26T14:03:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19371.129273 secs 2024-07-26T14:03:48.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:47 smithi044 ceph-mon[32404]: pgmap v10648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:48.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:47 smithi116 ceph-mon[37097]: pgmap v10648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:48.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:47 smithi171 ceph-mon[37789]: pgmap v10648: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:49 smithi044 ceph-mon[32404]: pgmap v10649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:50.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:49 smithi116 ceph-mon[37097]: pgmap v10649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:49 smithi171 ceph-mon[37789]: pgmap v10649: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:03:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19376.129432 secs 2024-07-26T14:03:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19376.129432 secs 2024-07-26T14:03:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19376.129432 secs 2024-07-26T14:03:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:52 smithi044 ceph-mon[32404]: pgmap v10650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:52 smithi116 ceph-mon[37097]: pgmap v10650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:52 smithi171 ceph-mon[37789]: pgmap v10650: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:54.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:53 smithi044 ceph-mon[32404]: pgmap v10651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:54.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:53 smithi116 ceph-mon[37097]: pgmap v10651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:54.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:53 smithi171 ceph-mon[37789]: pgmap v10651: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:56.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:55 smithi044 ceph-mon[32404]: pgmap v10652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:56.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:55 smithi116 ceph-mon[37097]: pgmap v10652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:55 smithi171 ceph-mon[37789]: pgmap v10652: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19381.129599 secs 2024-07-26T14:03:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19381.129599 secs 2024-07-26T14:03:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19381.129599 secs 2024-07-26T14:03:58.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:57 smithi044 ceph-mon[32404]: pgmap v10653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:58.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:57 smithi116 ceph-mon[37097]: pgmap v10653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:03:58.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:57 smithi171 ceph-mon[37789]: pgmap v10653: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:00.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:03:59 smithi044 ceph-mon[32404]: pgmap v10654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:00.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:03:59 smithi116 ceph-mon[37097]: pgmap v10654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:03:59 smithi171 ceph-mon[37789]: pgmap v10654: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:01.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:01.307+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:04:01.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:01.307+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:01.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:01.307+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:04:01.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19386.129765 secs 2024-07-26T14:04:01.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19386.129765 secs 2024-07-26T14:04:01.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19386.129765 secs 2024-07-26T14:04:02.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:02 smithi044 ceph-mon[32404]: pgmap v10655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:02.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:02 smithi116 ceph-mon[37097]: pgmap v10655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:02.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:02 smithi171 ceph-mon[37789]: pgmap v10655: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:04.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:03 smithi044 ceph-mon[32404]: pgmap v10656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:04.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:03 smithi116 ceph-mon[37097]: pgmap v10656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:04.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:03 smithi171 ceph-mon[37789]: pgmap v10656: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:05 smithi044 ceph-mon[32404]: pgmap v10657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:06.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:05 smithi116 ceph-mon[37097]: pgmap v10657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:05 smithi171 ceph-mon[37789]: pgmap v10657: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:07.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19391.129927 secs 2024-07-26T14:04:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19391.129927 secs 2024-07-26T14:04:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19391.129927 secs 2024-07-26T14:04:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:07 smithi044 ceph-mon[32404]: pgmap v10658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:07 smithi116 ceph-mon[37097]: pgmap v10658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:07 smithi171 ceph-mon[37789]: pgmap v10658: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:09 smithi044 ceph-mon[32404]: pgmap v10659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:10.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:09 smithi116 ceph-mon[37097]: pgmap v10659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:09 smithi171 ceph-mon[37789]: pgmap v10659: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:11.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19396.130092 secs 2024-07-26T14:04:11.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19396.130092 secs 2024-07-26T14:04:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19396.130092 secs 2024-07-26T14:04:12.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:12 smithi044 ceph-mon[32404]: pgmap v10660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:12.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:12 smithi116 ceph-mon[37097]: pgmap v10660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:12 smithi171 ceph-mon[37789]: pgmap v10660: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:13 smithi044 ceph-mon[32404]: pgmap v10661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:13 smithi116 ceph-mon[37097]: pgmap v10661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:14.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:13 smithi171 ceph-mon[37789]: pgmap v10661: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:15 smithi044 ceph-mon[32404]: pgmap v10662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:16.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:15 smithi116 ceph-mon[37097]: pgmap v10662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:16.313+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:04:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:16.313+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:04:16.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:16.313+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:15 smithi171 ceph-mon[37789]: pgmap v10662: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19401.130240 secs 2024-07-26T14:04:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19401.130240 secs 2024-07-26T14:04:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19401.130240 secs 2024-07-26T14:04:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:17 smithi044 ceph-mon[32404]: pgmap v10663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:18.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:17 smithi116 ceph-mon[37097]: pgmap v10663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:17 smithi171 ceph-mon[37789]: pgmap v10663: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:20.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:19 smithi044 ceph-mon[32404]: pgmap v10664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:19 smithi116 ceph-mon[37097]: pgmap v10664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:20.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:19 smithi171 ceph-mon[37789]: pgmap v10664: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:21.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19406.130386 secs 2024-07-26T14:04:21.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19406.130386 secs 2024-07-26T14:04:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19406.130386 secs 2024-07-26T14:04:22.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:22 smithi044 ceph-mon[32404]: pgmap v10665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:22.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:22 smithi116 ceph-mon[37097]: pgmap v10665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:22.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:22 smithi171 ceph-mon[37789]: pgmap v10665: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:23 smithi044 ceph-mon[32404]: pgmap v10666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:24.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:23 smithi116 ceph-mon[37097]: pgmap v10666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:23 smithi171 ceph-mon[37789]: pgmap v10666: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:25 smithi044 ceph-mon[32404]: pgmap v10667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:25 smithi116 ceph-mon[37097]: pgmap v10667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:26.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:25 smithi171 ceph-mon[37789]: pgmap v10667: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19411.130554 secs 2024-07-26T14:04:27.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19411.130554 secs 2024-07-26T14:04:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19411.130554 secs 2024-07-26T14:04:28.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:27 smithi044 ceph-mon[32404]: pgmap v10668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:27 smithi116 ceph-mon[37097]: pgmap v10668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:27 smithi171 ceph-mon[37789]: pgmap v10668: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:29 smithi044 ceph-mon[32404]: pgmap v10669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:29 smithi116 ceph-mon[37097]: pgmap v10669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:29 smithi171 ceph-mon[37789]: pgmap v10669: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:31.313+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:04:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:31.313+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:31.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:31.313+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:04:31.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19416.130723 secs 2024-07-26T14:04:31.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19416.130723 secs 2024-07-26T14:04:31.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19416.130723 secs 2024-07-26T14:04:32.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:32 smithi044 ceph-mon[32404]: pgmap v10670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:32.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:32 smithi116 ceph-mon[37097]: pgmap v10670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:32 smithi171 ceph-mon[37789]: pgmap v10670: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:33 smithi044 ceph-mon[32404]: pgmap v10671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:34.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:33 smithi116 ceph-mon[37097]: pgmap v10671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:33 smithi171 ceph-mon[37789]: pgmap v10671: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:34 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:34 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:34 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:35.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:34 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:34 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:35.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:34 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:35 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:35 smithi044 ceph-mon[32404]: pgmap v10672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:35 smithi044 ceph-mon[32404]: pgmap v10673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:36.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:35 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:35 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:35 smithi116 ceph-mon[37097]: pgmap v10672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:35 smithi116 ceph-mon[37097]: pgmap v10673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:36.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:35 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:35 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:35 smithi171 ceph-mon[37789]: pgmap v10672: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:35 smithi171 ceph-mon[37789]: pgmap v10673: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:35 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19421.130838 secs 2024-07-26T14:04:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19421.130838 secs 2024-07-26T14:04:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19421.130838 secs 2024-07-26T14:04:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:37 smithi044 ceph-mon[32404]: pgmap v10674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:38.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:37 smithi116 ceph-mon[37097]: pgmap v10674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:37 smithi171 ceph-mon[37789]: pgmap v10674: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:41.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:40 smithi044 ceph-mon[32404]: pgmap v10675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:40 smithi171 ceph-mon[37789]: pgmap v10675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:41.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:40 smithi116 ceph-mon[37097]: pgmap v10675: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19426.130992 secs 2024-07-26T14:04:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19426.130992 secs 2024-07-26T14:04:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19426.130992 secs 2024-07-26T14:04:43.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:42 smithi044 ceph-mon[32404]: pgmap v10676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:04:43.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:42 smithi116 ceph-mon[37097]: pgmap v10676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:04:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:42 smithi171 ceph-mon[37789]: pgmap v10676: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:04:45.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:44 smithi044 ceph-mon[32404]: pgmap v10677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:44 smithi116 ceph-mon[37097]: pgmap v10677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:44 smithi171 ceph-mon[37789]: pgmap v10677: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:46.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:46.318+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:04:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:46.318+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:46.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:04:46.318+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:04:47.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:46 smithi044 ceph-mon[32404]: pgmap v10678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:04:47.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19431.131149 secs 2024-07-26T14:04:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:46 smithi116 ceph-mon[37097]: pgmap v10678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:04:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19431.131149 secs 2024-07-26T14:04:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:46 smithi171 ceph-mon[37789]: pgmap v10678: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:04:47.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19431.131149 secs 2024-07-26T14:04:49.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:48 smithi044 ceph-mon[32404]: pgmap v10679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:49.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:48 smithi116 ceph-mon[37097]: pgmap v10679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:48 smithi171 ceph-mon[37789]: pgmap v10679: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:50.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:49 smithi044 ceph-mon[32404]: pgmap v10680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:50.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:49 smithi116 ceph-mon[37097]: pgmap v10680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:50.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:49 smithi171 ceph-mon[37789]: pgmap v10680: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19436.131297 secs 2024-07-26T14:04:51.521 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19436.131297 secs 2024-07-26T14:04:51.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19436.131297 secs 2024-07-26T14:04:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:52 smithi044 ceph-mon[32404]: pgmap v10681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:52 smithi116 ceph-mon[37097]: pgmap v10681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:52.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:52 smithi171 ceph-mon[37789]: pgmap v10681: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:54 smithi171 ceph-mon[37789]: pgmap v10682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:55.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:54 smithi044 ceph-mon[32404]: pgmap v10682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:55.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:54 smithi116 ceph-mon[37097]: pgmap v10682: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:57.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:56 smithi044 ceph-mon[32404]: pgmap v10683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:57.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19441.131441 secs 2024-07-26T14:04:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:56 smithi116 ceph-mon[37097]: pgmap v10683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:57.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19441.131441 secs 2024-07-26T14:04:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:56 smithi171 ceph-mon[37789]: pgmap v10683: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:57.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19441.131441 secs 2024-07-26T14:04:59.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:04:58 smithi044 ceph-mon[32404]: pgmap v10684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:59.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:04:58 smithi116 ceph-mon[37097]: pgmap v10684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:59.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:04:58 smithi171 ceph-mon[37789]: pgmap v10684: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:01.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:00 smithi044 ceph-mon[32404]: pgmap v10685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:01.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:00 smithi116 ceph-mon[37097]: pgmap v10685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:01.320+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:05:01.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:01.320+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:01.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:01.320+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:05:01.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:00 smithi171 ceph-mon[37789]: pgmap v10685: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:02.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19446.131598 secs 2024-07-26T14:05:02.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19446.131598 secs 2024-07-26T14:05:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19446.131598 secs 2024-07-26T14:05:03.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:02 smithi044 ceph-mon[32404]: pgmap v10686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:03.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:02 smithi116 ceph-mon[37097]: pgmap v10686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:03.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:02 smithi171 ceph-mon[37789]: pgmap v10686: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:05.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:04 smithi044 ceph-mon[32404]: pgmap v10687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:05.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:04 smithi116 ceph-mon[37097]: pgmap v10687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:05.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:04 smithi171 ceph-mon[37789]: pgmap v10687: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:06.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:05 smithi044 ceph-mon[32404]: pgmap v10688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:06.274 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:05 smithi116 ceph-mon[37097]: pgmap v10688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:05 smithi171 ceph-mon[37789]: pgmap v10688: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:07.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19451.131727 secs 2024-07-26T14:05:07.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19451.131727 secs 2024-07-26T14:05:07.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19451.131727 secs 2024-07-26T14:05:08.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:07 smithi044 ceph-mon[32404]: pgmap v10689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:08.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:07 smithi116 ceph-mon[37097]: pgmap v10689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:08.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:07 smithi171 ceph-mon[37789]: pgmap v10689: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:11.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:10 smithi044 ceph-mon[32404]: pgmap v10690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:10 smithi116 ceph-mon[37097]: pgmap v10690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:11.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:10 smithi171 ceph-mon[37789]: pgmap v10690: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19456.131897 secs 2024-07-26T14:05:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19456.131897 secs 2024-07-26T14:05:12.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19456.131897 secs 2024-07-26T14:05:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:12 smithi044 ceph-mon[32404]: pgmap v10691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:12 smithi116 ceph-mon[37097]: pgmap v10691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:13.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:12 smithi171 ceph-mon[37789]: pgmap v10691: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:15.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:14 smithi044 ceph-mon[32404]: pgmap v10692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:14 smithi116 ceph-mon[37097]: pgmap v10692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:14 smithi171 ceph-mon[37789]: pgmap v10692: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:15 smithi044 ceph-mon[32404]: pgmap v10693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:16.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:15 smithi116 ceph-mon[37097]: pgmap v10693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:16.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:16.326+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:05:16.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:16.326+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:16.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:16.326+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:05:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:15 smithi171 ceph-mon[37789]: pgmap v10693: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19461.132065 secs 2024-07-26T14:05:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19461.132065 secs 2024-07-26T14:05:17.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19461.132065 secs 2024-07-26T14:05:18.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:18.202+0000 7fcf68c00640 1 -- 172.21.15.44:0/1872798322 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T14:05:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:18.202+0000 7fcf52000640 1 -- 172.21.15.44:0/2544213633 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T14:05:18.204 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:17 smithi044 ceph-mon[32404]: pgmap v10694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:18.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:18.203+0000 7fcf6a000640 1 -- 172.21.15.44:0/1872798322 <== mon.1 v2:172.21.15.171:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fcf400038a0 con 0x7fcf6c1057b0 2024-07-26T14:05:18.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:18.203+0000 7fcf53400640 1 -- 172.21.15.44:0/2544213633 <== mon.0 v2:172.21.15.44:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fcf3c0064b0 con 0x55c94889a070 2024-07-26T14:05:18.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:17 smithi116 ceph-mon[37097]: pgmap v10694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:17 smithi171 ceph-mon[37789]: pgmap v10694: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:20 smithi044 ceph-mon[32404]: pgmap v10695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:20 smithi116 ceph-mon[37097]: pgmap v10695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:20 smithi171 ceph-mon[37789]: pgmap v10695: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:22.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19466.132231 secs 2024-07-26T14:05:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19466.132231 secs 2024-07-26T14:05:22.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19466.132231 secs 2024-07-26T14:05:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:22 smithi044 ceph-mon[32404]: pgmap v10696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:22 smithi116 ceph-mon[37097]: pgmap v10696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:23.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:22 smithi171 ceph-mon[37789]: pgmap v10696: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:23 smithi044 ceph-mon[32404]: pgmap v10697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:24.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:23 smithi116 ceph-mon[37097]: pgmap v10697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:23 smithi171 ceph-mon[37789]: pgmap v10697: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19471.132379 secs 2024-07-26T14:05:27.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:26 smithi044 ceph-mon[32404]: pgmap v10698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:27.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19471.132379 secs 2024-07-26T14:05:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:26 smithi116 ceph-mon[37097]: pgmap v10698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19471.132379 secs 2024-07-26T14:05:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:26 smithi171 ceph-mon[37789]: pgmap v10698: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:28 smithi044 ceph-mon[32404]: pgmap v10699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:29.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:28 smithi116 ceph-mon[37097]: pgmap v10699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:29.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:28 smithi171 ceph-mon[37789]: pgmap v10699: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:30 smithi044 ceph-mon[32404]: pgmap v10700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:31.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:30 smithi116 ceph-mon[37097]: pgmap v10700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:31.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:31.326+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:05:31.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:31.326+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:05:31.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:31.326+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:30 smithi171 ceph-mon[37789]: pgmap v10700: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19476.132545 secs 2024-07-26T14:05:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19476.132545 secs 2024-07-26T14:05:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19476.132545 secs 2024-07-26T14:05:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:32 smithi044 ceph-mon[32404]: pgmap v10701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:33.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:32 smithi116 ceph-mon[37097]: pgmap v10701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:32 smithi171 ceph-mon[37789]: pgmap v10701: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:33 smithi044 ceph-mon[32404]: pgmap v10702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:34.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:33 smithi116 ceph-mon[37097]: pgmap v10702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:33 smithi171 ceph-mon[37789]: pgmap v10702: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:37.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19481.132713 secs 2024-07-26T14:05:37.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: pgmap v10703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:37.126 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:37.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.127 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:36 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.141 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19481.132713 secs 2024-07-26T14:05:37.141 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: pgmap v10703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:37.141 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:37.142 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.142 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.142 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.142 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:36 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.206 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19481.132713 secs 2024-07-26T14:05:37.206 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: pgmap v10703: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:37.206 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:37.206 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.207 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.207 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:37.207 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:36 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: pgmap v10704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:38 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: pgmap v10704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:38 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: pgmap v10704: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:39.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:38 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:40 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:40 smithi044 ceph-mon[32404]: pgmap v10705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:40 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:40 smithi116 ceph-mon[37097]: pgmap v10705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:41.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:40 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:40 smithi171 ceph-mon[37789]: pgmap v10705: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19486.132800 secs 2024-07-26T14:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:41 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:42.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:41 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19486.132800 secs 2024-07-26T14:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:41 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:42.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:41 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19486.132800 secs 2024-07-26T14:05:42.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:41 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:41 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:43.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:42 smithi044 ceph-mon[32404]: pgmap v10706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:05:43.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:42 smithi044 ceph-mon[32404]: pgmap v10707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:42 smithi116 ceph-mon[37097]: pgmap v10706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:05:43.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:42 smithi116 ceph-mon[37097]: pgmap v10707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:42 smithi171 ceph-mon[37789]: pgmap v10706: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:05:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:42 smithi171 ceph-mon[37789]: pgmap v10707: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:45.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:44 smithi044 ceph-mon[32404]: pgmap v10708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:45.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:44 smithi116 ceph-mon[37097]: pgmap v10708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:44 smithi171 ceph-mon[37789]: pgmap v10708: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:46.332+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:05:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:46.332+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:05:46.332+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:05:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:46 smithi044 ceph-mon[32404]: pgmap v10709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:47.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19491.132968 secs 2024-07-26T14:05:47.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:46 smithi116 ceph-mon[37097]: pgmap v10709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:47.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19491.132968 secs 2024-07-26T14:05:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:46 smithi171 ceph-mon[37789]: pgmap v10709: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19491.132968 secs 2024-07-26T14:05:49.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:48 smithi044 ceph-mon[32404]: pgmap v10710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:49.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:48 smithi116 ceph-mon[37097]: pgmap v10710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:49.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:48 smithi171 ceph-mon[37789]: pgmap v10710: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:51.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:50 smithi044 ceph-mon[32404]: pgmap v10711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:51.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:50 smithi116 ceph-mon[37097]: pgmap v10711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:51.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:50 smithi171 ceph-mon[37789]: pgmap v10711: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19496.133139 secs 2024-07-26T14:05:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19496.133139 secs 2024-07-26T14:05:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19496.133139 secs 2024-07-26T14:05:53.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:52 smithi044 ceph-mon[32404]: pgmap v10712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:53.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:52 smithi116 ceph-mon[37097]: pgmap v10712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:52 smithi171 ceph-mon[37789]: pgmap v10712: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:54.977 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:54 smithi044 ceph-mon[32404]: pgmap v10713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:05:55.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:54 smithi116 ceph-mon[37097]: pgmap v10713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:05:55.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:54 smithi171 ceph-mon[37789]: pgmap v10713: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:05:57.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:56 smithi044 ceph-mon[32404]: pgmap v10714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:05:57.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19501.133272 secs 2024-07-26T14:05:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:56 smithi116 ceph-mon[37097]: pgmap v10714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:05:57.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19501.133272 secs 2024-07-26T14:05:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:56 smithi171 ceph-mon[37789]: pgmap v10714: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:05:57.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19501.133272 secs 2024-07-26T14:05:59.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:05:58 smithi044 ceph-mon[32404]: pgmap v10715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:59.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:05:58 smithi116 ceph-mon[37097]: pgmap v10715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:59.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:05:58 smithi171 ceph-mon[37789]: pgmap v10715: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:01.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:00 smithi044 ceph-mon[32404]: pgmap v10716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:06:01.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:00 smithi116 ceph-mon[37097]: pgmap v10716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:06:01.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:00 smithi171 ceph-mon[37789]: pgmap v10716: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:06:01.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:01.333+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:06:01.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:01.333+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:06:01.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:01.333+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19506.133418 secs 2024-07-26T14:06:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19506.133418 secs 2024-07-26T14:06:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19506.133418 secs 2024-07-26T14:06:03.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:02 smithi044 ceph-mon[32404]: pgmap v10717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:06:03.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:02 smithi116 ceph-mon[37097]: pgmap v10717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:06:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:02 smithi171 ceph-mon[37789]: pgmap v10717: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:06:05.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:04 smithi044 ceph-mon[32404]: pgmap v10718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:06:05.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:04 smithi116 ceph-mon[37097]: pgmap v10718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:06:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:04 smithi171 ceph-mon[37789]: pgmap v10718: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:06:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:06 smithi044 ceph-mon[32404]: pgmap v10719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:07.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19511.133575 secs 2024-07-26T14:06:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:06 smithi116 ceph-mon[37097]: pgmap v10719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:07.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19511.133575 secs 2024-07-26T14:06:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:06 smithi171 ceph-mon[37789]: pgmap v10719: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19511.133575 secs 2024-07-26T14:06:09.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:08 smithi116 ceph-mon[37097]: pgmap v10720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:08 smithi171 ceph-mon[37789]: pgmap v10720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:09.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:08 smithi044 ceph-mon[32404]: pgmap v10720: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:11.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:10 smithi044 ceph-mon[32404]: pgmap v10721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:10 smithi171 ceph-mon[37789]: pgmap v10721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:11.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:10 smithi116 ceph-mon[37097]: pgmap v10721: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19516.133685 secs 2024-07-26T14:06:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19516.133685 secs 2024-07-26T14:06:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19516.133685 secs 2024-07-26T14:06:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:12 smithi171 ceph-mon[37789]: pgmap v10722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:13.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:12 smithi044 ceph-mon[32404]: pgmap v10722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:13.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:12 smithi116 ceph-mon[37097]: pgmap v10722: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:14 smithi171 ceph-mon[37789]: pgmap v10723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:15.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:14 smithi044 ceph-mon[32404]: pgmap v10723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:15.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:14 smithi116 ceph-mon[37097]: pgmap v10723: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:16.339+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:06:16.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:16.339+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:06:16.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:16.339+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:16 smithi171 ceph-mon[37789]: pgmap v10724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19521.133797 secs 2024-07-26T14:06:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:16 smithi044 ceph-mon[32404]: pgmap v10724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19521.133797 secs 2024-07-26T14:06:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:16 smithi116 ceph-mon[37097]: pgmap v10724: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19521.133797 secs 2024-07-26T14:06:19.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:18 smithi171 ceph-mon[37789]: pgmap v10725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:19.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:18 smithi044 ceph-mon[32404]: pgmap v10725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:19.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:18 smithi116 ceph-mon[37097]: pgmap v10725: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:21.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:20 smithi171 ceph-mon[37789]: pgmap v10726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:21.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:20 smithi044 ceph-mon[32404]: pgmap v10726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:21.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:20 smithi116 ceph-mon[37097]: pgmap v10726: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19526.133932 secs 2024-07-26T14:06:22.173 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19526.133932 secs 2024-07-26T14:06:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19526.133932 secs 2024-07-26T14:06:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:22 smithi171 ceph-mon[37789]: pgmap v10727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:23.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:22 smithi044 ceph-mon[32404]: pgmap v10727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:23.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:22 smithi116 ceph-mon[37097]: pgmap v10727: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:25.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:24 smithi171 ceph-mon[37789]: pgmap v10728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:25.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:24 smithi044 ceph-mon[32404]: pgmap v10728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:25.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:24 smithi116 ceph-mon[37097]: pgmap v10728: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:27.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:26 smithi044 ceph-mon[32404]: pgmap v10729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:27.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19531.134092 secs 2024-07-26T14:06:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:26 smithi116 ceph-mon[37097]: pgmap v10729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:27.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19531.134092 secs 2024-07-26T14:06:27.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:26 smithi171 ceph-mon[37789]: pgmap v10729: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:27.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19531.134092 secs 2024-07-26T14:06:29.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:28 smithi171 ceph-mon[37789]: pgmap v10730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:29.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:28 smithi044 ceph-mon[32404]: pgmap v10730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:29.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:28 smithi116 ceph-mon[37097]: pgmap v10730: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:31.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:30 smithi044 ceph-mon[32404]: pgmap v10731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:31.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:30 smithi116 ceph-mon[37097]: pgmap v10731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:31.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:30 smithi171 ceph-mon[37789]: pgmap v10731: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:31.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:31.339+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:06:31.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:31.339+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:06:31.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:31.339+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19536.134223 secs 2024-07-26T14:06:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19536.134223 secs 2024-07-26T14:06:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19536.134223 secs 2024-07-26T14:06:33.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:32 smithi044 ceph-mon[32404]: pgmap v10732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:33.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:32 smithi116 ceph-mon[37097]: pgmap v10732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:33.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:32 smithi171 ceph-mon[37789]: pgmap v10732: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:35.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:34 smithi044 ceph-mon[32404]: pgmap v10733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:35.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:34 smithi116 ceph-mon[37097]: pgmap v10733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:34 smithi171 ceph-mon[37789]: pgmap v10733: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:36 smithi044 ceph-mon[32404]: pgmap v10734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:37.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19541.134371 secs 2024-07-26T14:06:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:36 smithi116 ceph-mon[37097]: pgmap v10734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:37.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19541.134371 secs 2024-07-26T14:06:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:36 smithi171 ceph-mon[37789]: pgmap v10734: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19541.134371 secs 2024-07-26T14:06:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:38 smithi044 ceph-mon[32404]: pgmap v10735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:39.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:38 smithi116 ceph-mon[37097]: pgmap v10735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:39.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:38 smithi171 ceph-mon[37789]: pgmap v10735: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:40.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:39 smithi116 ceph-mon[37097]: pgmap v10736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:39 smithi044 ceph-mon[32404]: pgmap v10736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:39 smithi171 ceph-mon[37789]: pgmap v10736: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:06:41.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19546.134506 secs 2024-07-26T14:06:41.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19546.134506 secs 2024-07-26T14:06:41.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19546.134506 secs 2024-07-26T14:06:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:42 smithi171 ceph-mon[37789]: pgmap v10737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:42.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:42 smithi044 ceph-mon[32404]: pgmap v10737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:42.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:42.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:42 smithi116 ceph-mon[37097]: pgmap v10737: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:42.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:43.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:43 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:43 smithi171 ceph-mon[37789]: pgmap v10738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:43 smithi171 ceph-mon[37789]: pgmap v10739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:06:43.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:43 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:43.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:43 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:43 smithi044 ceph-mon[32404]: pgmap v10738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:43 smithi044 ceph-mon[32404]: pgmap v10739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:06:43.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:43 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:43 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:43 smithi116 ceph-mon[37097]: pgmap v10738: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:43 smithi116 ceph-mon[37097]: pgmap v10739: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:06:43.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:43 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:46.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:45 smithi044 ceph-mon[32404]: pgmap v10740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:46.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:45 smithi116 ceph-mon[37097]: pgmap v10740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:46.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:45 smithi171 ceph-mon[37789]: pgmap v10740: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:46.345+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:06:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:46.345+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:06:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:06:46.346+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:47.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:46 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19551.134670 secs 2024-07-26T14:06:47.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:46 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19551.134670 secs 2024-07-26T14:06:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:46 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19551.134670 secs 2024-07-26T14:06:48.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:47 smithi044 ceph-mon[32404]: pgmap v10741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:06:48.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:47 smithi116 ceph-mon[37097]: pgmap v10741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:06:48.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:47 smithi171 ceph-mon[37789]: pgmap v10741: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:06:50.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:49 smithi044 ceph-mon[32404]: pgmap v10742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:50.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:49 smithi116 ceph-mon[37097]: pgmap v10742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:49 smithi171 ceph-mon[37789]: pgmap v10742: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:52.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:51 smithi044 ceph-mon[32404]: pgmap v10743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:52.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:51 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19556.134768 secs 2024-07-26T14:06:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:51 smithi116 ceph-mon[37097]: pgmap v10743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:52.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:51 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19556.134768 secs 2024-07-26T14:06:52.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:51 smithi171 ceph-mon[37789]: pgmap v10743: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:06:52.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:51 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19556.134768 secs 2024-07-26T14:06:54.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:53 smithi044 ceph-mon[32404]: pgmap v10744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:54.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:53 smithi116 ceph-mon[37097]: pgmap v10744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:53 smithi171 ceph-mon[37789]: pgmap v10744: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:55.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:55 smithi044 ceph-mon[32404]: pgmap v10745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:55.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:55 smithi116 ceph-mon[37097]: pgmap v10745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:55 smithi171 ceph-mon[37789]: pgmap v10745: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:06:56.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:56 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19561.134920 secs 2024-07-26T14:06:56.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:56 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19561.134920 secs 2024-07-26T14:06:56.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:56 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19561.134920 secs 2024-07-26T14:06:57.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:57 smithi044 ceph-mon[32404]: pgmap v10746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:57.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:57 smithi116 ceph-mon[37097]: pgmap v10746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:57.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:57 smithi171 ceph-mon[37789]: pgmap v10746: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:00.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:06:59 smithi044 ceph-mon[32404]: pgmap v10747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:00.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:06:59 smithi116 ceph-mon[37097]: pgmap v10747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:06:59 smithi171 ceph-mon[37789]: pgmap v10747: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:01.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:01.347+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:07:01.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:01.347+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:07:01.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:01.347+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:02.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:01 smithi044 ceph-mon[32404]: pgmap v10748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:02.016 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:01 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19566.135087 secs 2024-07-26T14:07:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:01 smithi116 ceph-mon[37097]: pgmap v10748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:02.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:01 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19566.135087 secs 2024-07-26T14:07:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:01 smithi171 ceph-mon[37789]: pgmap v10748: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:01 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19566.135087 secs 2024-07-26T14:07:04.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:03 smithi044 ceph-mon[32404]: pgmap v10749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:04.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:03 smithi116 ceph-mon[37097]: pgmap v10749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:04.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:03 smithi171 ceph-mon[37789]: pgmap v10749: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:06.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:05 smithi044 ceph-mon[32404]: pgmap v10750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:05 smithi116 ceph-mon[37097]: pgmap v10750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:06.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:05 smithi171 ceph-mon[37789]: pgmap v10750: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:06.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:06 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19571.135235 secs 2024-07-26T14:07:07.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:06 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19571.135235 secs 2024-07-26T14:07:07.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:06 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19571.135235 secs 2024-07-26T14:07:08.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:07 smithi044 ceph-mon[32404]: pgmap v10751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:08.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:07 smithi116 ceph-mon[37097]: pgmap v10751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:07 smithi171 ceph-mon[37789]: pgmap v10751: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:10.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:09 smithi116 ceph-mon[37097]: pgmap v10752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:10.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:09 smithi171 ceph-mon[37789]: pgmap v10752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:10.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:09 smithi044 ceph-mon[32404]: pgmap v10752: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:12.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:11 smithi171 ceph-mon[37789]: pgmap v10753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:12.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:11 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19576.135400 secs 2024-07-26T14:07:12.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:11 smithi044 ceph-mon[32404]: pgmap v10753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:12.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:11 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19576.135400 secs 2024-07-26T14:07:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:11 smithi116 ceph-mon[37097]: pgmap v10753: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:12.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:11 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19576.135400 secs 2024-07-26T14:07:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:13 smithi171 ceph-mon[37789]: pgmap v10754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:14.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:13 smithi044 ceph-mon[32404]: pgmap v10754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:14.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:13 smithi116 ceph-mon[37097]: pgmap v10754: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:15 smithi171 ceph-mon[37789]: pgmap v10755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:16.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:15 smithi044 ceph-mon[32404]: pgmap v10755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:16.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:15 smithi116 ceph-mon[37097]: pgmap v10755: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:16.352+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:07:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:16.352+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:07:16.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:16.352+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:17.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:16 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19581.135505 secs 2024-07-26T14:07:17.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:16 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19581.135505 secs 2024-07-26T14:07:17.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:16 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19581.135505 secs 2024-07-26T14:07:18.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:17 smithi171 ceph-mon[37789]: pgmap v10756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:18.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:17 smithi044 ceph-mon[32404]: pgmap v10756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:18.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:17 smithi116 ceph-mon[37097]: pgmap v10756: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:19 smithi171 ceph-mon[37789]: pgmap v10757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:20.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:19 smithi044 ceph-mon[32404]: pgmap v10757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:20.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:19 smithi116 ceph-mon[37097]: pgmap v10757: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:21 smithi171 ceph-mon[37789]: pgmap v10758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:22.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:21 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19586.135669 secs 2024-07-26T14:07:22.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:21 smithi044 ceph-mon[32404]: pgmap v10758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:22.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:21 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19586.135669 secs 2024-07-26T14:07:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:21 smithi116 ceph-mon[37097]: pgmap v10758: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:22.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:21 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19586.135669 secs 2024-07-26T14:07:24.074 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:23 smithi116 ceph-mon[37097]: pgmap v10759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:23 smithi171 ceph-mon[37789]: pgmap v10759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:24.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:23 smithi044 ceph-mon[32404]: pgmap v10759: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:26.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:25 smithi171 ceph-mon[37789]: pgmap v10760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:26.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:25 smithi044 ceph-mon[32404]: pgmap v10760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:26.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:25 smithi116 ceph-mon[37097]: pgmap v10760: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:26 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19591.135765 secs 2024-07-26T14:07:27.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:26 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19591.135765 secs 2024-07-26T14:07:27.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:26 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19591.135765 secs 2024-07-26T14:07:28.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:27 smithi044 ceph-mon[32404]: pgmap v10761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:28.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:27 smithi116 ceph-mon[37097]: pgmap v10761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:27 smithi171 ceph-mon[37789]: pgmap v10761: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:30.139 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:29 smithi171 ceph-mon[37789]: pgmap v10762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:30.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:29 smithi044 ceph-mon[32404]: pgmap v10762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:30.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:29 smithi116 ceph-mon[37097]: pgmap v10762: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:31.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:31.353+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:07:31.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:31.353+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:07:31.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:31.353+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:32.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:31 smithi044 ceph-mon[32404]: pgmap v10763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:32.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:31 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19596.135901 secs 2024-07-26T14:07:32.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:31 smithi116 ceph-mon[37097]: pgmap v10763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:32.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:31 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19596.135901 secs 2024-07-26T14:07:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:31 smithi171 ceph-mon[37789]: pgmap v10763: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:31 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19596.135901 secs 2024-07-26T14:07:34.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:33 smithi044 ceph-mon[32404]: pgmap v10764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:34.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:33 smithi116 ceph-mon[37097]: pgmap v10764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:33 smithi171 ceph-mon[37789]: pgmap v10764: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:36.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:35 smithi044 ceph-mon[32404]: pgmap v10765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:36.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:35 smithi116 ceph-mon[37097]: pgmap v10765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:35 smithi171 ceph-mon[37789]: pgmap v10765: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:37.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:36 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19601.136066 secs 2024-07-26T14:07:37.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:36 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19601.136066 secs 2024-07-26T14:07:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:36 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19601.136066 secs 2024-07-26T14:07:38.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:37 smithi044 ceph-mon[32404]: pgmap v10766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:38.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:37 smithi116 ceph-mon[37097]: pgmap v10766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:38.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:37 smithi171 ceph-mon[37789]: pgmap v10766: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:40.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:39 smithi044 ceph-mon[32404]: pgmap v10767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:40.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:39 smithi116 ceph-mon[37097]: pgmap v10767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:39 smithi171 ceph-mon[37789]: pgmap v10767: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:42.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:41 smithi044 ceph-mon[32404]: pgmap v10768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:42.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:41 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19606.136228 secs 2024-07-26T14:07:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:41 smithi116 ceph-mon[37097]: pgmap v10768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:42.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:41 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19606.136228 secs 2024-07-26T14:07:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:41 smithi171 ceph-mon[37789]: pgmap v10768: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:42.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:41 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19606.136228 secs 2024-07-26T14:07:43.167 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:42 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:43.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:42 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:43.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:42 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:44.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:43 smithi044 ceph-mon[32404]: pgmap v10769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:44.271 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:43 smithi116 ceph-mon[37097]: pgmap v10769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:44.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:43 smithi171 ceph-mon[37789]: pgmap v10769: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:44 smithi044 ceph-mon[32404]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:44 smithi044 ceph-mon[32404]: pgmap v10770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:07:45.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:44 smithi044 ceph-mon[32404]: pgmap v10771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:07:45.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:44 smithi044 ceph-mon[32404]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:44 smithi116 ceph-mon[37097]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:44 smithi116 ceph-mon[37097]: pgmap v10770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:44 smithi116 ceph-mon[37097]: pgmap v10771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:07:45.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:44 smithi116 ceph-mon[37097]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:44 smithi171 ceph-mon[37789]: from='mgr.14211 172.21.15.44:0/1808376537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:44 smithi171 ceph-mon[37789]: pgmap v10770: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:44 smithi171 ceph-mon[37789]: pgmap v10771: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:07:45.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:44 smithi171 ceph-mon[37789]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:46.353+0000 7fcf71c00640 1 -- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 msgr2=0x55c948c5c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.171:6834/372796843 2024-07-26T14:07:46.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:46.353+0000 7fcf53400640 1 client.15669 ms_handle_refused on v2:172.21.15.171:6834/372796843 2024-07-26T14:07:46.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-07-26T14:07:46.353+0000 7fcf71c00640 1 --2- 172.21.15.44:0/2544213633 >> [v2:172.21.15.171:6834/372796843,v1:172.21.15.171:6835/372796843] conn(0x55c9488589f0 0x55c948c5c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:47 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19611.136371 secs 2024-07-26T14:07:47.272 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:47 smithi116 ceph-mon[37097]: pgmap v10772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:07:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:47 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19611.136371 secs 2024-07-26T14:07:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:47 smithi171 ceph-mon[37789]: pgmap v10772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:07:47.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:47 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19611.136371 secs 2024-07-26T14:07:47.516 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:47 smithi044 ceph-mon[32404]: pgmap v10772: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:07:49.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:49 smithi171 ceph-mon[37789]: pgmap v10773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:07:49.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:49 smithi044 ceph-mon[32404]: pgmap v10773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:07:49.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:49 smithi116 ceph-mon[37097]: pgmap v10773: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:07:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:51 smithi171 ceph-mon[37789]: pgmap v10774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:07:51.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:51 smithi044 ceph-mon[32404]: pgmap v10774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:07:51.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:51 smithi116 ceph-mon[37097]: pgmap v10774: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:07:52.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:52 smithi171 ceph-mon[37789]: 1 slow requests, 0 included below; oldest blocked for > 19616.136513 secs 2024-07-26T14:07:52.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:52 smithi044 ceph-mon[32404]: 1 slow requests, 0 included below; oldest blocked for > 19616.136513 secs 2024-07-26T14:07:52.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:52 smithi116 ceph-mon[37097]: 1 slow requests, 0 included below; oldest blocked for > 19616.136513 secs 2024-07-26T14:07:53.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:53 smithi171 ceph-mon[37789]: pgmap v10775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:53.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:53 smithi044 ceph-mon[32404]: pgmap v10775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:53.522 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:53 smithi116 ceph-mon[37097]: pgmap v10775: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:07:55.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Jul 26 14:07:55 smithi116 ceph-mon[37097]: pgmap v10776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:07:55.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Jul 26 14:07:55 smithi171 ceph-mon[37789]: pgmap v10776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:07:55.515 INFO:journalctl@ceph.mon.a.smithi044.stdout:Jul 26 14:07:55 smithi044 ceph-mon[32404]: pgmap v10776: 129 pgs: 129 active+clean; 60 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:07:55.629 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-26T14:07:55.629 DEBUG:teuthology.task.console_log:Killing console logger for smithi044 2024-07-26T14:07:55.630 DEBUG:teuthology.task.console_log:Killing console logger for smithi116 2024-07-26T14:07:55.630 DEBUG:teuthology.task.console_log:Killing console logger for smithi171 2024-07-26T14:07:55.631 DEBUG:teuthology.exit:Finished running handlers